[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15645/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Sat, 23 Mar 2024 05:10:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. IMPALA-12896 (Part 2): JDBC table must be created as external table In some of the deployment environments, default table type is transactional. In these scenarios, JDBC tables which are created as non external table are not accepted by HMS due to strict managed table check failures. This patch forces JDBC tables to be created as external table, and requires at least 1 column for JDBC tables. Testing: - Updated frontend unit tests and end-to-end unit tests to create JDBC tables as external tables. - Passed core tests Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M testdata/bin/create-ext-data-source-table.sql M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 9 files changed, 38 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/21159/5 -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10419/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Sat, 23 Mar 2024 04:48:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10418/ -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Sat, 23 Mar 2024 04:30:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4545: Simplify test dimension in test decimal casting.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21174 ) Change subject: IMPALA-4545: Simplify test dimension in test_decimal_casting.py .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe269e08a955097ad9e924d5d64b42438ad15be2 Gerrit-Change-Number: 21174 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Sat, 23 Mar 2024 02:34:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4545: Simplify test dimension in test decimal casting.py
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21174 ) Change subject: IMPALA-4545: Simplify test dimension in test_decimal_casting.py .. IMPALA-4545: Simplify test dimension in test_decimal_casting.py This patch splits precision and scale as independent dimensions and then constrains them to yield a valid decimal type. With this split, core exploration will have the same test dimension as pairwise exploration, while exhaustive exploration still permutes all possible decimal types. Also did minor refactoring to reduce test skipping and pass flake8. After this patch, core exploration has 214 test items and exhaustive exploration has 12312 test items. Before, they were 408 and 12464 respectively. Testing: - Pass test_decimal_casting.py in core and exhaustive exploration. Change-Id: Ibe269e08a955097ad9e924d5d64b42438ad15be2 Reviewed-on: http://gerrit.cloudera.org:8080/21174 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_decimal_casting.py 1 file changed, 48 insertions(+), 29 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibe269e08a955097ad9e924d5d64b42438ad15be2 Gerrit-Change-Number: 21174 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/21191 )
Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the
minicluster during TSAN runs
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21191/2/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/21191/2/testdata/bin/run-hive-server.sh@169
PS2, Line 169: if [[ ! ${SKIP_LOADING_LIBFESUPPORT_IN_HIVE} ]]; then
This check is always false. What we want is either:
if ! $SKIP_LOADING_LIBFESUPPORT_IN_HIVE ; then
or
if [[ "$SKIP_LOADING_LIBFESUPPORT_IN_HIVE" != "true" ]]; then
We'll have a problem if SKIP_LOADING_LIBFESUPPORT_IN_HIVE is not set as that
will trip up the "set -euo pipefail" above. So, running this from a place other
than buildall.sh is a problem (and developers do that).
One thing we could do is look at the ${IMPALA_HOME}/.cmake_build_type file
(produced by our CMake build) and see if it contains TSAN. We use that file
from tests/common/environ.py to determine the build type.
if ! grep -q "TSAN" ${IMPALA_HOME}/.cmake_build_type ; then
The file would definitely exist for automated test jobs, and if it doesn't
exist, the code would just go ahead and add libfesupport.so.
--
To view, visit http://gerrit.cloudera.org:8080/21191
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20
Gerrit-Change-Number: 21191
Gerrit-PatchSet: 2
Gerrit-Owner: Laszlo Gaal
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Sat, 23 Mar 2024 00:47:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12933: Avoid fetching unneccessary event types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: WIP IMPALA-12933: Avoid fetching unneccessary event types .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15644/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 23 Mar 2024 00:18:58 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12933: Avoid fetching unneccessary event types
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21186 to look at the new patch set (#2). Change subject: WIP IMPALA-12933: Avoid fetching unneccessary event types .. WIP IMPALA-12933: Avoid fetching unneccessary event types There are several places that catalogd will fetch all events of a specifit type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. Also adds UPDATE_PART_COL_STAT_EVENT to the default skip list. Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 8 files changed, 138 insertions(+), 48 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/2 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite data files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21190 ) Change subject: IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21190 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f Gerrit-Change-Number: 21190 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 23:36:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20762 ) Change subject: IMPALA-12540: Query Live Table .. Patch Set 47: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15643/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 Gerrit-Change-Number: 20762 Gerrit-PatchSet: 47 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Fri, 22 Mar 2024 23:33:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12626: Add Tables Queried to profile/history
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20886 ) Change subject: IMPALA-12626: Add Tables Queried to profile/history .. Patch Set 29: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15642/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c9c80b2adf7f3e44225a191fe8eb9df3c4bc5aa Gerrit-Change-Number: 20886 Gerrit-PatchSet: 29 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Mar 2024 23:31:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20762 ) Change subject: IMPALA-12540: Query Live Table .. Patch Set 46: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15641/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 Gerrit-Change-Number: 20762 Gerrit-PatchSet: 46 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Fri, 22 Mar 2024 23:30:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 3: Thanks Joe -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 22 Mar 2024 23:20:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10418/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 22 Mar 2024 23:21:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21159 ) Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external table .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9 Gerrit-Change-Number: 21159 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Fri, 22 Mar 2024 23:21:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/21159 )
Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external
table
..
Patch Set 3: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21159/3/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/21159/3/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@272
PS3, Line 272: THdfsFileFormat.ICEBERG) {
> That was added in patch https://gerrit.cloudera.org/#/c/21016/.
Ah, ok, now I get it.
--
To view, visit http://gerrit.cloudera.org:8080/21159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9
Gerrit-Change-Number: 21159
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Fri, 22 Mar 2024 23:12:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Hello Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20762 to look at the new patch set (#47). Change subject: IMPALA-12540: Query Live Table .. IMPALA-12540: Query Live Table Defines SystemTable which are in-memory tables that can provide access to Impala state. Adds the 'impala_query_live' to the database 'sys', which already exists for 'sys.impala_query_log'. Implements the 'impala_query_live' table to view active queries across all coordinators sharing the same statestore. SystemTables create new SystemTableScanNodes for their scan node implementation. When computing scan range locations, SystemTableScanNodes creates a scan range for each in the cluster (identified via ClusterMembershipMgr). This produces a plan that looks like: Query: explain select * from sys.impala_query_live ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=4.00MB Threads=2 | | Per-Host Resource Estimates: Memory=11MB | | WARNING: The following tables are missing relevant table | | and/or column statistics. | | sys.impala_query_live | || | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED]| | | | | 00:SCAN SYSTEM_TABLE [sys.impala_query_live] | |row-size=72B cardinality=20 | ++ Impala's scheduler checks for whether the query contains fragments that can be scheduled on coordinators, and if present includes an ExecutorGroup containing all coordinators. These are used to schedule scan ranges that are flagged as 'use_coordinator', allowing SystemTableScanNodes to be scheduled on dedicated coordinators and outside the selected executor group. Execution will pull data from ImpalaServer on the backend via a SystemTableScanner implementation based on table name. In the query profile, SYSTEM_TABLE_SCAN_NODE includes ActiveQueryCollectionTime and PendingQueryCollectionTime to track time spent collecting QueryState from ImpalaServer. Grants QueryScanner private access to ImpalaServer, identical to how ImpalaHttpHandler access internal server state. Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 --- M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc M be/src/exec/scan-node.cc A be/src/exec/system-table-scan-node.cc A be/src/exec/system-table-scan-node.h A be/src/exec/system-table-scanner.cc A be/src/exec/system-table-scanner.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/fe-support.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-state-record.h M be/src/service/workload-management-fields.cc M be/src/service/workload-management.cc M be/src/util/sharded-query-map-util.h M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/PlanNodes.thrift M common/thrift/Query.thrift A common/thrift/SystemTables.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/SystemTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java A fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/catalog/SystemTableTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20762 ) Change subject: IMPALA-12540: Query Live Table .. Patch Set 46: (1 comment) http://gerrit.cloudera.org:8080/#/c/20762/41/tests/custom_cluster/test_query_live.py File tests/custom_cluster/test_query_live.py: http://gerrit.cloudera.org:8080/#/c/20762/41/tests/custom_cluster/test_query_live.py@24 PS41, Line 24: from tests.util.workload_management import a > Ok, that was incorrect. But something weird is going on when I try to run t Had to modify the scheduler to have a mode that will schedule specific scans across all coordinators, independent of executor groups. -- To view, visit http://gerrit.cloudera.org:8080/20762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 Gerrit-Change-Number: 20762 Gerrit-PatchSet: 46 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Fri, 22 Mar 2024 23:08:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Hello Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20762 to look at the new patch set (#46). Change subject: IMPALA-12540: Query Live Table .. IMPALA-12540: Query Live Table Defines SystemTable which are in-memory tables that can provide access to Impala state. Adds the 'impala_query_live' to the database 'sys', which already exists for 'sys.impala_query_log'. Implements the 'impala_query_live' table to view active queries across all coordinators sharing the same statestore. SystemTables create new SystemTableScanNodes for their scan node implementation. When computing scan range locations, SystemTableScanNodes creates a scan range for each in the cluster (identified via ClusterMembershipMgr). This produces a plan that looks like: Query: explain select * from sys.impala_query_live ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=4.00MB Threads=2 | | Per-Host Resource Estimates: Memory=11MB | | WARNING: The following tables are missing relevant table | | and/or column statistics. | | sys.impala_query_live | || | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED]| | | | | 00:SCAN SYSTEM_TABLE [sys.impala_query_live] | |row-size=72B cardinality=20 | ++ Impala's scheduler checks for whether the query contains fragments that can be scheduled on coordinators, and if present includes an ExecutorGroup containing all coordinators. These are used to schedule scan ranges that are flagged as 'use_coordinator', allowing SystemTableScanNodes to be scheduled on dedicated coordinators and outside the selected executor group. Execution will pull data from ImpalaServer on the backend via a SystemTableScanner implementation based on table name. In the query profile, SYSTEM_TABLE_SCAN_NODE includes ActiveQueryCollectionTime and PendingQueryCollectionTime to track time spent collecting QueryState from ImpalaServer. Grants QueryScanner private access to ImpalaServer, identical to how ImpalaHttpHandler access internal server state. Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 --- M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc M be/src/exec/scan-node.cc A be/src/exec/system-table-scan-node.cc A be/src/exec/system-table-scan-node.h A be/src/exec/system-table-scanner.cc A be/src/exec/system-table-scanner.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/fe-support.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-state-record.h M be/src/service/workload-management-fields.cc M be/src/service/workload-management.cc M be/src/util/sharded-query-map-util.h M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/PlanNodes.thrift M common/thrift/Query.thrift A common/thrift/SystemTables.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/SystemTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java A fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/catalog/SystemTableTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
[Impala-ASF-CR] IMPALA-12626: Add Tables Queried to profile/history
Hello Andrew Sherman, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20886 to look at the new patch set (#29). Change subject: IMPALA-12626: Add Tables Queried to profile/history .. IMPALA-12626: Add Tables Queried to profile/history Adds "Tables Queried" to the query profile, enumerating a comma-separated list of tables accessed during a query: Tables Queried: tpch.customer,tpch.lineitem Also adds "tables_queried" to impala_query_log and impala_query_live with the same content. Requires 'drop table sys.impala_query_log' to recreate it with the new column. Change-Id: I9c9c80b2adf7f3e44225a191fe8eb9df3c4bc5aa --- M be/src/exec/system-table-scanner.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/query-state-record.cc M be/src/service/query-state-record.h M be/src/service/workload-management-fields.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/error-util.cc M common/thrift/Frontend.thrift M common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_query_live.py M tests/query_test/test_observability.py M tests/util/workload_management.py 15 files changed, 87 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20886/29 -- To view, visit http://gerrit.cloudera.org:8080/20886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c9c80b2adf7f3e44225a191fe8eb9df3c4bc5aa Gerrit-Change-Number: 20886 Gerrit-PatchSet: 29 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] WIP IMPALA-5323: Support Kudu BINARY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18868 ) Change subject: WIP IMPALA-5323: Support Kudu BINARY .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10415/ -- To view, visit http://gerrit.cloudera.org:8080/18868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff701a4b3a09ce7b6982c5d238e65f3d4f3d1151 Gerrit-Change-Number: 18868 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Fri, 22 Mar 2024 23:10:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20762 ) Change subject: IMPALA-12540: Query Live Table .. Patch Set 46: (1 comment) http://gerrit.cloudera.org:8080/#/c/20762/46/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/20762/46/be/src/scheduling/scheduler.cc@101 PS46, Line 101: // Coordinator host may not be in executor_config's executor group if it's a dedicated line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/20762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 Gerrit-Change-Number: 20762 Gerrit-PatchSet: 46 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Fri, 22 Mar 2024 23:08:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21191 ) Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15640/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 Gerrit-Change-Number: 21191 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 22:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21159 )
Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external
table
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21159/3/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/21159/3/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@272
PS3, Line 272: THdfsFileFormat.ICEBERG) {
> Why did we drop JDBC from here?
That was added in patch https://gerrit.cloudera.org/#/c/21016/.
But now we don't provide a way to get column schema automatically from remote
JDBC server if JDBC has empty column defs. So user has to provide column defs
which require at least one column.
--
To view, visit http://gerrit.cloudera.org:8080/21159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9
Gerrit-Change-Number: 21159
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Fri, 22 Mar 2024 22:01:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21191 ) Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15639/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 Gerrit-Change-Number: 21191 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 22:04:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21191 ) Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21191/1/testdata/bin/run-hive-server.sh File testdata/bin/run-hive-server.sh: http://gerrit.cloudera.org:8080/#/c/21191/1/testdata/bin/run-hive-server.sh@176 PS1, Line 176: ex > Ooops Done -- To view, visit http://gerrit.cloudera.org:8080/21191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 Gerrit-Change-Number: 21191 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 21:44:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21191 to look at the new patch set (#2). Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs .. IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs During TSAN runs all Impala binaries (including libfesupport.so) are built with TSAN options, which include a reference to the external symbol __tsan_init. This causes a problem for libfesupport.so when it is loaded into Hive during minicluster startup, because the Java VM running Hive's code cannot supply this symbol (the stock JVM is obviously not built with TSAN). Unfortunately this symbol resolution failure causes Hive's JVM simply to abort on Red Hat 8 (or later) and on Ubuntu 20.04 (or later), unlike on earlier versions of the same platforms, where the JVM turns the failure into an exception that is actually handled by Hive. This patch prevents libfesupport.so from being loaded into Hive for TSAN runs so that the minicluster can actually be started. This is achieved by not adding the directory containing libfesupport.so to JAVA_LIBRARY_PATH, preventing the JVM from finding it. Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 --- M buildall.sh M testdata/bin/run-hive-server.sh 2 files changed, 18 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21191/2 -- To view, visit http://gerrit.cloudera.org:8080/21191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 Gerrit-Change-Number: 21191 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21191 ) Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs .. Patch Set 1: Code-Review-1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21191/1/testdata/bin/run-hive-server.sh File testdata/bin/run-hive-server.sh: http://gerrit.cloudera.org:8080/#/c/21191/1/testdata/bin/run-hive-server.sh@176 PS1, Line 176: # Ooops -- To view, visit http://gerrit.cloudera.org:8080/21191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 Gerrit-Change-Number: 21191 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 21:42:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21191 Change subject: IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs .. IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during TSAN runs During TSAN runs all Impala binaries (including libfesupport.so) are built with TSAN options, which include a reference to the external symbol __tsan_init. This causes a problem for libfesupport.so when it is loaded into Hive during minicluster startup, because the Java VM running Hive's code cannot supply this symbol (the stock JVM is obviously not built with TSAN). Unfortunately this symbol resolution failure causes Hive's JVM simply to abort on Red Hat 8 (or later) and on Ubuntu 20.04 (or later), unlike on earlier versions of the same platforms, where the JVM turns the failure into an exception that is actually handled by Hive. This patch prevents libfesupport.so from being loaded into Hive for TSAN runs so that the minicluster can actually be started. This is achieved by not adding the directory containing libfesupport.so to JAVA_LIBRARY_PATH, preventing the JVM from finding it. Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 --- M buildall.sh M testdata/bin/run-hive-server.sh 2 files changed, 18 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21191/1 -- To view, visit http://gerrit.cloudera.org:8080/21191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20 Gerrit-Change-Number: 21191 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] IMPALA-4545: Simplify test dimension in test decimal casting.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21174 ) Change subject: IMPALA-4545: Simplify test dimension in test_decimal_casting.py .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10417/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe269e08a955097ad9e924d5d64b42438ad15be2 Gerrit-Change-Number: 21174 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 22 Mar 2024 21:32:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4545: Simplify test dimension in test decimal casting.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21174 ) Change subject: IMPALA-4545: Simplify test dimension in test_decimal_casting.py .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe269e08a955097ad9e924d5d64b42438ad15be2 Gerrit-Change-Number: 21174 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 22 Mar 2024 21:32:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4545: Simplify test dimension in test decimal casting.py
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21174 ) Change subject: IMPALA-4545: Simplify test dimension in test_decimal_casting.py .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21174 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe269e08a955097ad9e924d5d64b42438ad15be2 Gerrit-Change-Number: 21174 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 22 Mar 2024 21:24:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 21:24:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12896 (Part 2): JDBC table must be created as external table
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/21159 )
Change subject: IMPALA-12896 (Part 2): JDBC table must be created as external
table
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21159/3/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/21159/3/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@272
PS3, Line 272: THdfsFileFormat.ICEBERG) {
Why did we drop JDBC from here?
--
To view, visit http://gerrit.cloudera.org:8080/21159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5533b52434cdf1c430e30ac28a0146ab4d9d4b9
Gerrit-Change-Number: 21159
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Fri, 22 Mar 2024 21:18:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15638/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 12 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 22 Mar 2024 21:21:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21179 ) Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10413/ -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 21:08:06 + Gerrit-HasComments: No
[native-toolchain-CR] Add libpfm / google benchmark
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21164 ) Change subject: Add libpfm / google benchmark .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a Gerrit-Change-Number: 21164 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 20:57:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Hello Aman Sinha, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21109 to look at the new patch set (#12). Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. IMPALA-12872: Use Calcite for optimization - part 1: simple queries This is the first commit to use the Calcite library to parse, analyze, and optimize queries. The hook for the planner is through an override of the JniFrontend. The CalciteJniFrontend class is the driver that walks through each of the Calcite steps which are as follows: CalciteQueryParser: Takes the string query and outputs an AST in the form of Calcite's SqlNode object. CalciteMetadataHandler: Iterate through the SqlNode from the previous step and make sure all essential table metadata is retrieved from catalogd. CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer. CalciteRelNodeConverter: Change the AST into a logical plan. In this first commit, the only logical nodes used are LogicalTableScan and LogicalProject. The LogicalTableScan will serve as the node that reads from an Hdfs Table and the LogicalProject will only project out the used columns in the query. In later versions, the LogicalProject will also handle function changes. CalciteOptimizer: This step is to optimize the query. In this cut, it will be a nop, but in later versions, it will perform logical optimizations via Calcite's rule mechanism. CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into Impala's PlanNode physical tree ExecRequestCreator: Implement the existing Impala steps that turn a Single Node Plan into a Distributed Plan. It will also create the TExecRequest object needed by the runtime server. Only some very basic queries will work with this commit. These include: select * from tbl <-- only needs the LogicalTableScan select c1 from tbl <-- Also uses the LogicalProject In the CalciteJniFrontend, there is some basic checks to make sure only select statements will get processed. Any non-query statement will revert back to the current Impala planner. In this iteration, any queries besides the minimal ones listed above will result in a caught exception which will then be run through the current Impala planner. The tests that do work can be found in calcite.test and run through the custom cluster test test_experimental_planner.py Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 --- M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/calcite-planner/pom.xml A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeWithExprs.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaCalciteCatalogReader.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java A
[native-toolchain-CR] Add RE2 2024-03-01 and Abseil
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21165 ) Change subject: Add RE2 2024-03-01 and Abseil .. Patch Set 3: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/21165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 Gerrit-Change-Number: 21165 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Mar 2024 20:47:41 + Gerrit-HasComments: No
[native-toolchain-CR] Add libpfm / google benchmark
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21164 ) Change subject: Add libpfm / google benchmark .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21164/2/source/googlebenchmark/build.sh File source/googlebenchmark/build.sh: http://gerrit.cloudera.org:8080/#/c/21164/2/source/googlebenchmark/build.sh@2 PS2, Line 2: 2015 > nit: 2024 Done http://gerrit.cloudera.org:8080/#/c/21164/2/source/libpfm/build.sh File source/libpfm/build.sh: http://gerrit.cloudera.org:8080/#/c/21164/2/source/libpfm/build.sh@2 PS2, Line 2: 2016 > nit: 2024 Done -- To view, visit http://gerrit.cloudera.org:8080/21164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a Gerrit-Change-Number: 21164 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 20:47:31 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add libpfm / google benchmark
Hello Laszlo Gaal, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21164 to look at the new patch set (#3). Change subject: Add libpfm / google benchmark .. Add libpfm / google benchmark Google benchmark is a framework for writing benchmarks similar to Googletest's framework for tests. An interesting feature of google benchmark is that it integrates with libpfm / perfmon2 to get hardware performance counters as described in https://abseil.io/fast/53 . This adds a build for both libpfm and google benchmark to allow testing this capability for microbenchmarks. Testing: - Ran a toolchain build Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a --- M buildall.sh M functions.sh A source/googlebenchmark/build.sh A source/libpfm/build.sh 4 files changed, 105 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/21164/3 -- To view, visit http://gerrit.cloudera.org:8080/21164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a Gerrit-Change-Number: 21164 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Add libpfm / google benchmark
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21164 ) Change subject: Add libpfm / google benchmark .. Patch Set 2: (2 comments) Just two copyright headers... http://gerrit.cloudera.org:8080/#/c/21164/2/source/googlebenchmark/build.sh File source/googlebenchmark/build.sh: http://gerrit.cloudera.org:8080/#/c/21164/2/source/googlebenchmark/build.sh@2 PS2, Line 2: 2015 nit: 2024 http://gerrit.cloudera.org:8080/#/c/21164/2/source/libpfm/build.sh File source/libpfm/build.sh: http://gerrit.cloudera.org:8080/#/c/21164/2/source/libpfm/build.sh@2 PS2, Line 2: 2016 nit: 2024 -- To view, visit http://gerrit.cloudera.org:8080/21164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a Gerrit-Change-Number: 21164 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2024 20:44:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10412/ -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 20:47:27 + Gerrit-HasComments: No
[native-toolchain-CR] Add RE2 2024-03-01 and Abseil
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21165 ) Change subject: Add RE2 2024-03-01 and Abseil .. Patch Set 2: Code-Review+2 Thank you :) -- To view, visit http://gerrit.cloudera.org:8080/21165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 Gerrit-Change-Number: 21165 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Mar 2024 20:39:17 + Gerrit-HasComments: No
[native-toolchain-CR] Add RE2 2024-03-01 and Abseil
Hello Laszlo Gaal, Michael Smith, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21165 to look at the new patch set (#2). Change subject: Add RE2 2024-03-01 and Abseil .. Add RE2 2024-03-01 and Abseil This adds a build of the latest RE2 release 2024-03-01. RE2 took a dependency on Abseil starting in 2023-06-01, so this adds Abseil as well. Abseil is a C++ library developed by Google that provides various functionality including string libraries, hash tables, etc. It is useful on its own. Abseil's shared library build produces a large number of shared libraries that depend on each other. This implements a basic optimization of the symlink_required_libs function to check the destination directory to see if the required library has already been symlinked (or is otherwise present). This makes a big difference in the time take for symlink_required_libs for Abseil. Testing: - Ran a toolchain build Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 --- M buildall.sh M functions.sh A source/abseil-cpp/build.sh M source/re2/build.sh 4 files changed, 146 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/21165/2 -- To view, visit http://gerrit.cloudera.org:8080/21165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 Gerrit-Change-Number: 21165 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15637/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 11 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 22 Mar 2024 20:15:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15636/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 6 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Fri, 22 Mar 2024 19:54:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/21109/11/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java: http://gerrit.cloudera.org:8080/#/c/21109/11/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java@200 PS11, Line 200: localTableNames.add(new TableName(parts.get(0).toLowerCase(), parts.get(1).toLowerCase())); line too long (101 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 11 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Fri, 22 Mar 2024 19:46:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 10: (9 comments) http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java: http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java@69 PS10, Line 69: nor or partitions > Nit: "nor are partitions" Done http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java@97 PS10, Line 97: int totalCols = table.getColumns().size(); > For my own understanding: Is it true that totalCols == numFields? Yeah, I think so. This *might* be an issue when we deal with Acid tables that have virtual columns? But since we're not doing that right now and I haven't tested with that yet, it doesn't make sense to have 2 separate variables. I'm not sure I changed this as you desired. But I did get rid of the extra variable http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java: http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java@70 PS10, Line 70: // O BENEVOLENT REVIEWER AND CODE INSPECTOR... : // TODO: Please hold off on reviewing this file. I held off on cleaning this up until : // this gets past the experimental stage. Some of the code in SingleNodePlanner : // is duplicated here, so this will involve a general rewrite. After more Calcite : // code gets committed and the planner works for a good portion of the queries, this : // will get rewritten into its final form. > Is this comment still true? Are there rewrites to come for this file? Sigh, unfortunately, yes, it's still true. My goal with this commit was to get a first pass Calcite commit in and make as few changes to Impala under "fe" as I possibly can. This allows the code review to be a bit simpler. The code in here is mostly common code with existing Impala code as all this happens after we have done the conversion into PlanNode. So to do this right, the code under fe/.../planner/* should be refactored. I do want to do this later, which is why I left this comment. I'm open to doing this sooner rather than later if you think though. http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java: http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java@107 PS10, Line 107: case DECIMAL: : RelDataType decimalDefinedRetType = factory.createSqlType(SqlTypeName.DECIMAL, : scalarType.decimalPrecision(), scalarType.decimalScale()); : return factory.createTypeWithNullability(decimalDefinedRetType, true); : case VARCHAR: : return createCharType(factory, SqlTypeName.VARCHAR, scalarType.getLength()); : case CHAR: : return createCharType(factory, SqlTypeName.CHAR, scalarType.getLength()); > If I understand this right, we could omit DECIMAL, VARCHAR, and CHAR from t In this commit, you are correct, so I shall remove it. I think I need to put this back in future commits, but I'll do that when the time comes. http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java@169 PS10, Line 169: Charset charSetName = Charset.forName(ConversionUtil.NATIVE_UTF16_CHARSET_NAME); > Does this charset do anything for execution? Nah, prolly not. I put this in because I saw this used in another project that used Calcite, but I'm gonna delete this unless we see a need for it in the future. http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java: http://gerrit.cloudera.org:8080/#/c/21109/10/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java@40 PS10, Line 40: private static final int MAX_BINARY_PRECISION = Integer.MAX_VALUE; :
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Hello Aman Sinha, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21109 to look at the new patch set (#11). Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. IMPALA-12872: Use Calcite for optimization - part 1: simple queries This is the first commit to use the Calcite library to parse, analyze, and optimize queries. The hook for the planner is through an override of the JniFrontend. The CalciteJniFrontend class is the driver that walks through each of the Calcite steps which are as follows: CalciteQueryParser: Takes the string query and outputs an AST in the form of Calcite's SqlNode object. CalciteMetadataHandler: Iterate through the SqlNode from the previous step and make sure all essential table metadata is retrieved from catalogd. CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer. CalciteRelNodeConverter: Change the AST into a logical plan. In this first commit, the only logical nodes used are LogicalTableScan and LogicalProject. The LogicalTableScan will serve as the node that reads from an Hdfs Table and the LogicalProject will only project out the used columns in the query. In later versions, the LogicalProject will also handle function changes. CalciteOptimizer: This step is to optimize the query. In this cut, it will be a nop, but in later versions, it will perform logical optimizations via Calcite's rule mechanism. CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into Impala's PlanNode physical tree ExecRequestCreator: Implement the existing Impala steps that turn a Single Node Plan into a Distributed Plan. It will also create the TExecRequest object needed by the runtime server. Only some very basic queries will work with this commit. These include: select * from tbl <-- only needs the LogicalTableScan select c1 from tbl <-- Also uses the LogicalProject In the CalciteJniFrontend, there is some basic checks to make sure only select statements will get processed. Any non-query statement will revert back to the current Impala planner. In this iteration, any queries besides the minimal ones listed above will result in a caught exception which will then be run through the current Impala planner. The tests that do work can be found in calcite.test and run through the custom cluster test test_experimental_planner.py Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 --- M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/calcite-planner/pom.xml A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeWithExprs.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaCalciteCatalogReader.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java A
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluster Tests
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21153 to look at the new patch set (#6). Change subject: IMPALA-12913: Refactor Workload Management Custom Cluster Tests .. IMPALA-12913: Refactor Workload Management Custom Cluster Tests The custom cluster tests that assert the workload management functionality to insert completed queries into the impala_query_log table were inefficient because they created their own database tables and added data to those tables. This patch updates these tests to use the existing tables in the functional database where possible. The few tests that need their own tables now have those tables set up in a database created by the pytest unique_database fixture instead of using the default database. A new table has also been added to the functional database. This table is named zipcode_timezones and contains two columns, the first having a few zipcodes and the second having their corresponding timezone. This table can be used to join the zipcode_incomes and alltimezones tables. This table is populated by a new csv file in the testdata directory. Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b --- A testdata/data/zipcodes_timezones.csv M testdata/datasets/functional/functional_schema_template.sql M tests/custom_cluster/test_query_log.py M tests/util/workload_management.py 4 files changed, 1,053 insertions(+), 491 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21153/6 -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 6 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite data files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21190 ) Change subject: IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15635/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21190 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f Gerrit-Change-Number: 21190 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 19:19:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite data files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21190 ) Change subject: IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15634/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21190 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f Gerrit-Change-Number: 21190 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 19:10:39 + Gerrit-HasComments: No
[Impala-ASF-CR] PRELIMINIARY COUNT(*)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21189 ) Change subject: PRELIMINIARY COUNT(*) .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15633/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13a7cbb926d4ca56bc17690d61652fb837ebd672 Gerrit-Change-Number: 21189 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 19:01:33 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12933: Avoid fetching unneccessary event types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: WIP IMPALA-12933: Avoid fetching unneccessary event types .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10411/ -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 18:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-5323: Support Kudu BINARY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18868 ) Change subject: WIP IMPALA-5323: Support Kudu BINARY .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15632/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff701a4b3a09ce7b6982c5d238e65f3d4f3d1151 Gerrit-Change-Number: 18868 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Fri, 22 Mar 2024 18:21:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. IMPALA-12898: Tidy up test dimensions of test_scanner.py This patch tidies up the test dimensions of test_scanner.py. 'exec_option' initialization is moved to add_test_dimensions() method as much as possible. It ensures correct permutation and execution of test cases. After this patch, the total collected tests of test_scanner.py is 1242 for core/pairwise exploration and 7514 for exhaustive exploration. Before, they were 794 and 11864 accordingly. The increase in test count after refactoring with core exploration is because exec option dimensions are now permuted correctly along with other default exec option dimensions. The reduction in exhaustive exploration is due to a reduction in the overall dimension to permute and a reduction in test skipping (the test was run, but only called pytest.skip()). Testing: - Pass query_test/test_scanners.py in exhaustive exploration. Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Reviewed-on: http://gerrit.cloudera.org:8080/21162 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_scanners.py 1 file changed, 82 insertions(+), 83 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite data files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21190 ) Change subject: IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10416/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21190 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f Gerrit-Change-Number: 21190 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 18:18:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 18:18:44 + Gerrit-HasComments: No
[Impala-ASF-CR] PRELIMINIARY COUNT(*)
Zoltan Borok-Nagy has abandoned this change. ( http://gerrit.cloudera.org:8080/21189 ) Change subject: PRELIMINIARY COUNT(*) .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/21189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I13a7cbb926d4ca56bc17690d61652fb837ebd672 Gerrit-Change-Number: 21189 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite data files
Zoltan Borok-Nagy has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21190 ) Change subject: IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files .. IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files Impala can return incorrect results if a table has dangling delete files. During analysis we check the existence of delete files based on the snapshot summary. But during planning in IcebergScanPlanner we do it based on planFiles(), i.e. dangling delete files don't count in the latter case. Because of this Impala can create incorrect plans for count(*) optimization. This patch fixes the FeIcebergTable.hasDeleteFiles() method, so it ignores dangling delete files. TODO: * introduce query option so we can completely disable the count(*) optimization Testing: * e2e tests * planner tests Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-hash-join.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test 7 files changed, 307 insertions(+), 431 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/21190/2 -- To view, visit http://gerrit.cloudera.org:8080/21190 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f Gerrit-Change-Number: 21190 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite data files
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21190 Change subject: IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files .. IMPALA-12894: Optimized count(*) for Iceberg gives wrong results after a Spark rewrite_data_files Impala can return incorrect results if a table has dangling delete files. During analysis we check the existence of delete files based on the snapshot summary. But during planning in IcebergScanPlanner we do it based on planFiles(), i.e. dangling delete files don't count in the latter case. Because of this Impala can create incorrect plans for count(*) optimization. This patch fixes the FeIcebergTable.hasDeleteFiles() method, so it ignores dangling delete files. TODO: * introduce query option so we can completely disable the count(*) optimization Testing: * e2e tests * planner tests Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-hash-join.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test 7 files changed, 307 insertions(+), 430 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/21190/1 -- To view, visit http://gerrit.cloudera.org:8080/21190 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie3aca0b0a104f9ca4589cde9643f3f341d4ff99f Gerrit-Change-Number: 21190 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] PRELIMINIARY COUNT(*)
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21189 Change subject: PRELIMINIARY COUNT(*) .. PRELIMINIARY COUNT(*) Change-Id: I13a7cbb926d4ca56bc17690d61652fb837ebd672 --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java 1 file changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/21189/1 -- To view, visit http://gerrit.cloudera.org:8080/21189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I13a7cbb926d4ca56bc17690d61652fb837ebd672 Gerrit-Change-Number: 21189 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] WIP IMPALA-5323: Support Kudu BINARY
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/18868 )
Change subject: WIP IMPALA-5323: Support Kudu BINARY
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18868/3/be/src/exprs/expr-test-common.h
File be/src/exprs/expr-test-common.h:
http://gerrit.cloudera.org:8080/#/c/18868/3/be/src/exprs/expr-test-common.h@552
PS3, Line 552: void ExprTestBase::TestTimestampValue(const string& expr, const
TimestampValue& expected_result) {
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18868
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff701a4b3a09ce7b6982c5d238e65f3d4f3d1151
Gerrit-Change-Number: 18868
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
Gerrit-Comment-Date: Fri, 22 Mar 2024 17:59:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5323: Support Kudu BINARY
Hello Daniel Becker, Peter Rozsa, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18868 to look at the new patch set (#3). Change subject: WIP IMPALA-5323: Support Kudu BINARY .. WIP IMPALA-5323: Support Kudu BINARY Change-Id: Iff701a4b3a09ce7b6982c5d238e65f3d4f3d1151 --- M be/src/exec/kudu/kudu-table-sink.cc M be/src/exec/kudu/kudu-util-ir.cc M be/src/exec/kudu/kudu-util.cc M be/src/exprs/cast-functions-ir.cc A be/src/exprs/expr-test-common.h M be/src/runtime/types.cc M be/src/runtime/types.h M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M tests/query_test/test_scanners.py 13 files changed, 666 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/18868/3 -- To view, visit http://gerrit.cloudera.org:8080/18868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iff701a4b3a09ce7b6982c5d238e65f3d4f3d1151 Gerrit-Change-Number: 18868 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] WIP IMPALA-5323: Support Kudu BINARY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18868 ) Change subject: WIP IMPALA-5323: Support Kudu BINARY .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10415/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff701a4b3a09ce7b6982c5d238e65f3d4f3d1151 Gerrit-Change-Number: 18868 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Fri, 22 Mar 2024 17:59:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11735: Handle CREATE TABLE event when the db is invisible to the impala server user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21188 ) Change subject: IMPALA-11735: Handle CREATE_TABLE event when the db is invisible to the impala server user .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15631/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90275bb8c065fc5af61186901ac7e9839a68c43b Gerrit-Change-Number: 21188 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 17:33:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11735: Handle CREATE TABLE event when the db is invisible to the impala server user
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21188 Change subject: IMPALA-11735: Handle CREATE_TABLE event when the db is invisible to the impala server user .. IMPALA-11735: Handle CREATE_TABLE event when the db is invisible to the impala server user It's possible that some dbs are invisible to Impala cluster due to authorization restrictions. However, the CREATE_TABLE events in such dbs will lead the event-processor into ERROR state. Event processor should ignore such CREAT_TABLE events when database is not found. Testing: - Manually verified this on local cluster. - Added automated unit test to verify the same. Change-Id: I90275bb8c065fc5af61186901ac7e9839a68c43b --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 21 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/21188/1 -- To view, visit http://gerrit.cloudera.org:8080/21188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I90275bb8c065fc5af61186901ac7e9839a68c43b Gerrit-Change-Number: 21188 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15630/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 16:39:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21179 ) Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15629/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 16:32:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Hello Quanlong Huang, Daniel Becker, Riza Suminto, Gabor Kaszab, Zihao Ye, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21148 to look at the new patch set (#6). Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala Impala generates segmentation fault when it queries the virtual column FILE__POSITION for TEXT or JSON tables. When the scanners that do not support the FILE__POSITION virtual column detect its presence they try to report an error and close themselves. The segfault is in the scanners' Close() method when they try to dereference a NULL stream object. This patch simply adds NULL-checks in Close(). Alternatively we could detect the presence of FILE__POSITION during planning in the HdfsScanNode, but doing it in the scanners lets us handle more queries, e.g. queries that dynamically prune partitions and the surviving partitions all have file formats that support FILE__POSITION. Testing: * added negative tests to properly report the errors * added tests for mixed file format tables Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 --- M be/src/exec/json/hdfs-json-scanner.cc M be/src/exec/text/hdfs-text-scanner.cc M testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-generic.test A testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-negative.test M tests/query_test/test_scanners.py 5 files changed, 94 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/21148/6 -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10414/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 16:17:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21179 ) Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. Patch Set 4: Instead of using rand() I switched to rand(SEED) as the seed-generation seems to be system-specific. -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 16:10:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Hello Daniel Becker, Gabor Kaszab, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21179 to look at the new patch set (#4). Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table The following query throws an error for Iceberg tables: select * from ice_tbl where rand() < 0.001; It's because the predicate 'rand() < 0.001' doesn't involve any table columns. Because of a bug in IcebergScanPlanner.hasPartitionTransformType() the method throws an IndexOutOfBoundsException. This patch fixes the method to handle such predicates. Testing: * added e2e tests Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 --- M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 2 files changed, 106 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/21179/4 -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21179 ) Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10413/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 16:09:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21187 ) Change subject: IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15628/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I83dc32c8d0fec1cdfdfe06e720561b2ae1adf5df Gerrit-Change-Number: 21187 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 15:59:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21187 Change subject: IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE .. IMPALA-12928: Mask JDBC table property dbcp.password for DESC FORMATTED and SHOW CREATE TABLE 'desc formatted' and 'show create table' commands show all of table properties in clear text. For external JDBC table, dbcp.password table property value should be masked in the output of these two commands. This patch makes dbcp.password property value been masked in the output of 'desc formatted' and 'show create table' commands. Testing: - Added end-to-end test cases. - Passed core tests. Change-Id: I83dc32c8d0fec1cdfdfe06e720561b2ae1adf5df --- M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/util/HiveMetadataFormatUtils.java M tests/query_test/test_ext_data_sources.py 4 files changed, 52 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/21187/1 -- To view, visit http://gerrit.cloudera.org:8080/21187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I83dc32c8d0fec1cdfdfe06e720561b2ae1adf5df Gerrit-Change-Number: 21187 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10412/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:39:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15627/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:39:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:39:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12611: Add support to MAP type Iceberg Metadata table columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21125 ) Change subject: IMPALA-12611: Add support to MAP type Iceberg Metadata table columns .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15626/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a8b3a574ca45c893315c3b41b33ce4e0eff865a Gerrit-Change-Number: 21125 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:36:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:21:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10408/ -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:25:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21179 ) Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10409/ -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:23:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Daniel Becker has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator On clusters with dedicated coordinators and executors the Iceberg metadata scanner fragment(s) can be scheduled to executors, for example during a join. The fragment in this case will fail a precondition check, because either the 'frontend_' object or the table will not be present. This change forces Iceberg metadata scanner fragments to be scheduled on the coordinator. It is not enough to set the DataPartition type to UNPARTITIONED, because unpartitioned fragments can still be scheduled on executors. This change introduces a new flag in the TPlanFragment thrift struct - if it is true, the fragment is always scheduled on the coordinator. Testing: - Added a regression test in test_coordinators.py. - Added a new planner test with two metadata tables and a regular table joined together. Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e --- M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler.cc M common/thrift/Planner.thrift M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-joined-with-regular-table.test M tests/custom_cluster/test_coordinators.py 9 files changed, 175 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/21138/4 -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21138/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java File fe/src/main/java/org/apache/impala/planner/PlanFragment.java: http://gerrit.cloudera.org:8080/#/c/21138/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@192 PS3, Line 192: // Coordinator-only fragments must be unpartitined as there is only one instance of : // them. > Could you please add a comment for this? Done -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:15:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12611: Add support to MAP type Iceberg Metadata table columns
Daniel Becker has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21125 ) Change subject: IMPALA-12611: Add support to MAP type Iceberg Metadata table columns .. IMPALA-12611: Add support to MAP type Iceberg Metadata table columns This change adds support for querying MAP types from Iceberg Metadata tables. The 'IcebergMetadataScanner.ArrayScanner' java class is renamed to 'CollectionScanner' and extended to be able to handle maps. For arrays the iteration returns the element as before, for maps it returns 'Map.Entry' objects. Note that collections in the FROM clause are still not supported. Testing: - Added E2E tests in iceberg-metadata-tables.test. Change-Id: I8a8b3a574ca45c893315c3b41b33ce4e0eff865a --- M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 6 files changed, 363 insertions(+), 123 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/21125/3 -- To view, visit http://gerrit.cloudera.org:8080/21125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8a8b3a574ca45c893315c3b41b33ce4e0eff865a Gerrit-Change-Number: 21125 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 4: Added a planner test also. -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 15:15:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 ) Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a Gerrit-Change-Number: 21178 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 22 Mar 2024 14:17:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21178 ) Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode .. IMPALA-12929: Skip loading HDFS permissions in local-catalog mode HDFS file/dir permissions are not used at all in local catalog mode - in LocalFsTable, hasWriteAccessToBaseDir() always returns true and getFirstLocationWithoutWriteAccess() always returns null. However, in catalogd, we still load them (in single thread for a table!) which could dominant the table loading time when there are lots of partitions. Note that the table loading process in catalogd is the same no matter what catalog mode is in used. The difference between catalog modes is mainly in how coordinators get metadata from catalogd. Local catalog mode is turned on by setting --catalog_topic_mode=minimal on catalogd and --use_local_catalog=true on coordinators. This patch skips loading HDFS permissions on catalogd when running in local catalog mode. We can revisit it in IMPALA-7539. Tests: - Ran CORE tests Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a Reviewed-on: http://gerrit.cloudera.org:8080/21178 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java 2 files changed, 8 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a Gerrit-Change-Number: 21178 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] WIP IMPALA-12933: Avoid fetching unneccessary event types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: WIP IMPALA-12933: Avoid fetching unneccessary event types .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15625/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 14:11:51 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12933: Avoid fetching unneccessary event types
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21186 Change subject: WIP IMPALA-12933: Avoid fetching unneccessary event types .. WIP IMPALA-12933: Avoid fetching unneccessary event types There are several places that catalogd will fetch all events of a specifit type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table. Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetches all events and filter them locally based on the event type and table name. This could take hours if there are lots of events (e.g 1M) in HMS. This patch sets the eventTypeSkipList with the complement set of the wanted type. So the get_next_notification RPC can filter out some events on HMS side. Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 --- M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 128 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/1 -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] WIP IMPALA-12933: Avoid fetching unneccessary event types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21186 ) Change subject: WIP IMPALA-12933: Avoid fetching unneccessary event types .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10411/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9 Gerrit-Change-Number: 21186 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 22 Mar 2024 13:50:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10410/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 13:19:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 13:19:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21138 ) Change subject: IMPALA-12809: Iceberg metadata table scanner should always be scheduled to the coordinator .. Patch Set 3: (1 comment) Just quickly went over the code. Looks good overall, but could you please add planner tests? http://gerrit.cloudera.org:8080/#/c/21138/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java File fe/src/main/java/org/apache/impala/planner/PlanFragment.java: http://gerrit.cloudera.org:8080/#/c/21138/3/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@192 PS3, Line 192: Preconditions.checkState(!coordinatorOnly || : dataPartition_.equals(DataPartition.UNPARTITIONED)); Could you please add a comment for this? -- To view, visit http://gerrit.cloudera.org:8080/21138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4397f64e9def42d2b84ffd7bc14ff31df27d58e Gerrit-Change-Number: 21138 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 10:54:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12856: Event processor should ignore processing partition with empty partition values
k.venureddy2...@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/21143 ) Change subject: IMPALA-12856: Event processor should ignore processing partition with empty partition values .. Patch Set 4: (1 comment) Have created HIVE-28145 for hive. http://gerrit.cloudera.org:8080/#/c/21143/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21143/3//COMMIT_MSG@7 PS3, Line 7: IMPALA-12856: Event processor should ignore processing partition > Thank Venu for digging into the HMS bug! Do we have a HIVE JIRA for this? I Have created a hive jira(HIVE-28145) -- To view, visit http://gerrit.cloudera.org:8080/21143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2469930ccd74948325f1723bd8b2bd6aad02d09 Gerrit-Change-Number: 21143 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Mar 2024 10:37:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12898: Tidy up test dimensions of test scanner.py
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21162 ) Change subject: IMPALA-12898: Tidy up test dimensions of test_scanner.py .. Patch Set 3: Code-Review+2 Thanks for applying the changes! LGTM! -- To view, visit http://gerrit.cloudera.org:8080/21162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5efd2b483338fb55b958d8e1a0acf6b365f8093e Gerrit-Change-Number: 21162 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 10:35:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21179 ) Change subject: IMPALA-12879: Conjunct not referring to table field causes ERROR for Iceberg table .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id43a6798df3f4cc3a0e00ac610e25aa3b5781342 Gerrit-Change-Number: 21179 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 22 Mar 2024 10:30:33 + Gerrit-HasComments: No
