[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10310/ The failure is due to IMPALA-12621 which is unrelated to this patch. Rerun the test. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 07:24:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10312/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 07:24:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 07:24:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21029 ) Change subject: IMPALA-12543: Detect self-events before finishing DDL .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15333/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 Gerrit-Change-Number: 21029 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 06:19:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21029 )
Change subject: IMPALA-12543: Detect self-events before finishing DDL
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21029/5/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/21029/5/fe/src/main/java/org/apache/impala/catalog/Table.java@1094
PS5, Line 1094:
> It look confusing to have resetInProgressTableModification and resetInProgr
I just understand what you mean.
Changed the modification tracking to use InProgressTableModification class in
ps5.
http://gerrit.cloudera.org:8080/#/c/21029/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/21029/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1107
PS5, Line 1107: }
: }
> It would be nice to simplify this. Could it work like first setting Infligh
Done
http://gerrit.cloudera.org:8080/#/c/21029/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5253
PS5, Line 5253:
> Is it necessary to increase the catalog version of the table? It DROP PARTI
This follows old code where table is returned as refreshedTable.
New version number then applied if refreshedTable is not null.
Initially, I did not do this, and some test related to drop partition failed.
http://gerrit.cloudera.org:8080/#/c/21029/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6368
PS5, Line 6368: } catch (TException e) {
: throw new ImpalaRuntimeException(
: String.format(HMS_RPC_ERROR_FORMAT_
> Why are inflight events not handled in the transactional case?
ps5 move the try scope to cover alterTableWithTransaction as well.
--
To view, visit http://gerrit.cloudera.org:8080/21029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79
Gerrit-Change-Number: 21029
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Tue, 27 Feb 2024 05:59:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10310/ -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 05:59:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Hello Quanlong Huang, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21029 to look at the new patch set (#6). Change subject: IMPALA-12543: Detect self-events before finishing DDL .. IMPALA-12543: Detect self-events before finishing DDL test_iceberg_self_events has been flaky for not having tbls_refreshed_before equal to tbls_refreshed_after in-between query executions. This flakiness happens in ARM, TEST_JDK_VERSION 11, and TEST_JDK_VERSION 17 environments. Further investigation reveals concurrency bug due to db/table level locks is not taken during db/table self-events check (IMPALA-12461 part1). The order of ALTER TABLE operation is as follow: 1. alter table starts in CatalogOpExecutor 2. table level lock is taken 3. HMS RPC starts (CatalogOpExecutor.applyAlterTable()) 4. HMS generates the event 5. HMS RPC returns 6. table is reloaded 7. catalog version is added to inflight event list 8. table level lock is releases Meanwhile the event processor thread fetches the new event after 4 and before 7. Because of IMPALA-12461 (part 1), it can also finish self-events checking before reaching 7. Before IMPALA-12461, self-events would have needed to wait for 8. Note that this issue is only relevant for table level events, as self-events checking for partition level events still takes table lock. This patch fix the issue by adding newCatalogVersion to the table's inflight event list before updating HMS. If HMS update does not complete (ie., an exception is thrown), the new newCatalogVersion that was added is then removed. This patch also fix few smaller issues, including: - Avoid incrementing EVENTS_SKIPPED_METRIC if numFilteredEvents == 0 in MetastoreEventFactory.getFilteredEvents(). - Increment EVENTS_SKIPPED_METRIC in MetastoreTableEvent.reloadTableFromCatalog() if table is already in the middle of reloading (revealed through flaky test_skipping_older_events). - Rephrase misleading log message in MetastoreEventProcessor.getNextMetastoreEvents(). Testing: - Pass exhaustive tests. Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 5 files changed, 261 insertions(+), 141 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/21029/6 -- To view, visit http://gerrit.cloudera.org:8080/21029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 Gerrit-Change-Number: 21029 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12426: QueryStateRecord Refactor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21059 ) Change subject: IMPALA-12426: QueryStateRecord Refactor .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15332/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57d470db6fea71ec12e43f86e3fd62ed6259c83a Gerrit-Change-Number: 21059 Gerrit-PatchSet: 7 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 27 Feb 2024 05:51:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: QueryStateRecord Refactor
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21059 ) Change subject: IMPALA-12426: QueryStateRecord Refactor .. Patch Set 7: (5 comments) http://gerrit.cloudera.org:8080/#/c/21059/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21059/5//COMMIT_MSG@7 PS5, Line 7: IMPALA-12426: QueryStateRecord Refactor > In my mind this patch should also replace the old QueryStateRecord as part Agreed, it makes more sense as part of this patch. http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/runtime/coordinator.h File be/src/runtime/coordinator.h: http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/runtime/coordinator.h@268 PS5, Line 268: /// Return the backends and their associated resource utilization. > nit: associated Done http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/client-request-state.h@285 PS5, Line 285: /// num_rows_fetched_counter_, row_materialization_rate_, and row_materialization_timer_ > row_materialization_rate_ repeated, I think you meant the last one to be ro Done http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/query-state-record.h File be/src/service/query-state-record.h: http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/query-state-record.h@219 PS5, Line 219: EventsTimelineIterator begin(); > It's a little funny that this is both a container view and an iterator. But Agreed, it's a combination of two patterns because there are two container object underlying the iterator instead of one. http://gerrit.cloudera.org:8080/#/c/21059/5/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/21059/5/common/thrift/Query.thrift@995 PS5, Line 995: 15: optional i64 planner_per_host_mem_estimate; > Can we pull in the Planner.java change that initializes this field? Done -- To view, visit http://gerrit.cloudera.org:8080/21059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57d470db6fea71ec12e43f86e3fd62ed6259c83a Gerrit-Change-Number: 21059 Gerrit-PatchSet: 7 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 27 Feb 2024 05:26:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: QueryStateRecord Refactor
Jason Fehr has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/21059 ) Change subject: IMPALA-12426: QueryStateRecord Refactor .. IMPALA-12426: QueryStateRecord Refactor The QueryStateRecord struct is used to store important information about a completed query for the Impala web UI page of recently completed queries. Since significant portions of this struct has data that is also needed in workload management, it has been refactored. The QueryStateRecord struct was a private child struct under the ImpalaServer class. It has now been moved to a top-level struct within the impala namespace. A new struct named QueryStateExpanded has also been created. This struct contains a shared pointer to a QueryStateRecord so the same QueryStateRecord instance can be used by both the web UI and workload management. The QueryStateExpanded struct also contains additional data that is used exclusively by workload management. New ctests have been added to exercise the added comparators. Change-Id: I57d470db6fea71ec12e43f86e3fd62ed6259c83a --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A be/src/service/query-state-record-test.cc A be/src/service/query-state-record.cc A be/src/service/query-state-record.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/planner/Planner.java 16 files changed, 910 insertions(+), 244 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/21059/7 -- To view, visit http://gerrit.cloudera.org:8080/21059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I57d470db6fea71ec12e43f86e3fd62ed6259c83a Gerrit-Change-Number: 21059 Gerrit-PatchSet: 7 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 04:42:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Reviewed-on: http://gerrit.cloudera.org:8080/21064 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 54 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12834: Add number of concurrent queries to profile
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21063 ) Change subject: IMPALA-12834: Add number of concurrent queries to profile .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8389215b60022b39e7d171d6fc2418acca7c0658 Gerrit-Change-Number: 21063 Gerrit-PatchSet: 3 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Tue, 27 Feb 2024 03:47:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12834: Add number of concurrent queries to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21063 ) Change subject: IMPALA-12834: Add number of concurrent queries to profile .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8389215b60022b39e7d171d6fc2418acca7c0658 Gerrit-Change-Number: 21063 Gerrit-PatchSet: 3 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Tue, 27 Feb 2024 03:30:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10311/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 15 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 03:26:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 15 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 03:26:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12834: Add number of concurrent queries to profile
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21063 ) Change subject: IMPALA-12834: Add number of concurrent queries to profile .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.h@1171 PS2, Line 1171: for the given executor group. > nit: for the given executor group Done http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.cc@189 PS2, Line 189: designate > nit: designated ? Done http://gerrit.cloudera.org:8080/#/c/21063/2/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/21063/2/tests/custom_cluster/test_admission_controller.py@928 PS2, Line 928: "Admission result: Admitted im > nit: indent spaces Done -- To view, visit http://gerrit.cloudera.org:8080/21063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8389215b60022b39e7d171d6fc2418acca7c0658 Gerrit-Change-Number: 21063 Gerrit-PatchSet: 3 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Tue, 27 Feb 2024 03:05:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12834: Add number of concurrent queries to profile
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21063 to look at the new patch set (#3). Change subject: IMPALA-12834: Add number of concurrent queries to profile .. IMPALA-12834: Add number of concurrent queries to profile This patch adds profile info string for the number of current running queries of the executor group on which the query is scheduled, to diagnose potential performance issues due to resource limit. Testing: - Add an e2e test to verify the information appears in profile Change-Id: I8389215b60022b39e7d171d6fc2418acca7c0658 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M tests/custom_cluster/test_admission_controller.py 3 files changed, 43 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/21063/3 -- To view, visit http://gerrit.cloudera.org:8080/21063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8389215b60022b39e7d171d6fc2418acca7c0658 Gerrit-Change-Number: 21063 Gerrit-PatchSet: 3 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
IMPALA-12793: Create JDBC table without data source
This patch changes syntax of creating JDBC table statement as
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
(col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
)
[COMMENT 'table_comment']
STORED BY JDBC
TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)
Both "STORED BY JDBC" and "STORED AS JDBC" are acceptable. A table
property '__IMPALA_DATA_SOURCE_NAME' is added to the JDBC table with
value 'impalajdbcdatasource', which is shown in the output of command
'show create table'.
Following required JDBC parameters must be specified as table
properties: database.type, jdbc.url, jdbc.driver, driver.url, and table.
Otherwise, AnalysisException will be thrown.
Testing:
- Added frontend unit tests for new syntax of creating JDBC table.
- Updated end-to-end unit tests to create JDBC tables without data
source.
- Passed core tests
Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Reviewed-on: http://gerrit.cloudera.org:8080/21016
Reviewed-by: Joe McDonnell
Tested-by: Impala Public Jenkins
---
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
A fe/src/main/java/org/apache/impala/util/JsonUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M testdata/bin/create-ext-data-source-table.sql
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M
testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test
M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
M
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_ext_data_sources.py
25 files changed, 476 insertions(+), 323 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 02:39:58 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15330/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 01:51:54 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Sai Hemanth Gantasala has uploaded a new patch set (#8) to the change originally created by k.venureddy2...@gmail.com. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. [WIP]IMPALA-12832: Implicit invalidate metadata on event failures At present, failure in event processing needs manual invalidate metadata. This patch implicitly invalidates the table upon failures in processing of table events with new 'process_event_failure' flag. And a new 'auto_global_invalidate_metadata' flag is added to global invalidate metadata automatically when event processor goes to non-active state. Note: Also introduced a config 'inject_process_event_failure_event_types' for automated tests to simulate event processor failures. This config is used to specify what event types can be intentionally failed. This config should only be used for testing purpose. Testing: - Added end-to-end tests to mimic failures in event processor and verified that event processor is active - Passed FE tests Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/common/impala_test_suite.py A tests/metadata/__init__.py M tests/metadata/test_event_processing.py A tests/metadata/test_event_processing_base.py A tests/metadata/test_event_processing_error.py M tests/util/event_processor_utils.py 15 files changed, 663 insertions(+), 311 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/21065/8 -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Code-Review+2 Carry +2 from Csaba -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 01:23:53 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 8: (4 comments) http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/__init__.py File tests/metadata/__init__.py: http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/__init__.py@1 PS8, Line 1: flake8: W292 no newline at end of file http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/test_event_processing_base.py File tests/metadata/test_event_processing_base.py: http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/test_event_processing_base.py@348 PS8, Line 348: flake8: W292 no newline at end of file http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/test_event_processing_error.py File tests/metadata/test_event_processing_error.py: http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/test_event_processing_error.py@30 PS8, Line 30: @SkipIfCatalogV2.hms_event_polling_disabled() flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/21065/8/tests/metadata/test_event_processing_error.py@64 PS8, Line 64: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 01:26:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 27 Feb 2024 01:25:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10310/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 01:24:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21074 to look at the new patch set (#3). Change subject: IMPALA-12602: Unregister queries on idle timeout .. IMPALA-12602: Unregister queries on idle timeout Queries cancelled due to idle_query_timeout/QUERY_TIMEOUT_S are now also Unregistered to free any remaining memory, as you cannot fetch results from a cancelled query. Adds a new structure - idle_query_statuses_ - to retain Status messages for queries closed this way so that we can continue to return a clear error message if the client returns and requests query status or attempts to fetch results. SessionState tracks queries added to idle_query_statuses_ so they can be cleared when the session is closed. Testing: updates test_query_expiration to verify number of waiting queries is only non-zero for queries cancelled by EXEC_TIME_LIMIT_S and not yet closed as an idle query. Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M docs/topics/impala_timeouts.xml M tests/custom_cluster/test_query_expiration.py 5 files changed, 102 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/21074/3 -- To view, visit http://gerrit.cloudera.org:8080/21074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 Gerrit-Change-Number: 21074 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21074 to look at the new patch set (#2). Change subject: IMPALA-12602: Unregister queries on idle timeout .. IMPALA-12602: Unregister queries on idle timeout Queries cancelled due to idle_query_timeout/QUERY_TIMEOUT_S are now also Unregistered to free any remaining memory, as you cannot fetch results from a cancelled query. Adds a new structure - idle_query_statuses_ - to retain Status messages for queries closed this way so that we can continue to return a clear error message if the client returns and requests query status or attempts to fetch results. SessionState tracks queries added to idle_query_statuses_ so they can be cleared when the session is closed. Testing: updates test_query_expiration to verify number of waiting queries is only non-zero for queries cancelled by EXEC_TIME_LIMIT_S and not yet closed as an idle query. Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M docs/topics/impala_timeouts.xml M tests/custom_cluster/test_query_expiration.py 5 files changed, 102 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/21074/2 -- To view, visit http://gerrit.cloudera.org:8080/21074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 Gerrit-Change-Number: 21074 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21074 ) Change subject: IMPALA-12602: Unregister queries on idle timeout .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21074/1/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/21074/1/be/src/service/impala-server.cc@2998 PS1, Line 2998: // Save status so we can report it for unregistered queries. We could choose to do this for all queries cancelled in ExpireQueries. I think it makes sense, but that change has more surface error it might affect. Theoretically any query cancelled via ExpireQuery can no longer have results fetched, and the only interaction a client can have is to view the query's EXCEPTION status. That would more clearly delineate that anything in the "waiting to be closed" section of inflight queries is because a client is still fetching completed results (shouldn't see any queries in the EXCEPTION state in that section). -- To view, visit http://gerrit.cloudera.org:8080/21074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 Gerrit-Change-Number: 21074 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 27 Feb 2024 00:43:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21074 Change subject: IMPALA-12602: Unregister queries on idle timeout .. IMPALA-12602: Unregister queries on idle timeout Queries cancelled due to idle_query_timeout/QUERY_TIMEOUT_S are now also Unregistered to free any remaining memory, as you cannot fetch results from a cancelled query. Adds a new structure to retain Status messages for queries closed this way so that we can continue to return a clear error message if the client returns and requests query status or attempts to fetch results. Testing: updates test_query_expiration to verify number of waiting queries is only non-zero for queries cancelled by EXEC_TIME_LIMIT_S and not yet closed as an idle query. Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M docs/topics/impala_timeouts.xml M tests/custom_cluster/test_query_expiration.py 5 files changed, 102 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/21074/1 -- To view, visit http://gerrit.cloudera.org:8080/21074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 Gerrit-Change-Number: 21074 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21074 ) Change subject: IMPALA-12602: Unregister queries on idle timeout .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/21074/1/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/21074/1/be/src/service/impala-server.cc@2934 PS1, Line 2934: // Query was expired already from a previous expiration event. Keep idle timeouts line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/21074/1/tests/custom_cluster/test_query_expiration.py File tests/custom_cluster/test_query_expiration.py: http://gerrit.cloudera.org:8080/#/c/21074/1/tests/custom_cluster/test_query_expiration.py@32 PS1, Line 32: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/21074/1/tests/custom_cluster/test_query_expiration.py@32 PS1, Line 32: flake8: E251 unexpected spaces around keyword / parameter equals -- To view, visit http://gerrit.cloudera.org:8080/21074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0 Gerrit-Change-Number: 21074 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 27 Feb 2024 00:41:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15329/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 11 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 00:31:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15328/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 00:30:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 11: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/21015/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21015/11//COMMIT_MSG@10 PS11, Line 10: The timestamp is of the ISO 8601 format(SQL standard): : '-mm-dd hh:mm:ss.ms'. It returns the timestamp as : string to the remote database. The description could be improved. Maybe, something like following: "Binary SCAN predicates involving timestamp literals are pushed down to remote Database. The current logic assumes ISO 8601 (SQL standard) format for timestamp literals - '-mm-dd hh:mm:ss.ms'" http://gerrit.cloudera.org:8080/#/c/21015/11//COMMIT_MSG@19 PS11, Line 19: - Added coverage for timestamp with/without milliseconds in timestamp. Also, include: "Added Planner tests to check predicate pushdown for date/timestamp literals, date/timestamp functions and CASTs" -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 11 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 00:19:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10309/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 00:08:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 00:08:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: Code-Review+2 LGTM. Carry Michael's +1. Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 27 Feb 2024 00:07:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#11). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 460 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/11 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 11 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 10: Extended planner test to include cast() and to_timestamp() functions. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 27 Feb 2024 00:04:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#10). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 460 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/10 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 10 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 14: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10306/ -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 23:58:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 22:33:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 22:24:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15327/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 22:24:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 7: Thanks Joe for your suggestion to change the syntax for JDBC table creation. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 22:08:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10308/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 22:09:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 7: Code-Review+2 Thanks, this looks good to me -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 22:06:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py@a102 PS6, Line 102: : : : : : : > Yes, will put the unit-test in this patch, and remove it in other patch. Brought it back in patch set #7. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 22:00:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/21016 )
Change subject: IMPALA-12793: Create JDBC table without data source
..
IMPALA-12793: Create JDBC table without data source
This patch changes syntax of creating JDBC table statement as
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
(col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
)
[COMMENT 'table_comment']
STORED BY JDBC
TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)
Both "STORED BY JDBC" and "STORED AS JDBC" are acceptable. A table
property '__IMPALA_DATA_SOURCE_NAME' is added to the JDBC table with
value 'impalajdbcdatasource', which is shown in the output of command
'show create table'.
Following required JDBC parameters must be specified as table
properties: database.type, jdbc.url, jdbc.driver, driver.url, and table.
Otherwise, AnalysisException will be thrown.
Testing:
- Added frontend unit tests for new syntax of creating JDBC table.
- Updated end-to-end unit tests to create JDBC tables without data
source.
- Passed core tests
Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
---
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
A fe/src/main/java/org/apache/impala/util/JsonUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M testdata/bin/create-ext-data-source-table.sql
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M
testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test
M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
M
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/query_test/test_ext_data_sources.py
25 files changed, 476 insertions(+), 323 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/21016/7
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@14 PS3, Line 14: aborted > typo Done http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@22 PS3, Line 22: verifi > nit: verified that Done http://gerrit.cloudera.org:8080/#/c/21071/3/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/21071/3/tests/custom_cluster/test_events_custom_configs.py@1191 PS3, Line 1191: # > flake8: E265 block comment should start with '# ' Done -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 21:58:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21071 to look at the new patch set (#4). Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up HdfsTable tracks the ValidWriteIdList from HMS. When the table is reloaded, the ValidWriteIdList is updated to the latest state. An ABORT_TXN event that is lagging behind could match to aborted write ids that have already been cleaned up by the HMS housekeeping thread. Such write ids can't be found in the cached ValidWriteIdList as opened or aborted write ids. This hits a Precondition check and fails the event processing. This patch fixes the check to allow this case. Also adds more logs for dealing with write ids. Tests - Add custom-cluster test to start Hive with the housekeeping thread turned on and verified that such ABORT_TXN event is processed correctly. Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 --- M bin/create-test-configuration.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java M fe/src/test/resources/hive-site.xml.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 84 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/4 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py@a102 PS6, Line 102: : : : : : : > Do you mind if we remove the test in a different code change? Yes, will put the unit-test in this patch, and remove it in other patch. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 21:47:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py@a102 PS6, Line 102: : : : : : : > I am fine with removing the test, but I don't see a direct connection betwe Do you mind if we remove the test in a different code change? -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 21:44:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15325/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 21:08:37 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15324/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 21:01:11 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10307/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 20:52:49 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21065 to look at the new patch set (#7). Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. [WIP]IMPALA-12832: Implicit invalidate metadata on event failures At present, failure in event processing needs manual invalidate metadata. This patch implicitly invalidates the table upon failures in processing of table events with new process_event_failure flag. And a new auto_global_invalidate_metadata flag is added to global invalidate metadata automatically when event processor goes to non-active state. Testing: - Passed FE tests Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java 9 files changed, 199 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/21065/7 -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21065 to look at the new patch set (#6). Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. [WIP]IMPALA-12832: Implicit invalidate metadata on event failures At present, failure in event processing needs manual invalidate metadata. This patch implicitly invalidates the table upon failures in processing of table events with new process_event_failure flag. And a new auto_global_invalidate_metadata flag is added to global invalidate metadata automatically when event processor goes to non-active state. Testing: - Passed FE tests Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java 9 files changed, 200 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/21065/6 -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15323/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 20:31:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15322/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 20:21:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#9). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 446 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/9 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15321/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 20:00:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java@551 PS3, Line 551:*catalog object that omit some field. > typo: omit Done -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:55:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 4: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java@551 PS3, Line 551:*catalog object that omit some field. > typo: omit Done -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:55:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21064 to look at the new patch set (#4). Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 54 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21064/4 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/21064/3/fe/src/main/java/org/apache/impala/catalog/Catalog.java@551 PS3, Line 551:*catalog object that ommit some field. typo: omit -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:51:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py@a102 PS6, Line 102: : : : : : : > In IMPALA-12833, we will set default value of flag variable 'catalogd_ha_r I am fine with removing the test, but I don't see a direct connection between this test and the rest of this code change. Why are we removing it in this particular code change? If IMPALA-12833 is making it obsolete, then maybe IMPALA-12833 can remove it. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 19:51:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java@709 PS2, Line 709:* Variant of toThrift() with lower detail. Intended to get a human readable output > nit: This is also returned in a user-facing API. Debugging to me implies de Done http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@404 PS2, Line 404:* slimmer than the one obtained from getCatalogObject() because binary data fields > typo: slimmer Done -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:35:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21064 to look at the new patch set (#3). Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject TestReusePartitions::test_reuse_partitions_transactional calls /catalog_object path of CatalogD's WebUI and decodes the JSON response. The "json_string" field from the response text often contains Unicode control characters that come from serialized binary data from THdfsFileDesc objects. That causes JSON decoding to fail with an error like this: ValueError: Invalid control character at: line 1 column 1850 (char 1849) This patch attempts to deflake the test by tuning the return value of getJsonCatalogObject() that excludes THdfsFileDesc, by lowering the detail level from ThriftObjectType.FULL to ThriftObjectType.DESCRIPTOR_ONLY. test_reuse_partitions_transactional is tweaked a bit to print the response / JSON object if an assertion fails. Testing: - Loop and pass the test for hundred times. Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/metadata/test_reuse_partitions.py 6 files changed, 54 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/21064/3 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py@a102 PS6, Line 102: : : : : : : > Nit: Is this test now obsolete? In IMPALA-12833, we will set default value of flag variable 'catalogd_ha_reset_metadata_on_failover' as true. After that change, this test is obsolete. Also this unit-test is a little flaky so I would like to remove it. -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 19:22:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10306/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:21:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 14 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:21:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15320/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 13 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:10:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 13 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 19:02:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 13: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 13 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 18:59:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12610: Support reading ARRAY columns for Iceberg Metadata tables
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21061 ) Change subject: IMPALA-12610: Support reading ARRAY columns for Iceberg Metadata tables .. Patch Set 3: (4 comments) Finished the C++ part, will continue with the Java part later. http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-row-reader.cc File be/src/exec/iceberg-metadata/iceberg-row-reader.cc: http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-row-reader.cc@194 PS3, Line 194: DCHECK(slot_desc != nullptr); Could also DCHECK that 'slot_desc' is a collection. http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-row-reader.cc@203 PS3, Line 203: array_size 'remaining_array_size' could be better because we're decreasing it in the loop at L228. http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-row-reader.cc@205 PS3, Line 205: array_size I think writing "[remaining]_array_size > 0" is easier to understand as it is more explicit. Also on L216. http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-row-reader.cc@207 PS3, Line 207: num_rows 'num_rows' could be misleading, IIANM it is the number of elements in the array, not the number of rows in a table. CollectionValueBuilder::GetFreeMemory() also uses 'num_tuples'. -- To view, visit http://gerrit.cloudera.org:8080/21061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb9bac1822a17bd3cd074b4b98e4d010703cecb1 Gerrit-Change-Number: 21061 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Feb 2024 18:48:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 13: (2 comments) http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/network-util.h File be/src/util/network-util.h: http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/network-util.h@103 PS12, Line 103: /// by port (numerically) and finally by uds address (alphabetically). > This description is no longer accurate. Now goes host > port > uds. Done http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/string-util.h File be/src/util/string-util.h: http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/string-util.h@99 PS12, Line 99: /// enable comma delimited string building where the last instance of the comma needs to > typo: delimited Done -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 13 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 18:44:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Hello Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21048 to look at the new patch set (#13). Change subject: IMPALA-12426: Workload Management Supporting Changes .. IMPALA-12426: Workload Management Supporting Changes Contains several disparate pieces of functionality that support the overall workload management work. 1. TNetworkAddressComparator An unused existing comparator for the Thrift class TNetworkAddress already existed in the thrift-util.h file. This comparator has been moved to the network-util.h file where it now resides in the same place as other utility functions that operator on TNetworkAddress instances. The existing comparator did not consider the uds address. It only considered hostname and port. The new comparator considers all three. Testing is accomplished by porting the existing ctests and adding additional ctests. 2. StringStreamPop This new class extends a std::basic_stringstream to add a function that enables removing a character from the end. Testing is accomplished using new ctests. 3. Ticker This new header-only class notifies a condition variable at periodic intervals. It is a lightweight that sleeps until the configured duration has passed at which point it wakes up and notifies the condition variable. It also enables consumers to offload spurious wakeup guards to this class. Ctests have been added to test the functionality of this new class. 4. TUniqueId Empty Utility Function A new function UUIDEmpty returns true if a provided TUniqueID does not contain a UUID or false otherwise. Ctests have been added to test this new function. 5. run_clang_tidy.sh Additional informational outputs have been added to this script to enable tracking the status of the script and to easily identify errors found by clang tidy. 6. Summary Util Test A ctest was developed for testing the text table generation code that generates the exec summary portion of the query profile. This ctest was developed as part of an idea that did not ultimately pan out. Rather than throwing away that test code, it has been added as a new ctest. Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 --- M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/util/CMakeLists.txt A be/src/util/network-util-test.cc M be/src/util/network-util.cc M be/src/util/network-util.h M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h A be/src/util/summary-util-test.cc A be/src/util/ticker-test.cc A be/src/util/ticker.h M be/src/util/uid-util-test.cc M be/src/util/uid-util.h M bin/run_clang_tidy.sh 16 files changed, 647 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/21048/13 -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 13 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21016 ) Change subject: IMPALA-12793: Create JDBC table without data source .. Patch Set 6: (1 comment) One small thing, but otherwise this looks good to me http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21016/6/tests/custom_cluster/test_ext_data_sources.py@a102 PS6, Line 102: : : : : : : Nit: Is this test now obsolete? If we think we want to remove the persisted data source, we can do that in a separate change (though I don't think we need to remove it). -- To view, visit http://gerrit.cloudera.org:8080/21016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920 Gerrit-Change-Number: 21016 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 18:30:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 3: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@14 PS3, Line 14: aboorted typo http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@22 PS3, Line 22: verify nit: verified that -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 18:07:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12834: Add number of concurrent queries to profile
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21063 ) Change subject: IMPALA-12834: Add number of concurrent queries to profile .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.h@1171 PS2, Line 1171: of the 'grp_name' executor group nit: for the given executor group http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/21063/2/be/src/scheduling/admission-controller.cc@189 PS2, Line 189: scheduled nit: designated ? http://gerrit.cloudera.org:8080/#/c/21063/2/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/21063/2/tests/custom_cluster/test_admission_controller.py@928 PS2, Line 928: nit: indent spaces -- To view, visit http://gerrit.cloudera.org:8080/21063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8389215b60022b39e7d171d6fc2418acca7c0658 Gerrit-Change-Number: 21063 Gerrit-PatchSet: 2 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 26 Feb 2024 17:51:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12825: Install thrift into the impala-python virtualenv
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21046 ) Change subject: IMPALA-12825: Install thrift into the impala-python virtualenv .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21046/1/infra/python/deps/requirements.txt File infra/python/deps/requirements.txt: http://gerrit.cloudera.org:8080/#/c/21046/1/infra/python/deps/requirements.txt@67 PS1, Line 67: thrift==0.16.0 Another place to potentially update Thrift version. Do we need to keep it in-sync? If so we should mention it in impala-config.sh. -- To view, visit http://gerrit.cloudera.org:8080/21046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib36e8a1ce8d446b69b08e81ea458f95c158e28f5 Gerrit-Change-Number: 21046 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 26 Feb 2024 17:45:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15319/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 17:47:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: QueryStateRecord Refactor
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21059 ) Change subject: IMPALA-12426: QueryStateRecord Refactor .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21059/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21059/5//COMMIT_MSG@7 PS5, Line 7: IMPALA-12426: QueryStateRecord Refactor In my mind this patch should also replace the old QueryStateRecord as part of the refactor. -- To view, visit http://gerrit.cloudera.org:8080/21059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57d470db6fea71ec12e43f86e3fd62ed6259c83a Gerrit-Change-Number: 21059 Gerrit-PatchSet: 5 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 26 Feb 2024 17:43:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: QueryStateRecord Refactor
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21059 ) Change subject: IMPALA-12426: QueryStateRecord Refactor .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/runtime/coordinator.h File be/src/runtime/coordinator.h: http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/runtime/coordinator.h@268 PS5, Line 268: /// Return the backends and their associate resource utilization. nit: associated http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/client-request-state.h@285 PS5, Line 285: /// num_rows_fetched_counter_, row_materialization_rate_, and row_materialization_rate_ row_materialization_rate_ repeated, I think you meant the last one to be row_materialization_timer_. http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/query-state-record.h File be/src/service/query-state-record.h: http://gerrit.cloudera.org:8080/#/c/21059/5/be/src/service/query-state-record.h@219 PS5, Line 219: EventsTimelineIterator begin(); It's a little funny that this is both a container view and an iterator. But I don't see a specific problem with that. http://gerrit.cloudera.org:8080/#/c/21059/5/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/21059/5/common/thrift/Query.thrift@995 PS5, Line 995: 15: optional i64 planner_per_host_mem_estimate; Can we pull in the Planner.java change that initializes this field? -- To view, visit http://gerrit.cloudera.org:8080/21059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57d470db6fea71ec12e43f86e3fd62ed6259c83a Gerrit-Change-Number: 21059 Gerrit-PatchSet: 5 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 26 Feb 2024 17:39:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12831: Fix HdfsTable.toMinimalTCatalogObject() failed by concurrent modification
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21072 ) Change subject: WIP IMPALA-12831: Fix HdfsTable.toMinimalTCatalogObject() failed by concurrent modification .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/21072/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/21072/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2133 PS1, Line 2133: // Try adding the partition ids and names if we can acquire the read lock. Would it be hard to synchronize access to partitionMap_? I could be done and in the few places where we add/remove partitions. In other places where we iterate through it the thread safety is already solved somehow I guess. Once the collection is synchronized, I think that there shouldn't be problems, as the partition ID is shouldn't change. http://gerrit.cloudera.org:8080/#/c/21072/1/tests/custom_cluster/test_concurrent_ddls.py File tests/custom_cluster/test_concurrent_ddls.py: http://gerrit.cloudera.org:8080/#/c/21072/1/tests/custom_cluster/test_concurrent_ddls.py@215 PS1, Line 215: handle refresh_handle would be clearer ase there are other operations in parallel -- To view, visit http://gerrit.cloudera.org:8080/21072 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9f8e65c0bd24000241eedf8ca765c1e4e14fdb3 Gerrit-Change-Number: 21072 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 17:30:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12426: Workload Management Supporting Changes
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21048 ) Change subject: IMPALA-12426: Workload Management Supporting Changes .. Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/network-util.h File be/src/util/network-util.h: http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/network-util.h@103 PS12, Line 103: /// uds address (alphabetically) and finally by port (numerically). This description is no longer accurate. Now goes host > port > uds. http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/string-util.h File be/src/util/string-util.h: http://gerrit.cloudera.org:8080/#/c/21048/12/be/src/util/string-util.h@99 PS12, Line 99: /// enable comma delimeted string building where the last instance of the comma needs to typo: delimited -- To view, visit http://gerrit.cloudera.org:8080/21048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee23334ec56a18b192a75d052468bf59159b6424 Gerrit-Change-Number: 21048 Gerrit-PatchSet: 12 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 26 Feb 2024 17:27:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 8: Rebased from latest master. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 17:23:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#8). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 461 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/8 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 7: please rebase and solve the merge conflicts. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 17:10:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 17:08:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21064 ) Change subject: IMPALA-12840: Exclude THdfsFileDesc in getJsonCatalogObject .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/catalog/Table.java@709 PS2, Line 709:* Variant of toThrift() with lower detail. Intended for test/debugging purpose. nit: This is also returned in a user-facing API. Debugging to me implies developer audience, so maybe broaden this a little to human-readable? http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/21064/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@404 PS2, Line 404:* slimer than the one obtained from getCatalogObject() because binary data fields typo: slimmer -- To view, visit http://gerrit.cloudera.org:8080/21064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f6840bf1267d1d99d321c0a6b4a0cab49543182 Gerrit-Change-Number: 21064 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 17:07:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12842: Make node.js installation aware of CPU architecture
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21070 ) Change subject: IMPALA-12842: Make node.js installation aware of CPU architecture .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21070/1/tests/run-js-tests.sh File tests/run-js-tests.sh: http://gerrit.cloudera.org:8080/#/c/21070/1/tests/run-js-tests.sh@32 PS1, Line 32: fi nit: an error message if neither is true would be nice. -- To view, visit http://gerrit.cloudera.org:8080/21070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie22da6237ed6d19cf8af721df471033fdfe6b23a Gerrit-Change-Number: 21070 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Surya Hebbar Gerrit-Comment-Date: Mon, 26 Feb 2024 17:02:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15318/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 17:01:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15317/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 17:00:34 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21065 ) Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15316/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 26 Feb 2024 16:36:39 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-12832: Implicit invalidate metadata on event failures
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21065 to look at the new patch set (#5). Change subject: [WIP]IMPALA-12832: Implicit invalidate metadata on event failures .. [WIP]IMPALA-12832: Implicit invalidate metadata on event failures At present, failure in event processing needs manual invalidate metadata. This patch implicitly invalidates the table upon failures in processing of table events with new process_event_failure flag. And a new auto_global_invalidate_metadata flag is added to global invalidate metadata automatically when event processor goes to non-active state. Testing: - Passed FE tests Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java 9 files changed, 229 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/21065/5 -- To view, visit http://gerrit.cloudera.org:8080/21065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c Gerrit-Change-Number: 21065 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/21015 ) Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. Patch Set 6: Added planner test for both date and timestamp SCAN predicates. -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 6 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 26 Feb 2024 16:35:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12815: Support timestamp for scan predicates for external data source table.
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21015 to look at the new patch set (#7). Change subject: IMPALA-12815: Support timestamp for scan predicates for external data source table. .. IMPALA-12815: Support timestamp for scan predicates for external data source table. The timestamp is of the ISO 8601 format(SQL standard): '-mm-dd hh:mm:ss.ms'. It returns the timestamp as string to the remote database. Testing: - Added custom cluster tests for timestamp predicates with operators: '=', '>', '<', '>=', '<=', '!=', 'BETWEEN' for postgres, mysql and remote impala. - Added coverage for timestamp with/without time in the timestamp - Added coverage for timestamp with/without milliseconds in timestamp. Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test M testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test 5 files changed, 446 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21015/7 -- To view, visit http://gerrit.cloudera.org:8080/21015 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6ffe672b4027e2cee094cec4f99b9df9308e441 Gerrit-Change-Number: 21015 Gerrit-PatchSet: 7 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12610: Support reading ARRAY columns for Iceberg Metadata tables
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/21061 )
Change subject: IMPALA-12610: Support reading ARRAY columns for Iceberg
Metadata tables
..
Patch Set 3:
(9 comments)
Thanks for the patch, Tamas! I checked the tests and the Java part. Will take a
deeper look at the rest tomorrow.
http://gerrit.cloudera.org:8080/#/c/21061/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21061/3//COMMIT_MSG@18
PS3, Line 18: the the
nit: duplicate 'the'
http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h
File be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h:
http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h@21
PS3, Line 21: #include "exec/scan-node.h"
Can we not include scan-node.h?
http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/service/impalad-main.cc
File be/src/service/impalad-main.cc:
http://gerrit.cloudera.org:8080/#/c/21061/3/be/src/service/impalad-main.cc@32
PS3, Line 32: #include "exec/iceberg-metadata/iceberg-metadata-scan-node.h"
This include can be removed I think
http://gerrit.cloudera.org:8080/#/c/21061/3/fe/src/main/java/org/apache/impala/analysis/FromClause.java
File fe/src/main/java/org/apache/impala/analysis/FromClause.java:
http://gerrit.cloudera.org:8080/#/c/21061/3/fe/src/main/java/org/apache/impala/analysis/FromClause.java@169
PS3, Line 169: tblRef.getDesc().getType().isMapType()) {
If I'm not mistaken, this check is for the case when a collection is provided
in the FROM clause. If this use case is deliberately supported then please add
test coverage too.
http://gerrit.cloudera.org:8080/#/c/21061/3/fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java
File fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java:
http://gerrit.cloudera.org:8080/#/c/21061/3/fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java@116
PS3, Line 116: StructLke
typo
http://gerrit.cloudera.org:8080/#/c/21061/3/fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java@137
PS3, Line 137: private Iterator iterator;
If we store the iterator as a member, is there a guarantee that the underlying
array is not destroyed?
http://gerrit.cloudera.org:8080/#/c/21061/3/fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java@146
PS3, Line 146: return iterator.next();
nit: could fit into the line above
http://gerrit.cloudera.org:8080/#/c/21061/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test
File
testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test:
http://gerrit.cloudera.org:8080/#/c/21061/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test@567
PS3, Line 567: # Test 11 : Query ARRAY type columns
The test above is also "Test 11". I don't think that giving a number of these
tests make sense, it just makes it complicated to maintain and to add anything
to the middle.
http://gerrit.cloudera.org:8080/#/c/21061/3/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test@591
PS3, Line 591:
row_regex:'/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_both_eq_and_pos/data/.*','NULL'
The name of the files in this table is fix, so we could spell the full path out
in the expected results, in my opinion.
--
To view, visit http://gerrit.cloudera.org:8080/21061
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieb9bac1822a17bd3cd074b4b98e4d010703cecb1
Gerrit-Change-Number: 21061
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 26 Feb 2024 16:22:30 +
Gerrit-HasComments: Yes
