[Impala-ASF-CR] PROTOTYPE: IMPALA-12905: Disk-based tuple caching
Hello Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21171 to look at the new patch set (#2). Change subject: PROTOTYPE: IMPALA-12905: Disk-based tuple caching .. PROTOTYPE: IMPALA-12905: Disk-based tuple caching This implements storage for the tuple cache using local files. This contains contributions from Michael Smith, Yida Wu, and Joe McDonnell. TODO: Add more detail to this commit message Change-Id: I13a65c4c0559cad3559d5f714a074dd06e9cc9bf --- M be/src/exec/CMakeLists.txt M be/src/exec/tuple-cache-node.cc M be/src/exec/tuple-cache-node.h A be/src/exec/tuple-read-write-test.cc A be/src/exec/tuple-reader.cc A be/src/exec/tuple-reader.h A be/src/exec/tuple-writer.cc A be/src/exec/tuple-writer.h M be/src/runtime/CMakeLists.txt M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h A be/src/runtime/tuple-cache-mgr-test.cc A be/src/runtime/tuple-cache-mgr.cc A be/src/runtime/tuple-cache-mgr.h M bin/start-impala-cluster.py M common/thrift/metrics.json A tests/custom_cluster/test_tuple_cache.py 17 files changed, 1,750 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21171/2 -- To view, visit http://gerrit.cloudera.org:8080/21171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13a65c4c0559cad3559d5f714a074dd06e9cc9bf Gerrit-Change-Number: 21171 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] PROTOTYPE: IMPALA-12905: Disk-based tuple caching
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21171 ) Change subject: PROTOTYPE: IMPALA-12905: Disk-based tuple caching .. Patch Set 1: I have a few more fixes to do, but posting this to get style feedback and run test jobs. -- To view, visit http://gerrit.cloudera.org:8080/21171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13a65c4c0559cad3559d5f714a074dd06e9cc9bf Gerrit-Change-Number: 21171 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 20 Mar 2024 05:42:44 + Gerrit-HasComments: No
[Impala-ASF-CR] PROTOTYPE: IMPALA-12905: Disk-based tuple caching
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21171 Change subject: PROTOTYPE: IMPALA-12905: Disk-based tuple caching .. PROTOTYPE: IMPALA-12905: Disk-based tuple caching This implements storage for the tuple cache using local files. This contains contributions from Michael Smith, Yida Wu, and Joe McDonnell. TODO: Add more detail to this commit message Change-Id: I13a65c4c0559cad3559d5f714a074dd06e9cc9bf --- M be/src/exec/CMakeLists.txt M be/src/exec/tuple-cache-node.cc M be/src/exec/tuple-cache-node.h A be/src/exec/tuple-read-write-test.cc A be/src/exec/tuple-reader.cc A be/src/exec/tuple-reader.h A be/src/exec/tuple-writer.cc A be/src/exec/tuple-writer.h M be/src/runtime/CMakeLists.txt M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h A be/src/runtime/tuple-cache-mgr-test.cc A be/src/runtime/tuple-cache-mgr.cc A be/src/runtime/tuple-cache-mgr.h M bin/start-impala-cluster.py M common/thrift/metrics.json A tests/custom_cluster/test_tuple_cache.py 17 files changed, 1,747 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21171/1 -- To view, visit http://gerrit.cloudera.org:8080/21171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I13a65c4c0559cad3559d5f714a074dd06e9cc9bf Gerrit-Change-Number: 21171 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] PROTOTYPE: IMPALA-12905: Disk-based tuple caching
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/21171 )
Change subject: PROTOTYPE: IMPALA-12905: Disk-based tuple caching
..
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/21171/1/be/src/runtime/tuple-cache-mgr.cc
File be/src/runtime/tuple-cache-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/21171/1/be/src/runtime/tuple-cache-mgr.cc@240
PS1, Line 240: // Insert into the cache. If immediately evicted,
evict_callback_ handles cleanup and
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/21171/1/tests/custom_cluster/test_tuple_cache.py
File tests/custom_cluster/test_tuple_cache.py:
http://gerrit.cloudera.org:8080/#/c/21171/1/tests/custom_cluster/test_tuple_cache.py@27
PS1, Line 27: def table_value(seed):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/21171/1/tests/custom_cluster/test_tuple_cache.py@35
PS1, Line 35: def assertCounters(profile, num_hits, num_halted, num_skipped):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/21171/1/tests/custom_cluster/test_tuple_cache.py@40
PS1, Line 40: class TestTupleCache(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/21171/1/tests/custom_cluster/test_tuple_cache.py@48
PS1, Line 48:
flake8: E201 whitespace after '{'
http://gerrit.cloudera.org:8080/#/c/21171/1/tests/custom_cluster/test_tuple_cache.py@48
PS1, Line 48:
flake8: E202 whitespace before '}'
--
To view, visit http://gerrit.cloudera.org:8080/21171
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I13a65c4c0559cad3559d5f714a074dd06e9cc9bf
Gerrit-Change-Number: 21171
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Wed, 20 Mar 2024 05:42:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21019 ) Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE events with trivial changes in StorageDescriptor .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10392/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac Gerrit-Change-Number: 21019 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 20 Mar 2024 04:14:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12699: Set timeout for catalog RPCs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21146 ) Change subject: IMPALA-12699: Set timeout for catalog RPCs .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10389/ -- To view, visit http://gerrit.cloudera.org:8080/21146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad39a79d0c89f2b04380f610a7e60558429e9c6e Gerrit-Change-Number: 21146 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 20 Mar 2024 04:14:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12802: Support ALTER TABLE for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21088 ) Change subject: IMPALA-12802: Support ALTER TABLE for JDBC tables .. Patch Set 4: Thanks Joe -- To view, visit http://gerrit.cloudera.org:8080/21088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ebb5de2c686d2015db78641f78299dd5f33621e Gerrit-Change-Number: 21088 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 20 Mar 2024 04:07:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12802: Support ALTER TABLE for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21088 ) Change subject: IMPALA-12802: Support ALTER TABLE for JDBC tables .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ebb5de2c686d2015db78641f78299dd5f33621e Gerrit-Change-Number: 21088 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 20 Mar 2024 04:06:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12802: Support ALTER TABLE for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21088 ) Change subject: IMPALA-12802: Support ALTER TABLE for JDBC tables .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10391/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ebb5de2c686d2015db78641f78299dd5f33621e Gerrit-Change-Number: 21088 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 20 Mar 2024 04:06:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20491 ) Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10390/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5 Gerrit-Change-Number: 20491 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Mar 2024 03:16:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21169 ) Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 20 Mar 2024 03:16:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12802: Support ALTER TABLE for JDBC tables
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21088 ) Change subject: IMPALA-12802: Support ALTER TABLE for JDBC tables .. Patch Set 4: Code-Review+2 This looks good to me -- To view, visit http://gerrit.cloudera.org:8080/21088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ebb5de2c686d2015db78641f78299dd5f33621e Gerrit-Change-Number: 21088 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Wed, 20 Mar 2024 03:07:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10387/ -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Mar 2024 01:24:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20491 ) Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15576/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5 Gerrit-Change-Number: 20491 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Mar 2024 01:19:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21167 ) Change subject: IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests .. Patch Set 1: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/21167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd Gerrit-Change-Number: 21167 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 20 Mar 2024 01:10:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20491 ) Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15575/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5 Gerrit-Change-Number: 20491 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Mar 2024 01:12:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20762 )
Change subject: IMPALA-12540: Query Live Table
..
Patch Set 41:
(12 comments)
Will do more thorough review tomorrow. In the meantime, here are some of my
comments.
http://gerrit.cloudera.org:8080/#/c/20762/41//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20762/41//COMMIT_MSG@19
PS41, Line 19:
: Query: explain select * from sys.impala_query_live
Consider adding planner test for queries against sys.impala_query_live.
Also add test that show error if user attempt to create sys.impala_query_live
by themself.
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/exec/system-table-scan-node.h
File be/src/exec/system-table-scan-node.h:
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/exec/system-table-scan-node.h@43
PS41, Line 43: NYI
nit: Not Yet Implemented?
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/exec/system-table-scan-node.cc@90
PS41, Line 90: NYI
nit: spell it out.
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/exec/system-table-scan-node.cc@91
PS41, Line 91: NYI
nit: Replace with
Status(ErrorMsg(TErrorCode::NOT_IMPLEMENTED_ERROR,...));
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/service/impala-server.h@508
PS41, Line 508: /// Return the number of live queries managed by this server.
: size_t NumLiveQueries();
mention that this is implemented in workload-management.cc
http://gerrit.cloudera.org:8080/#/c/20762/41/be/src/service/impala-server.h@1138
PS41, Line 1138: /// Returns a list of completed queries that have not yet been
written to storage.
: std::vector>
GetCompletedQueries();
mention that this is implemented in workload-management.cc
http://gerrit.cloudera.org:8080/#/c/20762/41/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/20762/41/common/thrift/CatalogObjects.thrift@672
PS41, Line 672: QUERY_LIVE
Since QUERY_LIVE name is constant, should this be a const instead?
const string QUERY_LIVE = "impala_query_live"
http://gerrit.cloudera.org:8080/#/c/20762/41/fe/src/main/java/org/apache/impala/catalog/SystemTable.java
File fe/src/main/java/org/apache/impala/catalog/SystemTable.java:
http://gerrit.cloudera.org:8080/#/c/20762/41/fe/src/main/java/org/apache/impala/catalog/SystemTable.java@176
PS41, Line 176:
: private static org.apache.hadoop.hive.metastore.api.Table
: createMetastoreTable(String dbName, String tableName,
String owner) {
This is not registering the table to HMS, isn't it?
http://gerrit.cloudera.org:8080/#/c/20762/41/fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java
File fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java:
http://gerrit.cloudera.org:8080/#/c/20762/41/fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java@121
PS41, Line 121: 1024L * 1024L
SQL, PLAN, and EXEC_SUMMARY can be long. Has this take account for that?
http://gerrit.cloudera.org:8080/#/c/20762/41/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/20762/41/tests/custom_cluster/test_query_live.py@24
PS41, Line 24: class TestQueryLive(CustomClusterTestSuite):
Add test querying sys.impala_query_live over cluster with multiple coordinators.
http://gerrit.cloudera.org:8080/#/c/20762/41/tests/custom_cluster/test_query_live.py@27
PS41, Line 27:
@CustomClusterTestSuite.with_args(impalad_args="--enable_workload_mgmt "
:
"--cluster_id=test_query_live_1",
:
catalogd_args="--enable_workload_mgmt")
Can this and the other test work without workload management enabled?
I guess GetCompletedQueries will return empty list in that case, but live
queries can still be served?
If it does work, please add EE test for that scenario.
http://gerrit.cloudera.org:8080/#/c/20762/41/tests/custom_cluster/test_query_live.py@33
PS41, Line 33: fite
nit: fit?
--
To view, visit http://gerrit.cloudera.org:8080/20762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743
Gerrit-Change-Number: 20762
Gerrit-PatchSet: 41
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Wed, 20 Mar 2024 01:01:29
[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20491 ) Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles .. Patch Set 15: (1 comment) Thanks for the review! Resolved the conflicts. http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1514 PS12, Line 1514: params.getAlter_type()); > How about "Iceberg operations are prepared for commit"? Done -- To view, visit http://gerrit.cloudera.org:8080/20491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5 Gerrit-Change-Number: 20491 Gerrit-PatchSet: 15 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Mar 2024 00:56:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles
Hello Zoltan Borok-Nagy, Zihao Ye, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20491 to look at the new patch set (#15). Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles .. IMPALA-12443: Add catalog timeline for all DDL profiles This is a follow-up work of IMPALA-12024 where we add the catalog timeline for CreateTable statements. Using the same mechanism, this patch adds catalog timeline for all DDL/DML profiles, including REFRESH and INSERT. The goal is to add timeline markers after each step that could be blocked, e.g. acquiring locks, external RPCs. So we can better debug slow DDLs with the catalog timeline in profiles. Tried to add some constant strings for widely used events, e.g. "Fetched table from Metastore". Didn't do so for events that only occurs once. Most of the catalog methods now have a new argument for tracking the execution timeline. To avoid adding null checks everywhere, for code paths that don't need a catalog profile, e.g. EventProcessor, uses a static noop EventSequence as the argument. We can replace it in future works, e.g. expose execution timeline of a slow processing on an HMS event. This patch also removes some unused overloads of HdfsTable#load() and HdfsTable#reloadPartitionsFromNames(). Example timeline for a REFRESH statement on an unloaded table (IncompleteTable): Catalog Server Operation: 2s300ms - Got catalog version read lock: 26.407us (26.407us) - Start loading table: 314.663us (288.256us) - Got Metastore client: 629.599us (314.936us) - Fetched table from Metastore: 7.248ms (6.618ms) - Loaded table schema: 27.947ms (20.699ms) - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms) - Got access level: 1s514ms (588.314us) - Created partition builders: 2s103ms (588.270ms) - Start loading file metadata: 2s103ms (49.760us) - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms) - Async loaded table: 2s289ms (6.931ms) - Loaded table from scratch: 2s289ms (72.038us) - Got table read lock: 2s289ms (2.289us) - Finished resetMetadata request: 2s300ms (10.188ms) Example timeline for an INSERT statement: Catalog Server Operation: 178.120ms - Got catalog version read lock: 4.238us (4.238us) - Got catalog version write lock and table write lock: 52.768us (48.530us) - Got Metastore client: 15.768ms (15.715ms) - Fired Metastore events: 156.650ms (140.882ms) - Got Metastore client: 163.317ms (6.666ms) - Fetched table from Metastore: 166.561ms (3.244ms) - Start refreshing file metadata: 167.961ms (1.399ms) - Loaded file metadata for 24 partitions: 177.679ms (9.717ms) - Reloaded table metadata: 178.021ms (342.261us) - Finished updateCatalog request: 178.120ms (98.929us) Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales": Catalog Server Operation: 6s737ms - Got catalog version read lock: 19.971us (19.971us) - Got catalog version write lock and table write lock: 50.255us (30.284us) - Got Metastore client: 171.819us (121.564us) - Updated column stats: 25.560ms (25.388ms) - Got Metastore client: 69.298ms (43.738ms) - Altered 500 partitions in Metastore: 1s894ms (1s825ms) - Altered 1000 partitions in Metastore: 3s558ms (1s664ms) - Altered 1500 partitions in Metastore: 5s144ms (1s586ms) - Altered 1824 partitions in Metastore: 6s205ms (1s060ms) - Got Metastore client: 6s205ms (329.481us) - Altered table in Metastore: 6s216ms (11.073ms) - Got Metastore client: 6s216ms (13.377us) - Fetched table from Metastore: 6s219ms (2.419ms) - Loaded table schema: 6s223ms (4.130ms) - Got current Metastore event id 19017: 6s639ms (415.690ms) - Start loading file metadata: 6s639ms (9.591us) - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms) - Reloaded table metadata: 6s735ms (5.865ms) - DDL finished: 6s737ms (2.255ms) Example timeline for a global INVALIDATE METADATA: Catalog Server Operation: 301.618ms - Got catalog version write lock: 9.908ms (9.908ms) - Got Metastore client: 9.922ms (14.013us) - Got database list: 11.396ms (1.473ms) - Loaded functions of default: 44.919ms (33.523ms) - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms) - Loaded functions of functional: 50.846ms (3.321ms) - Loaded TableMeta of 101 tables in database functional: 52.580ms (1.734ms) - Loaded functions of functional_avro: 54.861ms (2.281ms) - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms (928.120us) ... - Loaded functions of tpch_text_gzip: 299.503ms (1.710ms) - Loaded TableMeta of 8 tables in database tpch_text_gzip: 300.288ms (784.725us) - Updated catalog cache: 300.366ms (78.045us) - Finished resetMetadata request: 301.618ms (1.251ms) Tests: - Add e2e test to verify the catalog timeline in some DDLs. -
[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles
Hello Zoltan Borok-Nagy, Zihao Ye, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20491 to look at the new patch set (#14). Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles .. IMPALA-12443: Add catalog timeline for all DDL profiles This is a follow-up work of IMPALA-12024 where we add the catalog timeline for CreateTable statements. Using the same mechanism, this patch adds catalog timeline for all DDL/DML profiles, including REFRESH and INSERT. The goal is to add timeline markers after each step that could be blocked, e.g. acquiring locks, external RPCs. So we can better debug slow DDLs with the catalog timeline in profiles. Tried to add some constant strings for widely used events, e.g. "Fetched table from Metastore". Didn't do so for events that only occurs once. Most of the catalog methods now have a new argument for tracking the execution timeline. To avoid adding null checks everywhere, for code paths that don't need a catalog profile, e.g. EventProcessor, uses a static noop EventSequence as the argument. We can replace it in future works, e.g. expose execution timeline of a slow processing on an HMS event. This patch also removes some unused overloads of HdfsTable#load() and HdfsTable#reloadPartitionsFromNames(). Example timeline for a REFRESH statement on an unloaded table (IncompleteTable): Catalog Server Operation: 2s300ms - Got catalog version read lock: 26.407us (26.407us) - Start loading table: 314.663us (288.256us) - Got Metastore client: 629.599us (314.936us) - Fetched table from Metastore: 7.248ms (6.618ms) - Loaded table schema: 27.947ms (20.699ms) - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms) - Got access level: 1s514ms (588.314us) - Created partition builders: 2s103ms (588.270ms) - Start loading file metadata: 2s103ms (49.760us) - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms) - Async loaded table: 2s289ms (6.931ms) - Loaded table from scratch: 2s289ms (72.038us) - Got table read lock: 2s289ms (2.289us) - Finished resetMetadata request: 2s300ms (10.188ms) Example timeline for an INSERT statement: Catalog Server Operation: 178.120ms - Got catalog version read lock: 4.238us (4.238us) - Got catalog version write lock and table write lock: 52.768us (48.530us) - Got Metastore client: 15.768ms (15.715ms) - Fired Metastore events: 156.650ms (140.882ms) - Got Metastore client: 163.317ms (6.666ms) - Fetched table from Metastore: 166.561ms (3.244ms) - Start refreshing file metadata: 167.961ms (1.399ms) - Loaded file metadata for 24 partitions: 177.679ms (9.717ms) - Reloaded table metadata: 178.021ms (342.261us) - Finished updateCatalog request: 178.120ms (98.929us) Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales": Catalog Server Operation: 6s737ms - Got catalog version read lock: 19.971us (19.971us) - Got catalog version write lock and table write lock: 50.255us (30.284us) - Got Metastore client: 171.819us (121.564us) - Updated column stats: 25.560ms (25.388ms) - Got Metastore client: 69.298ms (43.738ms) - Altered 500 partitions in Metastore: 1s894ms (1s825ms) - Altered 1000 partitions in Metastore: 3s558ms (1s664ms) - Altered 1500 partitions in Metastore: 5s144ms (1s586ms) - Altered 1824 partitions in Metastore: 6s205ms (1s060ms) - Got Metastore client: 6s205ms (329.481us) - Altered table in Metastore: 6s216ms (11.073ms) - Got Metastore client: 6s216ms (13.377us) - Fetched table from Metastore: 6s219ms (2.419ms) - Loaded table schema: 6s223ms (4.130ms) - Got current Metastore event id 19017: 6s639ms (415.690ms) - Start loading file metadata: 6s639ms (9.591us) - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms) - Reloaded table metadata: 6s735ms (5.865ms) - DDL finished: 6s737ms (2.255ms) Example timeline for a global INVALIDATE METADATA: Catalog Server Operation: 301.618ms - Got catalog version write lock: 9.908ms (9.908ms) - Got Metastore client: 9.922ms (14.013us) - Got database list: 11.396ms (1.473ms) - Loaded functions of default: 44.919ms (33.523ms) - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms) - Loaded functions of functional: 50.846ms (3.321ms) - Loaded TableMeta of 101 tables in database functional: 52.580ms (1.734ms) - Loaded functions of functional_avro: 54.861ms (2.281ms) - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms (928.120us) ... - Loaded functions of tpch_text_gzip: 299.503ms (1.710ms) - Loaded TableMeta of 8 tables in database tpch_text_gzip: 300.288ms (784.725us) - Updated catalog cache: 300.366ms (78.045us) - Finished resetMetadata request: 301.618ms (1.251ms) Tests: - Add e2e test to verify the catalog timeline in some DDLs. -
[Impala-ASF-CR] IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21167 ) Change subject: IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd Gerrit-Change-Number: 21167 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 20 Mar 2024 00:25:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12699: Set timeout for catalog RPCs
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21146 ) Change subject: IMPALA-12699: Set timeout for catalog RPCs .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad39a79d0c89f2b04380f610a7e60558429e9c6e Gerrit-Change-Number: 21146 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 20 Mar 2024 00:11:05 + Gerrit-HasComments: No
[impala-udf-samples-CR] Resolve linker errors due to uda-sample-test & udf-sample-test
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21170 Change subject: Resolve linker errors due to uda-sample-test & udf-sample-test .. Resolve linker errors due to uda-sample-test & udf-sample-test Updated make file to link uda-sample-test & udf-sample-test properly. Also, updated the default Impala UDF shared library name to ImpalaUdf-retail. Change-Id: I703398294dd719818352d5070297a2f74decdeed --- M CMakeLists.txt 1 file changed, 4 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/impala-udf-samples refs/changes/70/21170/1 -- To view, visit http://gerrit.cloudera.org:8080/21170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: impala-udf-samples Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I703398294dd719818352d5070297a2f74decdeed Gerrit-Change-Number: 21170 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21029 ) Change subject: IMPALA-12543: Detect self-events before finishing DDL .. Patch Set 17: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 Gerrit-Change-Number: 21029 Gerrit-PatchSet: 17 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 19 Mar 2024 23:54:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12699: Set timeout for catalog RPCs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21146 ) Change subject: IMPALA-12699: Set timeout for catalog RPCs .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10389/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21146 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad39a79d0c89f2b04380f610a7e60558429e9c6e Gerrit-Change-Number: 21146 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 19 Mar 2024 23:26:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21019 ) Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE events with trivial changes in StorageDescriptor .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10385/ -- To view, visit http://gerrit.cloudera.org:8080/21019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac Gerrit-Change-Number: 21019 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 19 Mar 2024 23:26:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 19 Mar 2024 23:24:52 + Gerrit-HasComments: No
[Impala-ASF-CR] (DO NOT MERGE) IMPALA-12737: List key columns in query history
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21142 ) Change subject: (DO NOT MERGE) IMPALA-12737: List key columns in query history .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15573/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 Gerrit-Change-Number: 21142 Gerrit-PatchSet: 13 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 23:23:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21166 ) Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. IMPALA-12924: Upgrade xmlsec to 2.2.6 Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Reviewed-on: http://gerrit.cloudera.org:8080/21166 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21166 ) Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 23:14:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12626: Add Tables Queried to profile/history
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20886 ) Change subject: IMPALA-12626: Add Tables Queried to profile/history .. Patch Set 25: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15572/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c9c80b2adf7f3e44225a191fe8eb9df3c4bc5aa Gerrit-Change-Number: 20886 Gerrit-PatchSet: 25 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 23:10:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15571/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 19 Mar 2024 23:02:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. Patch Set 5: -Code-Review > Patch Set 5: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10383/ TestHS2.test_open_session_query_options failed here. The value of NUM_NODES in this test should be changed from 1234 to, say, 1. https://github.com/apache/impala/blob/711a9f2bad84f92dc4af61d49ae115f0dc4239da/tests/hs2/test_hs2.py#L75 -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anshula Jain Gerrit-Reviewer: Anshula Jain Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 22:50:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Abhishek Rawat has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/21168 )
Change subject: IMPALA-12920: Support ai_generate_text built-in function for
OpenAI's chat completion API
..
IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat
completion API
Added support for following built-in functions:
- ai_generate_text(prompt)
- ai_generate_text(ai_endpoint, prompt, ai_model,
ai_api_key_jceks_secret, additional_params)
'ai_endpoint', 'ai_model' and 'ai_api_key_jceks_secret' are flagfile
options. 'ai_generate_text(prompt)' syntax expects all these to be set
to proper values. The other syntax, will try to use the provided input
parameter values, but fallback to instance level values if the inputs
are NULL or empty.
Only public OpenAI (api.openai.com) and Azure OpenAI (openai.azure.com)
API endpoints are currently supported.
Testing:
- Added unit tests for AiGenerateTextInternal function
- Ran manual tests to make sure Impala can talk with OpenAI LLMs using
'ai_generate_text' built-in function. Example sql:
select ai_generate_text("https://api.openai.com/v1/chat/completions;,
"hello", "gpt-3.5-turbo", "open-ai-key",
'{"temperature": 0.9, "model": "gpt-4"}')
Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
---
M be/src/exprs/CMakeLists.txt
A be/src/exprs/ai-functions-ir.cc
A be/src/exprs/ai-functions.h
M be/src/exprs/expr-test.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/string-functions-ir.cc
M be/src/runtime/exec-env.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
11 files changed, 506 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/21168/3
--
To view, visit http://gerrit.cloudera.org:8080/21168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
Gerrit-Change-Number: 21168
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12626: Add Tables Queried to profile/history
Hello Andrew Sherman, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20886 to look at the new patch set (#25). Change subject: IMPALA-12626: Add Tables Queried to profile/history .. IMPALA-12626: Add Tables Queried to profile/history Adds "Tables Queried" to the query profile, enumerating a comma-separated list of tables accessed during a query: Tables Queried: tpch.customer,tpch.lineitem Also adds "tables_queried" to impala_query_log and impala_query_live with the same content. Requires 'drop table sys.impala_query_log' to recreate it with the new column. Change-Id: I9c9c80b2adf7f3e44225a191fe8eb9df3c4bc5aa --- M be/src/exec/system-table-scanner.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/query-state-record.cc M be/src/service/query-state-record.h M be/src/service/workload-management-fields.cc M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/error-util.cc M common/thrift/Frontend.thrift M common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_query_live.py M tests/query_test/test_observability.py M tests/util/workload_management.py 15 files changed, 87 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20886/25 -- To view, visit http://gerrit.cloudera.org:8080/20886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c9c80b2adf7f3e44225a191fe8eb9df3c4bc5aa Gerrit-Change-Number: 20886 Gerrit-PatchSet: 25 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] (DO NOT MERGE) IMPALA-12737: List key columns in query history
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21142 to look at the new patch set (#13). Change subject: (DO NOT MERGE) IMPALA-12737: List key columns in query history .. (DO NOT MERGE) IMPALA-12737: List key columns in query history Adds "Key Columns" to the query profile, enumerating a comma-separated list of the canonical path of columns used in the query: Key Columns: functional.alltypes.month,functional.alltypes.year Also adds "key_columns" to impala_query_log and impala_query_live with the same content. Requires 'drop table sys.impala_query_log' to recreate it with the new column. Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 --- M be/src/exec/system-table-scanner.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/query-state-record.cc M be/src/service/query-state-record.h M be/src/service/workload-management-fields.cc M common/thrift/Frontend.thrift M common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/util/workload_management.py 14 files changed, 105 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/21142/13 -- To view, visit http://gerrit.cloudera.org:8080/21142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 Gerrit-Change-Number: 21142 Gerrit-PatchSet: 13 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10383/ -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anshula Jain Gerrit-Reviewer: Anshula Jain Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 22:45:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Abhishek Rawat has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/21168 )
Change subject: IMPALA-12920: Support ai_generate_text built-in function for
OpenAI's chat completion API
..
IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat
completion API
Added support for following built-in functions:
- ai_generate_text(prompt)
- ai_generate_text(ai_endpoint, prompt, ai_model,
ai_api_key_jceks_secret, additional_params)
'ai_endpoint', 'ai_model' and 'ai_api_key_jceks_secret' are flagfile
options. 'ai_generate_text(prompt)' syntax expects all these to be set
to proper values. The other syntax, will try to use the provided input
parameter values, but fallback to instance level values if the inputs
are NULL or empty.
Only public OpenAI (api.openai.com) and Azure OpenAI (openai.azure.com)
API endpoints are currently supported.
Testing:
- Added unit tests for AiGenerateTextInternal function
- Ran manual tests to make sure Impala can talk with OpenAI LLMs using
'ai_generate_text' built-in function. Example sql:
select ai_generate_text("https://api.openai.com/v1/chat/completions;,
"hello", "gpt-3.5-turbo", "open-ai-key",
'{"temperature": 0.9, "model": "gpt-4"}')
Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
---
M be/src/exprs/CMakeLists.txt
A be/src/exprs/ai-functions-ir.cc
A be/src/exprs/ai-functions.h
M be/src/exprs/expr-test.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.cc
M be/src/runtime/exec-env.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
12 files changed, 514 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/21168/2
--
To view, visit http://gerrit.cloudera.org:8080/21168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
Gerrit-Change-Number: 21168
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21169 ) Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15570/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 22:31:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21169 ) Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15569/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 22:22:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. IMPALA-12426: Query History Table Adds the ability for users to specify that Impala will create and maintain an internal Iceberg table that contains data about all completed queries. This table is automatically created at startup by each coordinator if it does not exist. Then, most completed queries are queued in memory and flushed to the query history table at a set interval (either minutes or number of records). Set, use, and show queries are not written to this table. This commit leverages the InternalServer class to maintain the query history table. Ctest unit tests have been added to assert the various pieces of code. New custom cluster tests have been added to assert the query history table is properly populated with completed queries. Negative testing consists of attempting sql injection attacks and syntactically incorrect queries. Impala built-in string functions benchmarks have been updated to include the new built-in functions. Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Reviewed-on: http://gerrit.cloudera.org:8080/20770 Reviewed-by: Riza Suminto Reviewed-by: Michael Smith Tested-by: Impala Public Jenkins --- M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/CMakeLists.txt M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/internal-server-test.cc M be/src/service/internal-server.cc M be/src/service/internal-server.h M be/src/service/query-state-record.cc M be/src/service/query-state-record.h A be/src/service/workload-management-fields.cc A be/src/service/workload-management-flags.cc A be/src/service/workload-management.cc A be/src/service/workload-management.h M be/src/util/CMakeLists.txt M be/src/util/backend-gflag-util.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h A be/src/util/sql-util-test.cc A be/src/util/sql-util.cc A be/src/util/sql-util.h M common/thrift/BackendGflags.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java M tests/authorization/test_provider.py M tests/beeswax/impala_beeswax.py M tests/common/custom_cluster_test_suite.py M tests/common/impala_connection.py A tests/custom_cluster/test_query_log.py A tests/util/assert_time.py A tests/util/memory.py A tests/util/retry.py A tests/util/workload_management.py 36 files changed, 3,705 insertions(+), 72 deletions(-) Approvals: Riza Suminto: Looks good to me, but someone else must approve Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 54 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] WIP IMPALA-12915: Use libgtest.so when built with shared libs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21163 ) Change subject: WIP IMPALA-12915: Use libgtest.so when built with shared libs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15568/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21163 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I27d21217db219f52b072a4e5cfa1caaace35d1a2 Gerrit-Change-Number: 21163 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 22:17:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 22:14:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21169 to look at the new patch set (#2). Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure Event processor goes to error state before it tries to global invalidate It remains in error state for a very short period of time. If wait_for_synced_event_id() obtains event processor status during this period, it can get status as error. This test was introduced with IMPALA-12832. Testing: - Tested manually. Added sleep in code for testing so that event processor remains in error state for little longer time. Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf --- M tests/custom_cluster/test_event_processing_error.py M tests/util/event_processor_utils.py 2 files changed, 12 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/21169/2 -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21169 ) Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10388/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 22:02:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21168 ) Change subject: IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat completion API .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15567/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b Gerrit-Change-Number: 21168 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 19 Mar 2024 21:59:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21169 ) Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/21169/1/tests/util/event_processor_utils.py File tests/util/event_processor_utils.py: http://gerrit.cloudera.org:8080/#/c/21169/1/tests/util/event_processor_utils.py@96 PS1, Line 96: _ flake8: E501 line too long (107 > 90 characters) http://gerrit.cloudera.org:8080/#/c/21169/1/tests/util/event_processor_utils.py@105 PS1, Line 105: v flake8: E501 line too long (98 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 21:52:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12916: Fix test event processor error global invalidate test random failure
k.venureddy2...@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21169 Change subject: IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure .. IMPALA-12916: Fix test_event_processor_error_global_invalidate test random failure Event processor goes to error state before it tries to global invalidate It remains in error state for a very short period of time. If wait_for_synced_event_id() obtains event processor status during this period, it can get status as error. Testing: - Tested manually. Added sleep in code for testing so that event processor remains in error state for little longer time. Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf --- M tests/custom_cluster/test_event_processing_error.py M tests/util/event_processor_utils.py 2 files changed, 10 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/21169/1 -- To view, visit http://gerrit.cloudera.org:8080/21169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I787cff4cc9f9df345cd715c02b51b8d93a150edf Gerrit-Change-Number: 21169 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] WIP IMPALA-12915: Use libgtest.so when built with shared libs
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21163 to look at the new patch set (#2). Change subject: WIP IMPALA-12915: Use libgtest.so when built with shared libs .. WIP IMPALA-12915: Use libgtest.so when built with shared libs This fixes the double-free memory issue when running unifiedbetests built with shared libs Change-Id: I27d21217db219f52b072a4e5cfa1caaace35d1a2 --- M CMakeLists.txt M bin/impala-config.sh M cmake_modules/FindGTest.cmake 3 files changed, 19 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/21163/2 -- To view, visit http://gerrit.cloudera.org:8080/21163 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I27d21217db219f52b072a4e5cfa1caaace35d1a2 Gerrit-Change-Number: 21163 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Abhishek Rawat has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21168
Change subject: IMPALA-12920: Support ai_generate_text built-in function for
OpenAI's chat completion API
..
IMPALA-12920: Support ai_generate_text built-in function for OpenAI's chat
completion API
Added support for following built-in functions:
- ai_generate_text(prompt)
- ai_generate_text(ai_endpoint, prompt, ai_model,
ai_api_key_jceks_secret, additional_params)
'ai_endpoint', 'ai_model' and 'ai_api_key_jceks_secret' are flagfile
options. 'ai_generate_text(prompt)' syntax expects all these to be set
to proper values. The other syntax, will try to use the provided input
parameter values, but fallback to instance level values if the inputs
are NULL or empty.
Only public OpenAI (api.openai.com) and Azure OpenAI (openai.azure.com)
API endpoints are currently supported.
Testing:
- Added unit tests for AiGenerateTextInternal function
- Ran manual tests to make sure Impala can talk with OpenAI LLMs using
'ai_generate_text' built-in function. Example sql:
select ai_generate_text("https://api.openai.com/v1/chat/completions;,
"hello", "gpt-3.5-turbo", "open-ai-key",
'{"temperature": 0.9, "model": "gpt-4"}')
Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
---
M be/src/exprs/CMakeLists.txt
A be/src/exprs/ai-functions-ir.cc
A be/src/exprs/ai-functions.h
M be/src/exprs/expr-test.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.cc
M be/src/runtime/exec-env.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
12 files changed, 512 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/21168/1
--
To view, visit http://gerrit.cloudera.org:8080/21168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
Gerrit-Change-Number: 21168
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Rawat
[Impala-ASF-CR] IMPALA-12920: Support ai generate text built-in function for OpenAI's chat completion API
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/21168 )
Change subject: IMPALA-12920: Support ai_generate_text built-in function for
OpenAI's chat completion API
..
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/exprs/ai-functions-ir.cc
File be/src/exprs/ai-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/exprs/ai-functions-ir.cc@178
PS1, Line 178:<< ", offset input " << overrides.GetErrorOffset();
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/exprs/ai-functions-ir.cc@187
PS1, Line 187: } else {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/exprs/ai-functions-ir.cc@189
PS1, Line 189: }
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/exprs/expr-test.cc@11295
PS1, Line 11295: ctx, StringVal::null(), prompt, StringVal::null(),
jceks_secret, json_params, dry_run);
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/21168/1/be/src/runtime/exec-env.cc@512
PS1, Line 512: Status status =
frontend_->GetSecretFromKeyStore(FLAGS_ai_api_key_jceks_secret, _key);
line too long (94 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/21168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id4446957f6030bab1f985fdd69185c3da07d7c4b
Gerrit-Change-Number: 21168
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Tue, 19 Mar 2024 21:34:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12903: Querying virtual column FILE POSITION for TEXT and JSON tables crashes Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21148 ) Change subject: IMPALA-12903: Querying virtual column FILE__POSITION for TEXT and JSON tables crashes Impala .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10387/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e1af8d526f9046aceddb5944da9e6f9c63768b0 Gerrit-Change-Number: 21148 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Mar 2024 19:45:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21167 ) Change subject: IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10386/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd Gerrit-Change-Number: 21167 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 19:44:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21167 ) Change subject: IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15566/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd Gerrit-Change-Number: 21167 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 19:32:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21167 Change subject: IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests .. IMPALA-12893: (part 1) Specify 'format-version' explicitly in Iceberg tests This CR is the first step to upgrade to Iceberg 1.4.3. The biggest change in behavior in Iceberg 1.4.3 is that Iceberg V2 tables are the default. Because of this we update some test files to explicitly create V1/V2 tables. We also introduce test files that create Iceberg tables without explicitly specifying the format version, these tests have the name *-default.test. The latter tests will need to be updated when we actually upgrade to Iceberg 1.4.3. Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd --- R testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-default.test C testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v1.test C testdata/workloads/functional-query/queries/QueryTest/iceberg-alter-v2.test R testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-default.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v1.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert-v2.test M tests/query_test/test_iceberg.py 7 files changed, 1,672 insertions(+), 120 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/21167/1 -- To view, visit http://gerrit.cloudera.org:8080/21167 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ieb4f6c1b206d1d4fd878f07ea5f1436dcae560cd Gerrit-Change-Number: 21167 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/21131 )
Change subject: IMPALA-11499: Refactor UrlEncode function to handle special
characters
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21131/3/be/src/util/coding-util.cc
File be/src/util/coding-util.cc:
http://gerrit.cloudera.org:8080/#/c/21131/3/be/src/util/coding-util.cc@83
PS3, Line 83: if (!hive_compat) {
> I think '$' and other special characters are not supported or is it another
That was previously possible. The following works for me before this change:
create table foo (i int) partitioned by (s string);
insert into foo partition (s = "$") values (1), (2);
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zihao Ye
Gerrit-Comment-Date: Tue, 19 Mar 2024 18:52:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21019 ) Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE events with trivial changes in StorageDescriptor .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10385/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac Gerrit-Change-Number: 21019 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 19 Mar 2024 18:34:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21166 ) Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 18:18:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21166 ) Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10384/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 18:18:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21166 ) Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15565/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 18:15:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21019 ) Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE events with trivial changes in StorageDescriptor .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15564/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac Gerrit-Change-Number: 21019 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 19 Mar 2024 18:14:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15563/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 18:11:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21166 ) Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 18:06:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:59:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10383/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 18:00:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@422 PS52, Line 422: initialHmsCnxnTimeoutSec)); : } > Done Failing FE test passes locally for me with this fix. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:55:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12924: Upgrade xmlsec to 2.2.6
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21166 Change subject: IMPALA-12924: Upgrade xmlsec to 2.2.6 .. IMPALA-12924: Upgrade xmlsec to 2.2.6 Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/21166/1 -- To view, visit http://gerrit.cloudera.org:8080/21166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I59d49d801e23815ad7cc56e06728d3f506c3252b Gerrit-Change-Number: 21166 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Sai Hemanth Gantasala has posted comments on this change. (
http://gerrit.cloudera.org:8080/21019 )
Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE
events with trivial changes in StorageDescriptor
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/21019/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1922
PS7, Line 1922: infoLog("Change in table schema detected for table
{}.{} from {} ({}) " +
: "to {} ({}). So file metadata reload can be
skipped.", dbName_, tblName_,
: beforeCols.get(i).getName(),
beforeCols.get(i).getType(),
: afterCols.get(i).getName(),
afterCols.get(i).getType());
: return true;
> nit: might be a bit clearer to refactor this
Done
--
To view, visit http://gerrit.cloudera.org:8080/21019
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac
Gerrit-Change-Number: 21019
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Tue, 19 Mar 2024 17:51:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12487: Skip reloading file metadata for ALTER TABLE events with trivial changes in StorageDescriptor
Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21019 to look at the new patch set (#8). Change subject: IMPALA-12487: Skip reloading file metadata for ALTER_TABLE events with trivial changes in StorageDescriptor .. IMPALA-12487: Skip reloading file metadata for ALTER_TABLE events with trivial changes in StorageDescriptor IMPALA-11534 skips reloading file metadata for some trivial ALTER_TABLE events. However, ALTER_TABLE events that have trivial changes in StorageDescriptor are not handled in IMPALA-11534. The only changes that require file metadata reload are: location, rowformat, fileformat, serde, and storedAsSubDirectories. The file metadata reload can be skipped for all other changes in SD. Testing: 1) Manual testing by changing SD parameters in local environment. 2) Added unit tests for the same in MetastoreEventsProcessorTest class. Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 161 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/21019/8 -- To view, visit http://gerrit.cloudera.org:8080/21019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6fd9a9504bf93d2529dc7accbf436ad83e51d8ac Gerrit-Change-Number: 21019 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
j_ansh...@yahoo.in has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. Patch Set 5: > Uploaded patch set 5. Made the changes as suggested -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:48:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11938: Raised error if NUM NODES is set to invalid value.
j_ansh...@yahoo.in has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: Raised error if NUM_NODES is set to invalid value. .. IMPALA-11938: Raised error if NUM_NODES is set to invalid value. If NUM_NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM_NODES: is not in range [0, 1] Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc 2 files changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/21147/5 -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20762 ) Change subject: IMPALA-12540: Query Live Table .. Patch Set 40: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15562/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 Gerrit-Change-Number: 20762 Gerrit-PatchSet: 40 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 17:25:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12883: Support updating the charge on an entry in the cache
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21122 ) Change subject: IMPALA-12883: Support updating the charge on an entry in the cache .. IMPALA-12883: Support updating the charge on an entry in the cache The cache implementation currently assumes that a cache entry's charge will remain constant over time and does not support updating the charge. This works well for existing cache users like the data cache and codegen cache. However, it doesn't work as well for use cases where the final size is not known up front. Being able to update the charge also allows for avoiding concurrency issues where two different threads are trying to insert the same entry. This adds the ability to update an entry's charge after it has been inserted into the cache. This can trigger evictions if the size increases. This also adds a way to retrieve the maximum charge supported by the cache. This allows a cache user to tune its behavior to avoid generating entries that would exceed the maximum charge. Testing: - Added tests cases to the caching backend tests Change-Id: I34f54fb3a91a77821651c25d8d3bc3a2a3945025 Reviewed-on: http://gerrit.cloudera.org:8080/21122 Reviewed-by: Michael Smith Tested-by: Impala Public Jenkins Reviewed-by: Yida Wu --- M be/src/util/cache/cache-internal.h M be/src/util/cache/cache-test.cc M be/src/util/cache/cache-test.h M be/src/util/cache/cache.h M be/src/util/cache/lirs-cache-test.cc M be/src/util/cache/lirs-cache.cc M be/src/util/cache/rl-cache-test.cc M be/src/util/cache/rl-cache.cc 8 files changed, 514 insertions(+), 91 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Yida Wu: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I34f54fb3a91a77821651c25d8d3bc3a2a3945025 Gerrit-Change-Number: 21122 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12883: Support updating the charge on an entry in the cache
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21122 ) Change subject: IMPALA-12883: Support updating the charge on an entry in the cache .. Patch Set 7: Code-Review+2 Carry Michael's +1. -- To view, visit http://gerrit.cloudera.org:8080/21122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I34f54fb3a91a77821651c25d8d3bc3a2a3945025 Gerrit-Change-Number: 21122 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 19 Mar 2024 17:16:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15561/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:14:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: Code-Review+1 Looks good to me now. Will change to +2 if DRY_RUN pass. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:06:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10382/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:07:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: (2 comments) http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@422 PS52, Line 422: initialHmsCnxnTimeoutSec)); : } > That makes sense. It's not used in any static functions, and static initial Done http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@444 PS52, Line 444: return !impalaSysTables.contains(table.getTbl()); > My next commit adds another item to impalaSysTables. I left the variable in place and moved its initialization to the class constructor. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 17:04:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12540: Query Live Table
Hello Andrew Sherman, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20762 to look at the new patch set (#40). Change subject: IMPALA-12540: Query Live Table .. IMPALA-12540: Query Live Table Defines SystemTable which are in-memory tables that can provide access to Impala state. Adds the 'impala_query_live' to the database 'sys', which already exists for 'sys.impala_query_log'. Implements the 'impala_query_live' table to view active queries across all coordinators sharing the same statestore. SystemTables create new SystemTableScanNodes for their scan node implementation. When computing scan range locations, SystemTableScanNodes creates a scan range for each in the cluster (identified via ClusterMembershipMgr). This produces a plan that looks like: Query: explain select * from sys.impala_query_live ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=4.00MB Threads=2 | | Per-Host Resource Estimates: Memory=11MB | | WARNING: The following tables are missing relevant table | | and/or column statistics. | | sys.impala_query_live | || | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED]| | | | | 00:SCAN SYSTEM_TABLE [sys.impala_query_live] | |row-size=72B cardinality=20 | ++ Execution will pull data from ImpalaServer on the backend via a SystemTableScanner implementation based on table name. In the query profile, SYSTEM_TABLE_SCAN_NODE includes ActiveQueryCollectionTime and PendingQueryCollectionTime to track time spent collecting QueryState from ImpalaServer. Grants QueryScanner private access to ImpalaServer, identical to how ImpalaHttpHandler access internal server state. Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 --- M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc M be/src/exec/scan-node.cc A be/src/exec/system-table-scan-node.cc A be/src/exec/system-table-scan-node.h A be/src/exec/system-table-scanner.cc A be/src/exec/system-table-scanner.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/scheduling/scheduler.cc M be/src/service/fe-support.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-state-record.h M be/src/service/workload-management.cc M be/src/util/sharded-query-map-util.h M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/PlanNodes.thrift M common/thrift/StatestoreService.thrift A common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/SystemTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java A fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java A tests/custom_cluster/test_query_live.py M tests/util/workload_management.py 36 files changed, 1,277 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/20762/40 -- To view, visit http://gerrit.cloudera.org:8080/20762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743 Gerrit-Change-Number: 20762 Gerrit-PatchSet: 40 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] Add RE2 2024-03-01 and Abseil
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21165 ) Change subject: Add RE2 2024-03-01 and Abseil .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 Gerrit-Change-Number: 21165 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 16:55:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12426: Query History Table
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 53: (1 comment) http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@422 PS52, Line 422: initialHmsCnxnTimeoutSec)); : } > Several FE test broken in exhaustive test. Running CatalogHmsSyncToLatestEv That makes sense. It's not used in any static functions, and static initialization off BackendConfig.INSTANCE is probably a bad idea. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 16:52:01 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add RE2 2024-03-01 and Abseil
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21165 Change subject: Add RE2 2024-03-01 and Abseil .. Add RE2 2024-03-01 and Abseil This adds a build of the latest RE2 release 2024-03-01. RE2 took a dependency on Abseil starting in 2023-06-01, so this adds Abseil as well. Abseil is a C++ library developed by Google that provides various functionality including string libraries, hash tables, etc. It is useful on its own. Abseil's shared library build produces a large number of shared libraries that depend on each other. This implements a basic optimization of the symlink_required_libs function to check the destination directory to see if the required library has already been symlinked (or is otherwise present). This makes a big difference in the time take for symlink_required_libs for Abseil. Testing: - Ran a toolchain build Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 --- M buildall.sh M functions.sh A source/abseil-cpp/build.sh M source/re2/build.sh 4 files changed, 146 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/21165/1 -- To view, visit http://gerrit.cloudera.org:8080/21165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie8e8a40cb0e5243b68a9abe35a106872bd9bf7f8 Gerrit-Change-Number: 21165 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-12426: Query History Table
Hello Andrew Sherman, Riza Suminto, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20770 to look at the new patch set (#53). Change subject: IMPALA-12426: Query History Table .. IMPALA-12426: Query History Table Adds the ability for users to specify that Impala will create and maintain an internal Iceberg table that contains data about all completed queries. This table is automatically created at startup by each coordinator if it does not exist. Then, most completed queries are queued in memory and flushed to the query history table at a set interval (either minutes or number of records). Set, use, and show queries are not written to this table. This commit leverages the InternalServer class to maintain the query history table. Ctest unit tests have been added to assert the various pieces of code. New custom cluster tests have been added to assert the query history table is properly populated with completed queries. Negative testing consists of attempting sql injection attacks and syntactically incorrect queries. Impala built-in string functions benchmarks have been updated to include the new built-in functions. Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 --- M be/src/runtime/query-driver.cc M be/src/runtime/query-driver.h M be/src/service/CMakeLists.txt M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/internal-server-test.cc M be/src/service/internal-server.cc M be/src/service/internal-server.h M be/src/service/query-state-record.cc M be/src/service/query-state-record.h A be/src/service/workload-management-fields.cc A be/src/service/workload-management-flags.cc A be/src/service/workload-management.cc A be/src/service/workload-management.h M be/src/util/CMakeLists.txt M be/src/util/backend-gflag-util.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h A be/src/util/sql-util-test.cc A be/src/util/sql-util.cc A be/src/util/sql-util.h M common/thrift/BackendGflags.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java M tests/authorization/test_provider.py M tests/beeswax/impala_beeswax.py M tests/common/custom_cluster_test_suite.py M tests/common/impala_connection.py A tests/custom_cluster/test_query_log.py A tests/util/assert_time.py A tests/util/memory.py A tests/util/retry.py A tests/util/workload_management.py 36 files changed, 3,705 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/20770/53 -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 53 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[native-toolchain-CR] Add libpfm / google benchmark
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21164 Change subject: Add libpfm / google benchmark .. Add libpfm / google benchmark Google benchmark is a framework for writing benchmarks similar to Googletest's framework for tests. An interesting feature of google benchmark is that it integrates with libpfm / perfmon2 to get hardware performance counters as described in https://abseil.io/fast/53 . This adds a build for both libpfm and google benchmark to allow testing this capability for microbenchmarks. Testing: - Ran a toolchain build Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a --- M buildall.sh M functions.sh A source/googlebenchmark/build.sh A source/libpfm/build.sh 4 files changed, 105 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/21164/1 -- To view, visit http://gerrit.cloudera.org:8080/21164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I18a59e430850e45e7a28792925692a70ef9fce4a Gerrit-Change-Number: 21164 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11938: If NUM NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM NODES: is not in range [0, 1]
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: If NUM_NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM_NODES: is not in range [0, 1] .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15560/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 16:52:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11938: If NUM NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM NODES: is not in range [0, 1]
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21147 )
Change subject: IMPALA-11938: If NUM_NODES is set to a value other than 0 or 1
below error is raised ERROR: Invalid value for query option
NUM_NODES: is not in range [0, 1]
..
Patch Set 4:
(5 comments)
Looks good. Just few style nits to edit.
http://gerrit.cloudera.org:8080/#/c/21147/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21147/4//COMMIT_MSG@7
PS4, Line 7: IMPALA-11938: If NUM_NODES is set to a value other than 0 or 1
below error is raised
: ERROR: Invalid value for query option NUM_NODES:
is not in range [0, 1]
nit: Use short title and separate the title with the rest of commit message
with an empty line. I recommend to change it like this:
IMPALA-11938: Raise error if NUM_NODES is set to invalid value.
If NUM_NODES is set to a value other than 0 or 1 below error is raised:
ERROR: Invalid value for query option NUM_NODES: is not in range [0, 1]
http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options-test.cc
File be/src/service/query-options-test.cc:
http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options-test.cc@273
PS4, Line 273: {
nit: align the open curly brace with the one in L266
http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options-test.cc@274
PS4, Line 274: };
nit: keep this L274 the same as the original.
http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options.cc@246
PS4, Line 246: option, value, 0, 1, _t_val));
> line has trailing whitespace
nit: Please remove the trailing whitespace.
http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options.cc@246
PS4, Line 246: option
nit: add 4 spaces indentation before 'option'.
--
To view, visit http://gerrit.cloudera.org:8080/21147
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812
Gerrit-Change-Number: 21147
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Tue, 19 Mar 2024 16:45:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11938: If NUM NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM NODES: is not in range [0, 1]
j_ansh...@yahoo.in has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: If NUM_NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM_NODES: is not in range [0, 1] .. IMPALA-11938: If NUM_NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM_NODES: is not in range [0, 1] Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc 2 files changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/21147/4 -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-11938: If NUM NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM NODES: is not in range [0, 1]
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21147 ) Change subject: IMPALA-11938: If NUM_NODES is set to a value other than 0 or 1 below error is raised ERROR: Invalid value for query option NUM_NODES: is not in range [0, 1] .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/21147/4/be/src/service/query-options.cc@246 PS4, Line 246: option, value, 0, 1, _t_val)); line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/21147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib283ae350875a7ac217490c216a26281f1bb4812 Gerrit-Change-Number: 21147 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 16:29:28 + Gerrit-HasComments: Yes
[native-toolchain-CR] Omnibus: Add new versions of several libraries
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21144 ) Change subject: Omnibus: Add new versions of several libraries .. Omnibus: Add new versions of several libraries To economize on toolchain builds, this adds new versions for several libraries while keeping the existing version. This allows us to test and upgrade the libraries without a new toolchain build for each upgrade. Here are the included new versions: 1. RE2 2023-03-01 (for IMPALA-12363) 2. ZLib 1.3.1 3. Cloudflare Zlib 7aa510344e 4. Gperftools 2.15 5. LZ4 1.9.4 (for IMPALA-12368) 6. ZSTD 1.5.5 (for IMPALA-12369) 7. Libunwind 1.8.1 8. CCTZ 2.3 and 2.4 9. jwt-cpp 0.6.0 and 0.7.0 10. Arrow 13.0.0 11. Mold 2.4.1 12. Python 3.8.18 Since Arrow and Mold have not been used by Impala yet, this removes the code to build the old version. All others are additional builds without removing the old version. These are small libraries, so this has minimal impact on build time. Testing: - Ran x86_64 / ARM toolchain builds Change-Id: I2a900bc9b0c939391a3c6f970b1b5d99fbb42fcf Reviewed-on: http://gerrit.cloudera.org:8080/21144 Reviewed-by: Michael Smith Reviewed-by: Laszlo Gaal Tested-by: Joe McDonnell --- M buildall.sh D source/arrow/arrow-9.0.0-patches/0001-ARROW-17847-C-Support-unquoted-decimal-in-JSON-parse.patch D source/arrow/arrow-9.0.0-patches/0002-ARROW-17995-C-Fix-json-decimals-not-being-rescaled-b.patch M source/arrow/build.sh M source/re2/build.sh 5 files changed, 23 insertions(+), 469 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Laszlo Gaal: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/21144 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2a900bc9b0c939391a3c6f970b1b5d99fbb42fcf Gerrit-Change-Number: 21144 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] IMPALA-12900: Build binutils with -O3
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21145 ) Change subject: IMPALA-12900: Build binutils with -O3 .. IMPALA-12900: Build binutils with -O3 The binutils build happens before we have switched over to using the toolchain compiler. This means that it also does not set CFLAGS/CXXFLAGS. The default optimization level for binutils is -O2. It is possible that we could get a bit extra speed by using -O3, so this sets CFLAGS/CXXFLAGS to use -O3 for binutils. Testing: - Toolchain builds on x86_64 and ARM Change-Id: I2e75db0759b4d3d4e6cc2ce929b1741808f1b771 Reviewed-on: http://gerrit.cloudera.org:8080/21145 Reviewed-by: Michael Smith Reviewed-by: Laszlo Gaal Tested-by: Joe McDonnell --- M source/binutils/build.sh 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Laszlo Gaal: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/21145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2e75db0759b4d3d4e6cc2ce929b1741808f1b771 Gerrit-Change-Number: 21145 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] IMPALA-12900: Build binutils with -O3
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21145 ) Change subject: IMPALA-12900: Build binutils with -O3 .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e75db0759b4d3d4e6cc2ce929b1741808f1b771 Gerrit-Change-Number: 21145 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 16:20:13 + Gerrit-HasComments: No
[native-toolchain-CR] Omnibus: Add new versions of several libraries
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21144 ) Change subject: Omnibus: Add new versions of several libraries .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21144 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a900bc9b0c939391a3c6f970b1b5d99fbb42fcf Gerrit-Change-Number: 21144 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 16:20:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12729: Allow creating primary keys for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21149 ) Change subject: IMPALA-12729: Allow creating primary keys for Iceberg tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15559/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7bea787acdabd8cb04661f4ddb5c3309af0364a6 Gerrit-Change-Number: 21149 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2024 16:15:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21029 ) Change subject: IMPALA-12543: Detect self-events before finishing DDL .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15558/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 Gerrit-Change-Number: 21029 Gerrit-PatchSet: 17 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 19 Mar 2024 16:04:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12729: Allow creating primary keys for Iceberg tables
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21149 to look at the new patch set (#2). Change subject: IMPALA-12729: Allow creating primary keys for Iceberg tables .. IMPALA-12729: Allow creating primary keys for Iceberg tables There are writer engines that use Iceberg's identifier-field-ids from the Iceberg schema to identify the columns to be written into the equality delete files (Flink, NiFi). So far Impala wasn't able to populate this identifier-field-ids. This patch introduces the support for not enforced primary keys for Iceberg tables, where the primary key is going to be used for setting identifier-field-ids during Iceberg schema creation. Example syntax: CREATE TABLE ice_tbl ( i int NOT NULL, j int, s string NOT NULL primary key(i, s) not enforced) PARTITIONED BY SPEC (truncate(10, s)) STORED AS ICEBERG; There are some constraints with primary keys (PK) following the behavior of Flink: - Only NOT NULL columns can be in the PK. - PK is not allowed in the column definition level like 'i int NOT NULL PRIMARY KEY'. - If the table is partitioned then the partition columns have to be part of the PK. - Float and double columns are not allowed for the PK. - Not allowed to drop a column that is used as a PK. Testing: - New E2E tests added for different table creation scenarios. - Manual test to use Nifi for writing into a table with PK. Change-Id: I7bea787acdabd8cb04661f4ddb5c3309af0364a6 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 14 files changed, 272 insertions(+), 49 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/21149/2 -- To view, visit http://gerrit.cloudera.org:8080/21149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7bea787acdabd8cb04661f4ddb5c3309af0364a6 Gerrit-Change-Number: 21149 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21029 ) Change subject: IMPALA-12543: Detect self-events before finishing DDL .. Patch Set 17: (1 comment) http://gerrit.cloudera.org:8080/#/c/21029/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/21029/16/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3429 PS16, Line 3429: // if the table is being replicated we issue the HMS API to truncate the table > This will introduce dangling version if !isTableBeingReplicated and !params Moved this within try and check for those conditions. -- To view, visit http://gerrit.cloudera.org:8080/21029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 Gerrit-Change-Number: 21029 Gerrit-PatchSet: 17 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 19 Mar 2024 15:42:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12543: Detect self-events before finishing DDL
Hello Quanlong Huang, Jason Fehr, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21029 to look at the new patch set (#17). Change subject: IMPALA-12543: Detect self-events before finishing DDL .. IMPALA-12543: Detect self-events before finishing DDL test_iceberg_self_events has been flaky for not having tbls_refreshed_before equal to tbls_refreshed_after in-between query executions. Further investigation reveals concurrency bug due to db/table level lock is not taken during db/table self-events check (IMPALA-12461 part1). The order of ALTER TABLE operation is as follow: 1. alter table starts in CatalogOpExecutor 2. table level lock is taken 3. HMS RPC starts (CatalogOpExecutor.applyAlterTable()) 4. HMS generates the event 5. HMS RPC returns 6. table is reloaded 7. catalog version is added to inflight event list 8. table level lock is releases Meanwhile the event processor thread fetches the new event after 4 and before 7. Because of IMPALA-12461 (part 1), it can also finish self-events checking before reaching 7. Before IMPALA-12461, self-events would have needed to wait for 8. Note that this issue is only relevant for table level events, as self-events checking for partition level events still takes table lock. This patch fix the issue by adding newCatalogVersion to the table's inflight event list before updating HMS. If HMS update does not complete (ie., an exception is thrown), the new newCatalogVersion that was added is then removed. This patch also fix few smaller issues, including: - Avoid incrementing EVENTS_SKIPPED_METRIC if numFilteredEvents == 0 in MetastoreEventFactory.getFilteredEvents(). - Increment EVENTS_SKIPPED_METRIC in MetastoreTableEvent.reloadTableFromCatalog() if table is already in the middle of reloading (revealed through flaky test_skipping_older_events). - Rephrase misleading log message in MetastoreEventProcessor.getNextMetastoreEvents(). Testing: - Add TestEventProcessingWithImpala, run it with debug_action and sync_ddl dimensions. - Pass exhaustive tests. Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/DebugUtils.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 764 insertions(+), 512 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/21029/17 -- To view, visit http://gerrit.cloudera.org:8080/21029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8365c934349ad21a4d9327fc11594d2fc3445f79 Gerrit-Change-Number: 21029 Gerrit-PatchSet: 17 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12426: Query History Table
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 ) Change subject: IMPALA-12426: Query History Table .. Patch Set 52: (1 comment) http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20770/52/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@444 PS52, Line 444: return !impalaSysTables.contains(table.getTbl()); > Can replace with My next commit adds another item to impalaSysTables. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 52 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 15:10:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12913: Refactor Workload Management Custom Cluser Tests
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21153 ) Change subject: IMPALA-12913: Refactor Workload Management Custom Cluser Tests .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21153/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21153/1//COMMIT_MSG@7 PS1, Line 7: IMPALA-12913: Refactor Workload Management Custom Cluser Tests typo in the commit message: "Cluster" -- To view, visit http://gerrit.cloudera.org:8080/21153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e3249a8f306cf43de0d6f6586711c779399e83b Gerrit-Change-Number: 21153 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 19 Mar 2024 15:07:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12533: Support row materialization outside of fetch lock
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20850 ) Change subject: IMPALA-12533: Support row materialization outside of fetch lock .. Patch Set 15: (3 comments) http://gerrit.cloudera.org:8080/#/c/20850/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20850/11//COMMIT_MSG@24 PS11, Line 24: It is expected that the default : threshold will never be reached with a single fetch stream. > Added counter for number of streams as suggested. Thanks! http://gerrit.cloudera.org:8080/#/c/20850/13/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/20850/13/be/src/exec/buffered-plan-root-sink.cc@186 PS13, Line 186: Initialize > Added support to BlockingPlanRootSync. I am not sure if it is really useful to support this in BlockingPlanRootSink - it only holds a single row batch until the client fetches it, so parallel fetches are likely to just wait for BufferedPlanRootSink to get another batch from the plan tree. http://gerrit.cloudera.org:8080/#/c/20850/13/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/20850/13/be/src/service/impala-hs2-server.cc@251 PS13, Line 251: fetch_results->__set_hasMoreRows(!query_handle->eos()); > The query_handle reference keeps the QueryDriver/RunTimeState/MemPool aroun Was able to cause a crash by forcing delayed materialization + adding a sleep here. I used the following query: select replace(string_col, "0", "1") from functional.alltypes; replace() was chosen because it uses FunctionContext::FRAGMENT_LOCAL scope in its prepare/close functions. Uploaded the patch here. the commit message also includes the calltack: https://github.com/csringhofer/Impala/commit/0ebc8fd36134614fe406253e1b1080901905f4ae When disabling result spooling I could crash it even with a query with functions with prepare/close: set SPOOL_QUERY_RESULTS=0; select concat(string_col, string_col) from functional.alltypes; This failed at calar-expr-evaluator.h:232] Check failed: i < fn_ctxs_.size() (0 vs. 0) impala::ScalarFnCall::OpenEvaluator() impala::ScalarExprEvaluator::Clone() impala::HS2ColumnarResultSet::InitDelayedMaterialization() -- To view, visit http://gerrit.cloudera.org:8080/20850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9512aa6022dbcf81e7eb5f559853b89fe80bd23 Gerrit-Change-Number: 20850 Gerrit-PatchSet: 15 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Mar 2024 14:17:03 + Gerrit-HasComments: Yes
