[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in DeserializeThriftMsg
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/19179 )
Change subject: IMPALA-11669: (addendum) Set TConfiguration in
DeserializeThriftMsg
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/rpc/thrift-util.h
File be/src/rpc/thrift-util.h:
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/rpc/thrift-util.h@42
PS2, Line 42: int ThriftRpcMaxMessageSize()
define as inline function
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/rpc/thrift-util.h@44
PS2, Line 44: inline int ThriftDefaultMaxMessageSize() {
add a comment for this function
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/transport/TSaslTransport.cpp
File be/src/transport/TSaslTransport.cpp:
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/transport/TSaslTransport.cpp@43
PS2, Line 43: TMemoryBuffer(DEFAULT_MEM_BUF_SIZE
should we set DefaultTConfiguration() for this TMemoryBuffer?
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/transport/TSaslTransport.cpp@53
PS2, Line 53: TMemoryBuffer()
same comment as last one.
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/util/parquet-reader.cc
File be/src/util/parquet-reader.cc:
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/util/parquet-reader.cc@75
PS2, Line 75: TMemoryBuffer
Should we set TConfiguration?
--
To view, visit http://gerrit.cloudera.org:8080/19179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb
Gerrit-Change-Number: 19179
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 31 Oct 2022 06:02:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#8). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System This patch adds support for OSS (Aliyun Oobject Strorage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service Tests: Run CORE tests. Mdddanual ACID test verification. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M testdata/workloads/functional-planner/queries/PlannerTest/scan-node-fs-scheme.test M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 17 files changed, 113 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/8 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 8 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11741/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 8 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Mon, 31 Oct 2022 07:25:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8754/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 16:19:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9488: Add metrics for EC reads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19178 ) Change subject: IMPALA-9488: Add metrics for EC reads .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11744/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb06bac9dea4b632621653d2935e9a7b2dc81341 Gerrit-Change-Number: 19178 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 31 Oct 2022 16:29:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9488: Add metrics for EC reads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19178 ) Change subject: IMPALA-9488: Add metrics for EC reads .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11743/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieb06bac9dea4b632621653d2935e9a7b2dc81341 Gerrit-Change-Number: 19178 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 31 Oct 2022 16:29:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11749/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 18:11:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9488: Add metrics for EC reads
Hello Zoltan Borok-Nagy, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19178 to look at the new patch set (#3). Change subject: IMPALA-9488: Add metrics for EC reads .. IMPALA-9488: Add metrics for EC reads Adds metric tracking erasure-coded bytes read. TODO: add test case. Change-Id: Ieb06bac9dea4b632621653d2935e9a7b2dc81341 --- M be/src/exec/base-sequence-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-page-index.cc M be/src/exec/parquet/parquet-page-reader.cc M be/src/exec/scanner-context.cc M be/src/exec/text/hdfs-text-scanner.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/scheduler.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/PlanNodes.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M tests/custom_cluster/test_query_retries.py 23 files changed, 99 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19178/3 -- To view, visit http://gerrit.cloudera.org:8080/19178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieb06bac9dea4b632621653d2935e9a7b2dc81341 Gerrit-Change-Number: 19178 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9488: Add metrics for EC reads
Hello Zoltan Borok-Nagy, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19178 to look at the new patch set (#4). Change subject: IMPALA-9488: Add metrics for EC reads .. IMPALA-9488: Add metrics for EC reads Adds metric tracking erasure-coded bytes read. TODO: add test case. Change-Id: Ieb06bac9dea4b632621653d2935e9a7b2dc81341 --- M be/src/exec/base-sequence-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/orc/hdfs-orc-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-page-index.cc M be/src/exec/parquet/parquet-page-reader.cc M be/src/exec/scanner-context.cc M be/src/exec/text/hdfs-text-scanner.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/scheduler.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/PlanNodes.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java 22 files changed, 98 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19178/4 -- To view, visit http://gerrit.cloudera.org:8080/19178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieb06bac9dea4b632621653d2935e9a7b2dc81341 Gerrit-Change-Number: 19178 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10436: Support storage handler privileges for external Kudu table creation
Fang-Yu Rao has posted comments on this change. (
http://gerrit.cloudera.org:8080/17640 )
Change subject: IMPALA-10436: Support storage handler privileges for external
Kudu table creation
..
Patch Set 10:
(5 comments)
Hi all, I have slightly revised the previous patch set according to Quanlong's
suggestions. Let me know if you have any other comment on the patch. Thanks!
http://gerrit.cloudera.org:8080/#/c/17640/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17640/9//COMMIT_MSG@14
PS9, Line 14: supported by Apache Ranger once RANGER-3281 is resolved, which in
turn
: depends on the release of Apache Hive 4.0 that consists of
HIVE-24705.
> I see HIVE-24705 is resolved but RANGER-3281 is still open. Do we need to b
Thanks Quanlong!
We do not have to bump up the build number for this patch in that RANGER-3281
is already shipped in Cloudera's distribution of Ranger.
My current understanding is that Apache Ranger relies on the official
release(s) of Apache Hive's API's. Since HIVE-24705 exists only on Apache
Hive's master, which has not yet been officially released, RANGER-3281 could
not be committed to Apache Ranger's master at the moment. Thus RANGER-3281 is
still open.
At https://hive.apache.org/downloads.html we can see that 4.0.0-alpha-1 and
3.1.3 of Hive were released in the past March and April, respectively. We will
have to wait for Hive 4 until we can resolve RANGER-3281 .
http://gerrit.cloudera.org:8080/#/c/17640/9/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/17640/9/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@61
PS9, Line 61: import com.google.common.primitives.Ints;
> nit: move this to line 59 to keep the import list sorted
Done
http://gerrit.cloudera.org:8080/#/c/17640/9/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@369
PS9, Line 369: boolean isExternal = tableDef_.isExternal() ||
: MetaStoreUtil.findTblPropKeyCaseInsensitive(
: getTblProperties(), "EXTERNAL") != null;
: if (isExternal) {
> Should we update these?
Thanks Quanlong!
I will remove this stale code comment. :-)
http://gerrit.cloudera.org:8080/#/c/17640/9/fe/src/main/java/org/apache/impala/analysis/StorageHandlerUri.java
File fe/src/main/java/org/apache/impala/analysis/StorageHandlerUri.java:
http://gerrit.cloudera.org:8080/#/c/17640/9/fe/src/main/java/org/apache/impala/analysis/StorageHandlerUri.java@37
PS9, Line 37: storageHandlerUri.equals("*://*")) {
> nit: we can use equals() directly
Done
http://gerrit.cloudera.org:8080/#/c/17640/9/fe/src/main/java/org/apache/impala/analysis/StorageHandlerUri.java@47
PS9, Line 47: equals("*")) {
> nit: use equals() directly
Done
--
To view, visit http://gerrit.cloudera.org:8080/17640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4
Gerrit-Change-Number: 17640
Gerrit-PatchSet: 10
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Mon, 31 Oct 2022 17:14:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10436: Support storage handler privileges for external Kudu table creation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Support storage handler privileges for external Kudu table creation .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11747/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 Gerrit-Change-Number: 17640 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 31 Oct 2022 17:33:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11667: (Addendum) Remove explicit include of spring-core
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19153 ) Change subject: IMPALA-11667: (Addendum) Remove explicit include of spring-core .. Patch Set 3: Code-Review+2 I think this makes sense -- To view, visit http://gerrit.cloudera.org:8080/19153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc9b46b45ed24e5e924db88ef792c1a68a906394 Gerrit-Change-Number: 19153 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 17:59:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#7). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h A be/src/rpc/thrift-util.inline.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 11 files changed, 146 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/7 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11750/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 18:47:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19182 Change subject: IMPALA-11695: Reduce clang tidy warning output size .. IMPALA-11695: Reduce clang tidy warning output size The Clang Tidy build enables all warnings via -Wall and -Weverything. This produces enormous output. Looking at a recent failed Clang Tidy build, there are ~4.5 million warnings generated. Of these, about 4 million are from C++98 compatibility warnings. A further 250 thousand are from padding warnings. Since these are not particularly interesting, this disables both of those to reduce the output size. Testing: - Built with clang tidy locally Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 --- M be/CMakeLists.txt 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19182/1 -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19181 ) Change subject: IMPALA-11470: Add Cache For Codegen Functions .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/19181/1/be/src/runtime/fragment-state.cc File be/src/runtime/fragment-state.cc: http://gerrit.cloudera.org:8080/#/c/19181/1/be/src/runtime/fragment-state.cc@108 PS1, Line 108: return apache::thrift::ThriftDebugString(fragment_); Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead. http://gerrit.cloudera.org:8080/#/c/19181/1/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/19181/1/common/thrift/ImpalaService.thrift@746 PS1, Line 746: // Specify whether to use the debug mode for codegen cache. In debug mode, we will insert line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/19181/1/common/thrift/ImpalaService.thrift@747 PS1, Line 747: // a full key for the cache and allow more statistics, otherwise, a hashcode will be used line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/19181/1/tests/common/test_result_verifier.py File tests/common/test_result_verifier.py: http://gerrit.cloudera.org:8080/#/c/19181/1/tests/common/test_result_verifier.py@759 PS1, Line 759: def assert_codegen_cache_hit(profile_string, expect_hit): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/19181/1/tests/custom_cluster/test_codegen_cache.py File tests/custom_cluster/test_codegen_cache.py: http://gerrit.cloudera.org:8080/#/c/19181/1/tests/custom_cluster/test_codegen_cache.py@24 PS1, Line 24: @SkipIf.not_hdfs flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/19181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 Gerrit-Change-Number: 19181 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 31 Oct 2022 15:48:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7092: Restore tests after HDFS fixes
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19159 ) Change subject: IMPALA-7092: Restore tests after HDFS fixes .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b0bbc604601e6fab742f145c1adfb3c47b3fb6e Gerrit-Change-Number: 19159 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 16:07:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#4). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 10 files changed, 119 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/4 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 5: Code-Review+1 (1 comment) All of this looks good to me. Just a small nit about the commit message. http://gerrit.cloudera.org:8080/#/c/19172/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19172/5//COMMIT_MSG@17 PS5, Line 17: Impala schedules work to executors based on blocks reported by HDFS, : which for EC actually represent block groups. So with default block : size, a file in EC has 1/3rd the number of schedulable blocks. Can you add a sentence about how this specifically results in Parquet lineitem having fewer files? Load single file to text = 6 blocks on non-EC, 2 block groups on EC => Parquet load has 3 files vs 2 files. -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 16:53:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/19179 )
Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19179/4/be/src/rpc/thrift-util.h
File be/src/rpc/thrift-util.h:
http://gerrit.cloudera.org:8080/#/c/19179/4/be/src/rpc/thrift-util.h@49
PS4, Line 49: inline int ThriftRpcMaxMessageSize() {
Looks like inlining ThriftRpcMaxMessageSize makes clang-tidy unhappy. I'll try
revert this and see if it can pass clang-tidy.
--
To view, visit http://gerrit.cloudera.org:8080/19179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb
Gerrit-Change-Number: 19179
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 31 Oct 2022 17:12:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/19179/4/be/src/rpc/thrift-util.h File be/src/rpc/thrift-util.h: http://gerrit.cloudera.org:8080/#/c/19179/4/be/src/rpc/thrift-util.h@49 PS4, Line 49: /// Return the default Thrift's TConfi > Looks like inlining ThriftRpcMaxMessageSize makes clang-tidy unhappy. I'll How about to add a header file thrift-util.inline.h and move the inline function definitions to new header file? -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 17:23:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19181 Change subject: IMPALA-11470: Add Cache For Codegen Functions .. IMPALA-11470: Add Cache For Codegen Functions The patch adds supports of the cache for CodeGen functions to improve the performance of sub-second queries. The main idea is to store the codegen functions to a cache, and reuse them when it is appropriate to save the llvm optimization time which could take over hundreds of milliseconds. In this patch, we implement the cache in a fragment-level way, that is to store all the codegen functions of a fragment in a cache entry, if one exactly same fragment comes again, it should be able to find all the codegen functions it needs from the specific cache entry, therefore saving the time. The module bitcode is used as the key to the cache, which will be generated before the module optimization and final compilation. If codegen_cache_debug_mode is set to false, which is by default, we will only store the hash code of the module bitcode as the key to reduce memory consumption. Otherwise we will store the full bitcode string. KrpcDataStreamSenderConfig hash seed is changed to a constant if using codegen cache because it can't hit the cache for the fragment if using a dynamic hash seed. As a limitation, we don't cache the udf functions, because it may rely on the shared library in runtime, if something has been changed to the loading of the shared library, for example recreating a database containing the udf, the cache could become not reusable and result in a crash if reusing it. To disable the cache for udf, we add a logic in the expression analysis, if a fragment contains a function with a non-empty hdfs location, we won't cache any codegen functions of that fragment. Tpch performance test showed no significant difference after adding this logic. The patch also introduces following new flags for start and query options. start option: - codegen_cache_capacity The capacity of the cache, if set to 0, codegen cache is disabled. query option: - disable_codegen_cache Codegen cache will be disabled is set to false. - codegen_cache_debug_mode If enabled, more logs and statistics will be allowed, and the full key content will be stored to the cache, otherwise, the cache will only store the hash code of the key to reduce memory consumption. New impalad metrics: impala.codegen-cache.misses impala.codegen-cache.entries-in-use impala.codegen-cache.entries-in-use-bytes impala.codegen-cache.entries-evicted impala.codegen-cache.hits impala.codegen-cache.entry-sizes New profile Metrics: CodegenCacheLookupTime CodegenCacheSaveTime ModuleBitcodeGenTime NumCachedFunctions TPCH-1 performance evaluation(8 iteration) on AWS m5a.4xlarge: Query Cached(s) NoCache(s) Delta(Avg) NoCodegen(s) Delta(Avg) TPCH-Q1 0.385 1.11-65.32% 5.35 -92.8% TPCH-Q2 0.561.25-55.2%0.44 27.27% TPCH-Q3 0.370.79-53.16% 0.43 -13.95% TPCH-Q4 0.355 0.48-26.04% 0.33 7.58% TPCH-Q5 0.391.18-66.95% 0.39 0% TPCH-Q6 0.240.28-14.29% 0.78 -69.23% TPCH-Q7 0.385 1.27-69.69% 0.37 4.05% TPCH-Q8 0.575 1.5 -61.67% 0.42 36.9% TPCH-Q9 0.795 1.5 -47.35% 1.04 -23.56% TPCH-Q10 0.595 1.09-45.41% 0.84 -29.17% TPCH-Q11 0.295 1.01-70.79% 0.19 55.26% TPCH-Q12 0.275 0.53-48.11% 0.38 -27.63% TPCH-Q13 1.111.2 -7.5% 1.11 0% TPCH-Q14 0.550.84-34.52% 0.42 30.95% TPCH-Q15 0.325 0.76-57.24% 0.44 -26.14% TPCH-Q16 0.320.8 -60% 0.39 -17.95% TPCH-Q17 0.555 0.9 -38.33% 0.88 -36.93% TPCH-Q18 0.535 0.94-43.09% 0.85 -37.06% TPCH-Q19 0.782.48-68.55% 0.35 122.86% TPCH-Q20 0.335 1.09-69.27% 0.29 15.52% TPCH-Q21 0.825 1.14-27.63% 0.89 -7.3% TPCH-Q22 0.255 0.51-50% 0.26 -1.92% It shows good results compared to current codegen without cache, however, as expected, for short queries, compared to codegen disabled, it is not always faster, there could be room to improve. Tests: Ran core tests. E2e testcase TestCodegenCache is added. Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 --- M be/src/codegen/CMakeLists.txt A be/src/codegen/llvm-codegen-cache.cc A be/src/codegen/llvm-codegen-cache.h M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/exec-node.cc M be/src/exprs/agg-fn.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-state.cc
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/19179 )
Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
..
Patch Set 4:
(5 comments)
Thank you for catching those!
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/rpc/thrift-util.h
File be/src/rpc/thrift-util.h:
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/rpc/thrift-util.h@42
PS2, Line 42:
> define as inline function
Done
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/rpc/thrift-util.h@44
PS2, Line 44: inline int ThriftDefaultMaxMessageSize() {
> add a comment for this function
Done
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/transport/TSaslTransport.cpp
File be/src/transport/TSaslTransport.cpp:
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/transport/TSaslTransport.cpp@43
PS2, Line 43: TMemoryBuffer(DEFAULT_MEM_BUF_SIZE
> should we set DefaultTConfiguration() for this TMemoryBuffer?
Done
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/transport/TSaslTransport.cpp@53
PS2, Line 53: TMemoryBuffer(i
> same comment as last one.
Done
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/util/parquet-reader.cc
File be/src/util/parquet-reader.cc:
http://gerrit.cloudera.org:8080/#/c/19179/2/be/src/util/parquet-reader.cc@75
PS2, Line 75: memory trans
> Should we set TConfiguration?
Done
--
To view, visit http://gerrit.cloudera.org:8080/19179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb
Gerrit-Change-Number: 19179
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 31 Oct 2022 16:22:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7092: Restore tests after HDFS fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19159 ) Change subject: IMPALA-7092: Restore tests after HDFS fixes .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8755/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b0bbc604601e6fab742f145c1adfb3c47b3fb6e Gerrit-Change-Number: 19159 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 16:36:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11745/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 16:40:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#5). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 10 files changed, 119 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/5 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19172 to look at the new patch set (#6). Change subject: IMPALA-7098: Re-enable tests under EC .. IMPALA-7098: Re-enable tests under EC Re-enables tests under erasure coding, or provides more specific exceptions. Erasure coding uses multiple data blocks to construct a block group. Our tests use RS-3-2-1024k, which includes 3 data blocks in a block group. Each of these blocks is sized according to `dfs.block.size`, so block groups by default hold up to 384MB of data. Impala schedules work to executors based on blocks reported by HDFS, which for EC actually represent block groups. So with default block size, a file in EC has 1/3rd the number of schedulable blocks. In the case of tpch.lineitem, this produces 2 parquet files instead of 3 and reduces the number of executors scheduled to read parquet lineitem as 1. lineitem.tbl is loaded via Hive. With EC it uses 2 block groups, without EC it uses 6 blocks. 2. parquet lineitem is created by select/insert from lineitem.tbl. Impala schedules reads to executors based on available blocks, so with EC this gets scheduled across 2 executors instead of 3 and each executor writes a separate parquet file. Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df --- M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_auto_scaling.py M tests/custom_cluster/test_query_retries.py M tests/custom_cluster/test_runtime_profile.py M tests/metadata/test_explain.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_mt_dop.py M tests/query_test/test_nested_types.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/util/auto_scaler.py 13 files changed, 31 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/19172/6 -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/19172/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19172/5//COMMIT_MSG@17 PS5, Line 17: Impala schedules work to executors based on blocks reported by HDFS, : which for EC actually represent block groups. So with default block : size, a file in EC has 1/3rd the number of schedulable blocks. > Can you add a sentence about how this specifically results in Parquet linei Done -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 17:16:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11748/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 17:25:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/19179/6/be/src/util/parquet-reader.cc File be/src/util/parquet-reader.cc: http://gerrit.cloudera.org:8080/#/c/19179/6/be/src/util/parquet-reader.cc@78 PS6, Line 78: DefaultTConfiguration()) I looked through the Clang-Tidy output, and I see this: /home/ubuntu/Impala/be/src/util/parquet-reader.cc:78:7: error: use of undeclared identifier 'DefaultTConfiguration'; did you mean 'impala::DefaultTConfiguration'? DefaultTConfiguration())); ^ impala::DefaultTConfiguration /home/ubuntu/Impala/be/src/rpc/thrift-util.h:55:49: note: 'impala::DefaultTConfiguration' declared here std::shared_ptr DefaultTConfiguration(); Since we are not in the impala namespace, the compiler wants impala::DefaultTConfiguration(). -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 18:08:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#8). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h A be/src/rpc/thrift-util.inline.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 11 files changed, 150 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/8 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10436: Support storage handler privileges for external Kudu table creation
Hello Quanlong Huang, Aman Sinha, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17640
to look at the new patch set (#10).
Change subject: IMPALA-10436: Support storage handler privileges for external
Kudu table creation
..
IMPALA-10436: Support storage handler privileges for external Kudu table
creation
This patch lowers the privilege requirement for external Kudu table
creation. Before this patch, a user was required to have the ALL
privilege on SERVER if the user wanted to create an external Kudu table.
In this patch we introduce a new type of resources called storage
handler URI and a new access type called RWSTORAGE that will be
supported by Apache Ranger once RANGER-3281 is resolved, which in turn
depends on the release of Apache Hive 4.0 that consists of HIVE-24705.
Specifically, after this patch, a user will be allowed to create an
external Kudu table as long as the user is granted the RWSTORAGE
privilege on the resource specified by a storage handler URI that points
to an existing Kudu table.
For instance, in order for a user 'non_owner' to create an external Kudu
table based on an existing Kudu table 'impala::tpch_kudu.nation', it
suffices to execute the following command as an administrator to grant
the necessary privilege to the requesting user, where "localhost" is the
default address of Kudu master host assuming there is only one single
master host in this example.
GRANT RWSTORAGE ON
STORAGEHANDLER_URI 'kudu://localhost/impala::tpch_kudu.nation'
TO USER non_owner
One may be wondering why we do not simply cancel the privilege check
that required the ALL privilege on SERVER for external Kudu table
creation. One scenario in which such relaxation is not secure is when
the owner or the creator of the existing Kudu table is different from
the requesting user who wants to create an external Kudu table in
Impala. Not requiring any additional privilege check would allow a user
without any privilege to retrieve the contents of the existing Kudu
table.
On the other hand, after this patch we still require a user to have the
ALL privilege on SERVER when the table property of
'kudu.master_addresses' is specified in a query that tries to create a
Kudu table whether or not the table is external. To be more specific,
the user 'non_owner' would be able to create an external Kudu table
using the following statement once being granted the RWSTORAGE privilege
on the specified storage handler URI above.
CREATE EXTERNAL TABLE default.kudu_tbl STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='impala::tpch_kudu.nation')
However, the following query submitted by the same user would be
rejected due to the user 'non_owner' not being granted the ALL privilege
on SERVER.
CREATE EXTERNAL TABLE default.kudu_tbl STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='impala::tpch_kudu.nation',
'kudu.master_addresses'='localhost')
We do not relax such a requirement in that specifying the addresses of
Kudu master hosts to connect should still be considered as an
administrative operation.
Testing:
- Added various FE and E2E tests to verify Impala's behavior after this
patch with respect to external Kudu table creation.
- Verified that this patch passes the core tests in the DEBUG build.
Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4
---
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java
A fe/src/main/java/org/apache/impala/analysis/StorageHandlerUri.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
A
fe/src/main/java/org/apache/impala/authorization/AuthorizableStorageHandlerUri.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M fe/src/main/java/org/apache/impala/authorization/ranger/RangerUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 6: Code-Review+2 Thanks, looks good. I know there is a test job in progress on patch set 5, and only the commit message has changed. If that passes, I'll carry the +1 verified for this patch set. -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 17:26:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/19179/6/be/src/util/parquet-reader.cc File be/src/util/parquet-reader.cc: http://gerrit.cloudera.org:8080/#/c/19179/6/be/src/util/parquet-reader.cc@78 PS6, Line 78: DefaultTConfiguration()) > I looked through the Clang-Tidy output, and I see this: I think this would reproduce locally on any Clang build, like building with ASAN. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 18:22:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 20:35:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 9: Code-Review+2 (1 comment) This makes sense to me. http://gerrit.cloudera.org:8080/#/c/19179/9/be/src/rpc/thrift-util.h File be/src/rpc/thrift-util.h: http://gerrit.cloudera.org:8080/#/c/19179/9/be/src/rpc/thrift-util.h@44 PS9, Line 44: effectife Nit: effective -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 21:38:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 10: Code-Review+2 (1 comment) Thank you Wenzhe and Joe! Carry +2 from Joe. http://gerrit.cloudera.org:8080/#/c/19179/9/be/src/rpc/thrift-util.h File be/src/rpc/thrift-util.h: http://gerrit.cloudera.org:8080/#/c/19179/9/be/src/rpc/thrift-util.h@44 PS9, Line 44: effective > Nit: effective Done -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 22:06:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19181 ) Change subject: IMPALA-11470: Add Cache For Codegen Functions .. IMPALA-11470: Add Cache For Codegen Functions The patch adds supports of the cache for CodeGen functions to improve the performance of sub-second queries. The main idea is to store the codegen functions to a cache, and reuse them when it is appropriate to save the llvm optimization time which could take over hundreds of milliseconds. In this patch, we implement the cache in a fragment-level way, that is to store all the codegen functions of a fragment in a cache entry, if one exactly same fragment comes again, it should be able to find all the codegen functions it needs from the specific cache entry, therefore saving the time. The module bitcode is used as the key to the cache, which will be generated before the module optimization and final compilation. If codegen_cache_debug_mode is set to false, which is by default, we will only store the hash code of the module bitcode as the key to reduce memory consumption. Otherwise we will store the full bitcode string. KrpcDataStreamSenderConfig hash seed is changed to a constant if using codegen cache because it can't hit the cache for the fragment if using a dynamic hash seed. As a limitation, we don't cache the udf functions, because it may rely on the shared library in runtime, if something has been changed to the loading of the shared library, for example recreating a database containing the udf, the cache could become not reusable and result in a crash if reusing it. To disable the cache for udf, we add a logic in the expression analysis, if a fragment contains a function with a non-empty hdfs location, we won't cache any codegen functions of that fragment. Tpch performance test showed no significant difference after adding this logic. The patch also introduces following new flags for start and query options. start option: - codegen_cache_capacity The capacity of the cache, if set to 0, codegen cache is disabled. query option: - disable_codegen_cache Codegen cache will be disabled is set to false. - codegen_cache_debug_mode If enabled, more logs and statistics will be allowed, and the full key content will be stored to the cache, otherwise, the cache will only store the hash code of the key to reduce memory consumption. New impalad metrics: impala.codegen-cache.misses impala.codegen-cache.entries-in-use impala.codegen-cache.entries-in-use-bytes impala.codegen-cache.entries-evicted impala.codegen-cache.hits impala.codegen-cache.entry-sizes New profile Metrics: CodegenCacheLookupTime CodegenCacheSaveTime ModuleBitcodeGenTime NumCachedFunctions TPCH-1 performance evaluation(8 iteration) on AWS m5a.4xlarge: Query Cached(s) NoCache(s) Delta(Avg) NoCodegen(s) Delta(Avg) TPCH-Q1 0.385 1.11-65.32% 5.35 -92.8% TPCH-Q2 0.561.25-55.2%0.44 27.27% TPCH-Q3 0.370.79-53.16% 0.43 -13.95% TPCH-Q4 0.355 0.48-26.04% 0.33 7.58% TPCH-Q5 0.391.18-66.95% 0.39 0% TPCH-Q6 0.240.28-14.29% 0.78 -69.23% TPCH-Q7 0.385 1.27-69.69% 0.37 4.05% TPCH-Q8 0.575 1.5 -61.67% 0.42 36.9% TPCH-Q9 0.795 1.5 -47.35% 1.04 -23.56% TPCH-Q10 0.595 1.09-45.41% 0.84 -29.17% TPCH-Q11 0.295 1.01-70.79% 0.19 55.26% TPCH-Q12 0.275 0.53-48.11% 0.38 -27.63% TPCH-Q13 1.111.2 -7.5% 1.11 0% TPCH-Q14 0.550.84-34.52% 0.42 30.95% TPCH-Q15 0.325 0.76-57.24% 0.44 -26.14% TPCH-Q16 0.320.8 -60% 0.39 -17.95% TPCH-Q17 0.555 0.9 -38.33% 0.88 -36.93% TPCH-Q18 0.535 0.94-43.09% 0.85 -37.06% TPCH-Q19 0.782.48-68.55% 0.35 122.86% TPCH-Q20 0.335 1.09-69.27% 0.29 15.52% TPCH-Q21 0.825 1.14-27.63% 0.89 -7.3% TPCH-Q22 0.255 0.51-50% 0.26 -1.92% It shows good results compared to current codegen without cache, however, as expected, for short queries, compared to codegen disabled, it is not always faster, there could be room to improve. Tests: Ran core tests. E2e testcase TestCodegenCache is added. Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 --- M be/src/codegen/CMakeLists.txt A be/src/codegen/llvm-codegen-cache.cc A be/src/codegen/llvm-codegen-cache.h M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/exec-node.cc M be/src/exprs/agg-fn.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-state.cc M
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11751/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 19:22:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 1: Code-Review+2 Thanks to fix this. It makes sense to me. -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 19:17:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 21:28:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11667: (Addendum) Remove explicit include of spring-core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19153 ) Change subject: IMPALA-11667: (Addendum) Remove explicit include of spring-core .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8756/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc9b46b45ed24e5e924db88ef792c1a68a906394 Gerrit-Change-Number: 19153 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 21:28:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19181 ) Change subject: IMPALA-11470: Add Cache For Codegen Functions .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11756/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 Gerrit-Change-Number: 19181 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 31 Oct 2022 22:55:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#9). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/common/init.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h A be/src/rpc/thrift-util.inline.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 12 files changed, 152 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/9 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7092: Restore tests after HDFS fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19159 ) Change subject: IMPALA-7092: Restore tests after HDFS fixes .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b0bbc604601e6fab742f145c1adfb3c47b3fb6e Gerrit-Change-Number: 19159 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 21:38:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#10). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/common/init.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h A be/src/rpc/thrift-util.inline.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 12 files changed, 152 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/10 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8758/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 22:42:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11754/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 20:50:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 9: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 9 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 21:03:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 10: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 21:57:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11755/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 22:18:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19181 ) Change subject: IMPALA-11470: Add Cache For Codegen Functions .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19181/2/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/19181/2/be/src/codegen/llvm-codegen.cc@1954 PS2, Line 1954: out << "Fragment Plan: " << apache::thrift::ThriftDebugString(state_->fragment()) > Please make sure you don't output sensitive data with ThriftDebugString(). The ThriftDebugString should be fine here, a similar output in previous code can be found: https://github.com/apache/impala/blob/bd88b71edd305dd448e9ec33a0cfd10f97177c4e/be/src/exec/exec-node.cc#L111. Also, it would fail the build compilation if the fragment contains HasSecret struct: https://github.com/apache/impala/blob/bd88b71edd305dd448e9ec33a0cfd10f97177c4e/be/src/util/thrift-debug-util.h#L122 -- To view, visit http://gerrit.cloudera.org:8080/19181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 Gerrit-Change-Number: 19181 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 31 Oct 2022 22:56:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19182 to look at the new patch set (#2). Change subject: IMPALA-11695: Reduce clang tidy warning output size .. IMPALA-11695: Reduce clang tidy warning output size The Clang Tidy build enables all warnings via -Wall and -Weverything. This produces enormous output. Looking at a recent failed Clang Tidy build, there are ~4.5 million warnings generated. Of these, about 4 million are from C++98 compatibility warnings. A further 250 thousand are from padding warnings. Since these are not particularly interesting, this disables both of those to reduce the output size. Disabling these warnings allowed Clang Tidy to find some issues in DataSketches that it was previously missing. Perhaps there is some limit on the number or size of warnings that it was processing. This modifies the DataSketches code to fix those (which are all minor issues with const correctness). Testing: - Built with clang tidy locally Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 --- M be/CMakeLists.txt M be/src/thirdparty/datasketches/cpc_sketch_impl.hpp M be/src/thirdparty/datasketches/kll_sketch_impl.hpp 3 files changed, 17 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19182/2 -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 20:36:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19181 ) Change subject: IMPALA-11470: Add Cache For Codegen Functions .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19181/2/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/19181/2/be/src/codegen/llvm-codegen.cc@1954 PS2, Line 1954: out << "Fragment Plan: " << apache::thrift::ThriftDebugString(state_->fragment()) Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead. -- To view, visit http://gerrit.cloudera.org:8080/19181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 Gerrit-Change-Number: 19181 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 31 Oct 2022 22:35:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11752/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 19:25:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11753/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 20:25:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8757/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 22:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 10: SslTest.TLSVersionControl fail with the following error: F1031 23:36:19.797875 12083 thrift-server.h:79] Check failed: ThriftRpcMaxMessageSize() == trans->getConfiguration()->getMaxMessageSize() (131072 vs. 1073741824) -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 03:40:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11667: (Addendum) Remove explicit include of spring-core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19153 ) Change subject: IMPALA-11667: (Addendum) Remove explicit include of spring-core .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc9b46b45ed24e5e924db88ef792c1a68a906394 Gerrit-Change-Number: 19153 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 01 Nov 2022 02:35:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11757/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 05:12:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8760/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 05:29:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8757/ -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 03:09:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. IMPALA-11695: Reduce clang tidy warning output size The Clang Tidy build enables all warnings via -Wall and -Weverything. This produces enormous output. Looking at a recent failed Clang Tidy build, there are ~4.5 million warnings generated. Of these, about 4 million are from C++98 compatibility warnings. A further 250 thousand are from padding warnings. Since these are not particularly interesting, this disables both of those to reduce the output size. Disabling these warnings allowed Clang Tidy to find some issues in DataSketches that it was previously missing. Perhaps there is some limit on the number or size of warnings that it was processing. This modifies the DataSketches code to fix those (which are all minor issues with const correctness). Testing: - Built with clang tidy locally Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Reviewed-on: http://gerrit.cloudera.org:8080/19182 Reviewed-by: Wenzhe Zhou Reviewed-by: Michael Smith Tested-by: Impala Public Jenkins --- M be/CMakeLists.txt M be/src/thirdparty/datasketches/cpc_sketch_impl.hpp M be/src/thirdparty/datasketches/kll_sketch_impl.hpp 3 files changed, 17 insertions(+), 16 deletions(-) Approvals: Wenzhe Zhou: Looks good to me, approved Michael Smith: Looks good to me, but someone else must approve Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10436: Support storage handler privileges for external Kudu table creation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Support storage handler privileges for external Kudu table creation .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8761/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 Gerrit-Change-Number: 17640 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 01 Nov 2022 05:48:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10436: Support storage handler privileges for external Kudu table creation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Support storage handler privileges for external Kudu table creation .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 Gerrit-Change-Number: 17640 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 01 Nov 2022 05:48:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10436: Support storage handler privileges for external Kudu table creation
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Support storage handler privileges for external Kudu table creation .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 Gerrit-Change-Number: 17640 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 01 Nov 2022 05:47:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11695: Reduce clang tidy warning output size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19182 ) Change subject: IMPALA-11695: Reduce clang tidy warning output size .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19182 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28c6ed1e7a4f525d81a9c48e90d051b374d44941 Gerrit-Change-Number: 19182 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 03:47:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 11: Don't understand how ThriftRpcMaxMessageSize() return 131072. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 05:41:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#11). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Refactor AssignDefaultTConfiguration to ResetMaxMessageSize. - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/common/init.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/authentication.cc M be/src/rpc/thrift-client.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h A be/src/rpc/thrift-util.inline.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 14 files changed, 156 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/11 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 11: I assumed PS10 failure is due to assignment and access of std::shared_ptr over two different thread in TAcceptQueueServer.cpp. PS11 refactor AssignDefaultTConfiguration to ResetMaxMessageSize that is simply replace the max message size instead of the whole TConfiguration. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 11 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 01 Nov 2022 04:55:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11470: Add Cache For Codegen Functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19181 ) Change subject: IMPALA-11470: Add Cache For Codegen Functions .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11742/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If42c78a7f51fd582e5fe331fead494dadf544eb1 Gerrit-Change-Number: 19181 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 31 Oct 2022 16:08:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19172 to look at the new patch set (#5). Change subject: IMPALA-7098: Re-enable tests under EC .. IMPALA-7098: Re-enable tests under EC Re-enables tests under erasure coding, or provides more specific exceptions. Erasure coding uses multiple data blocks to construct a block group. Our tests use RS-3-2-1024k, which includes 3 data blocks in a block group. Each of these blocks is sized according to `dfs.block.size`, so block groups by default hold up to 384MB of data. Impala schedules work to executors based on blocks reported by HDFS, which for EC actually represent block groups. So with default block size, a file in EC has 1/3rd the number of schedulable blocks. Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df --- M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_auto_scaling.py M tests/custom_cluster/test_query_retries.py M tests/custom_cluster/test_runtime_profile.py M tests/metadata/test_explain.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_mt_dop.py M tests/query_test/test_nested_types.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/util/auto_scaler.py 13 files changed, 31 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/19172/5 -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-7098: Re-enable tests under EC
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19172 ) Change subject: IMPALA-7098: Re-enable tests under EC .. Patch Set 5: These all pass for me with EC enabled. -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 31 Oct 2022 16:19:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19179 ) Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11746/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 31 Oct 2022 16:32:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer
Hello Quanlong Huang, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19179 to look at the new patch set (#6). Change subject: IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer .. IMPALA-11669: (addendum) Set TConfiguration in TMemoryBuffer This patch adds DefaultTConfiguration into TMemoryBuffer used within DeserializeThriftMsg, ThriftSerializer, and TSaslTransport. This patch also makes some adjustment, including: - Supply DefaultTConfiguration into the constructor of THttpTransport and TSaslTransport. - Supply DefaultTConfiguration through the constructor of TBufferedTransport. Testing: - Add DCHECK_EQ in places where we expect that it should pick up DefaultTConfiguration. - Add SerDeBuffer100MB test. - Pass core tests. - Manually run and pass test scenario described in testdata/scale_test_metadata/ both in SSL and no SSL setup. Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h A be/src/rpc/thrift-util.inline.h M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md 11 files changed, 149 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19179/6 -- To view, visit http://gerrit.cloudera.org:8080/19179 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37a8e71c64a09ec8aeccb96c6ee59ca82c0b37cb Gerrit-Change-Number: 19179 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
