[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942 (part 2): Add query hints for predicate selectivities .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12386/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 12 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 16 Feb 2023 06:43:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
wangsheng has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942 (part 2): Add query hints for predicate selectivities .. IMPALA-7942 (part 2): Add query hints for predicate selectivities Currently, Impala only uses simple estimation to compute selectivity for some predicates, and this may lead to worse query plan due to CBO. Hence, we add new hints to reduce such errors. Maybe in the future, we can use histograms to get more precise query plan. This patch adds another query hints: 'SELECTIVITY', we can use this hint to original selectivity computing. Format like this: select col from t where (a=1) /* +SELECTIVITY(0.5) */; Besides, this hint is also valid for compound predicate like this: select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */; But pay attention, if we want to use 'SELECTIVITY' hint for predicate, we need to wrap the predicate by braket, even for single binary predicate. Testing: - Added new fe tests in 'PlannerTest' - Added new fe tests in 'AnalyzeStmtsTest' for negative cases Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test 12 files changed, 334 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/12 -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 12 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18729 ) Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables .. Patch Set 34: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 Gerrit-Change-Number: 18729 Gerrit-PatchSet: 34 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 16 Feb 2023 05:31:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 16 Feb 2023 04:46:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18729 ) Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables .. Patch Set 34: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 Gerrit-Change-Number: 18729 Gerrit-PatchSet: 34 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Thu, 16 Feb 2023 04:30:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 16 Feb 2023 04:24:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Xiaoqing Gao has posted comments on this change. (
http://gerrit.cloudera.org:8080/18729 )
Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables
..
Patch Set 33:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18729/33/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18729/33/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3934
PS33, Line 3934: assigned
> nit: created
Done
http://gerrit.cloudera.org:8080/#/c/18729/33/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3940
PS33, Line 3940: List primaryColumnNames = new ArrayList<>();
: for (String primaryColumnName :
kuduTable.getPrimaryKeyColumnNames()) {
: if
(KuduUtil.getAutoIncrementingColumnName().equals(primaryColumnName)) continue;
: primaryColumnNames.add(primaryColumnName);
: }
> nit: a little optimization to avoid checking auto-incrementing column for t
Done
--
To view, visit http://gerrit.cloudera.org:8080/18729
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42
Gerrit-Change-Number: 18729
Gerrit-PatchSet: 33
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Thu, 16 Feb 2023 04:12:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Hello Quanlong Huang, Kurt Deschler, Alexey Serbin, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18729 to look at the new patch set (#34). Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables .. IMPALA-4052: CREATE TABLE LIKE for Kudu tables This commit implements cloning between Kudu tables, including clone the schema and hash partitions. But there is one limitation, cloning of Kudu tables with range paritions is not supported. For cloning range partitions, it's tracked by IMPALA-11912. Cloning Kudu tables from other types of tables is not implemented, because the table creation statements are different. Testing: - e2e tests - AnalyzeDDLTest tests Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M testdata/datasets/functional/functional_schema_template.sql A testdata/workloads/functional-query/queries/QueryTest/kudu_create_table_like_table.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 8 files changed, 232 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/18729/34 -- To view, visit http://gerrit.cloudera.org:8080/18729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 Gerrit-Change-Number: 18729 Gerrit-PatchSet: 34 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11886: Data cache should support asynchronous writes
18770832...@163.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/19475 )
Change subject: IMPALA-11886: Data cache should support asynchronous writes
..
Patch Set 4:
(9 comments)
Thanks for the code review!
http://gerrit.cloudera.org:8080/#/c/19475/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19475/4//COMMIT_MSG@24
PS4, Line 24: Testing:
> What circumstances show better performance?
How can I show that? Maybe post a record of the comparative tests? I have no
relevant experience, please give me some advice.
http://gerrit.cloudera.org:8080/#/c/19475/4/be/src/runtime/io/data-cache-test.cc
File be/src/runtime/io/data-cache-test.cc:
http://gerrit.cloudera.org:8080/#/c/19475/4/be/src/runtime/io/data-cache-test.cc@106
PS4, Line 106: while (cache.current_buffer_size_.Load() != 0) continue;
> Let's add a short sleep so that this is not spinning.
Done
http://gerrit.cloudera.org:8080/#/c/19475/4/be/src/runtime/io/data-cache.h
File be/src/runtime/io/data-cache.h:
http://gerrit.cloudera.org:8080/#/c/19475/4/be/src/runtime/io/data-cache.h@271
PS4, Line 271: uint8_t* buffer_;
> I think it would be cleaner for this to be a unique_ptr.
Did you mean using unique_ptr instead of uint8_t*? I tried but did
not reduce the amount of code, did I misunderstand something?
http://gerrit.cloudera.org:8080/#/c/19475/4/be/src/runtime/io/data-cache.h@203
PS4, Line 203:
: /// The key used for look up in the cache.
: struct CacheKey {
:public:
: explicit CacheKey(const string& filename, int64_t mtime,
int64_t offset)
: : key_(filename.size() + sizeof(mtime) + sizeof(offset)) {
: DCHECK_GE(mtime, 0);
: DCHECK_GE(offset, 0);
: key_.append(, sizeof(mtime));
: key_.append(, sizeof(offset));
: key_.append(filename);
: }
:
: int64_t Hash() const {
: return HashUtil::FastHash64(key_.data(), key_.size(), 0);
: }
:
: Slice filename() const {
: return Slice(key_.data() + OFFSETOF_FILENAME, key_.size()
- OFFSETOF_FILENAME);
: }
:
: int64_t mtime() const {
: return UNALIGNED_LOAD64(key_.data() + OFFSETOF_MTIME);
: }
:
: int64_t offset() const {
: return UNALIGNED_LOAD64(key_.data() + OFFSETOF_OFFSET);
: }
:
: Slice ToSlice() const {
: return key_;
: }
:
:private:
: // Key encoding stored in key_:
: //
: // int64_t mtime;
: // int64_t offset;
: // filename;
: static constexpr int OFFSETOF_MTIME = 0;
: static constexpr int OFFSETOF_OFFSET = OFFSETOF_MTIME +
sizeof(int64_t);
: static constexpr int OFFSETOF_FILENAME = OFFSETOF_OFFSET +
sizeof(int64_t);
: kudu::faststring key_;
: };
:
: /// The class to abstruct store behavior, including copying
the buffer and holding it
: /// until store complete.
: class StoreTask {
:public:
: /// Creating a store task requires the filename, mtime,
offset that constitutes the
: /// cache key, and the buffer and length of the cached data
is required too. We
: /// allocate a new buffer in the constructor and copy the
cache data and update
: /// total_size which keeps track of the total buffer size
allocate by all store tasks.
: explicit StoreTask(const std::string& filename, int64_t
mtime, int64_t offset,
: const uint8_t* buffer, int64_t buffer_len, AtomicInt64&
total_size);
:
: /// When the store task is destroyed, the allocated buffer
is freed and total_size is
: /// updated.
: ~StoreTask();
:
: const CacheKey& key() const { return key_; }
: const uint8_t* buffer() const { return buffer_; }
: int64_t buffer_len() const { return buffer_len_; }
:
:private:
: DISALLOW_COPY_AND_ASSIGN(StoreTask);
:
: CacheKey key_;
: uint8_t* buffer_;
: int64_t buffer_len_;
: AtomicInt64& total_size_;
: };
> Small thing: It would be nice to keep these structures defined in
> data-cache.cc. The way to do that is to have a forward
[Impala-ASF-CR] IMPALA-11886: Data cache should support asynchronous writes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19475 ) Change subject: IMPALA-11886: Data cache should support asynchronous writes .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I878f7486d485b6288de1a9145f49576b7155d312 Gerrit-Change-Number: 19475 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 16 Feb 2023 03:09:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11922 Verify JWKS URL server TLS certificate by default.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19503 ) Change subject: IMPALA-11922 Verify JWKS URL server TLS certificate by default. .. Patch Set 1: (13 comments) http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG@13 PS1, Line 13: https nit: HTTPS http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG@46 PS1, Line 46: pem nit: PEM http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG@46 PS1, Line 46: certificates Should those be CA certificates or non-CA certs (e.g., exact TLS server certificates without CA capability) are also accepted? http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/kudu/util/curl_util.h File be/src/kudu/util/curl_util.h: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/kudu/util/curl_util.h@74 PS1, Line 74: pem nit: PEM http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/kudu/util/curl_util.h@74 PS1, Line 74: certificates Are these should be CA certificates or non-CA certs are also accepted? http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/kudu/util/curl_util.h@75 PS1, Line 75: https nit: HTTPS http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/kudu/util/curl_util.cc File be/src/kudu/util/curl_util.cc: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/kudu/util/curl_util.cc@122 PS1, Line 122: CHECK_EQ Does it make sense to switch to using CURL_RETURN_NOT_OK() here instead? http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/rpc/authentication.cc File be/src/rpc/authentication.cc: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/rpc/authentication.cc@171 PS1, Line 171: jwks_insecure_tls nit: 'jwks_insecure_tls' sounds a bit vague to me: it might be authentication-only TLS channel, not verifying certs on either of the sides, using weak ciphers for the handshake, using weak ciphers to encrypt the data sent over the established channel, etc. Maybe, something like 'jwks_verify_server_tls_cert' or similar would be more descriptive of what this flag is actually for? http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util-internal.h File be/src/util/jwt-util-internal.h: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util-internal.h@374 PS1, Line 374: to trust nit: what exactly this 'trust' covers? Is this just to verify authenticity of the JWKS server's TLS certificate or the certificates in the bundle are used for something else as well? http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util.h File be/src/util/jwt-util.h: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util.h@59 PS1, Line 59: const std::string& jwks_file_path Does it make sense to make this a parameter of one of the constructors for this class and have just one Init() method with the signature Status Init(); ? http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util.h@63 PS1, Line 63: Init ditto http://gerrit.cloudera.org:8080/#/c/19503/1/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java: http://gerrit.cloudera.org:8080/#/c/19503/1/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@83 PS1, Line 83: the for ? http://gerrit.cloudera.org:8080/#/c/19503/1/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@440 PS1, Line 440: webserverTLSCert Is this certificate also has the CA capability? If not, I'm a bit surprised a non-CA certificate is accepted here. Overall, is it possible to pass here not the server's certificate as is, but the CA certificate that the server's cert is signed with? I guess that would be the expected use case in the wild, no? -- To view, visit http://gerrit.cloudera.org:8080/19503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f1e887fae39b5fb82fa9a40352e4b507b7d8d35 Gerrit-Change-Number: 19503 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 Feb 2023 03:06:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11886: Data cache should support asynchronous writes
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19475 to look at the new patch set (#5). Change subject: IMPALA-11886: Data cache should support asynchronous writes .. IMPALA-11886: Data cache should support asynchronous writes This patch implements asynchronous write to the data cache to improve scan performance when cache miss happens. Previously, writes to the data cache are synchronized with hdfs file reads, and both are handled by remote hdfs IO threads. In other words, if a cache miss occurs, the IO thread needs to take additional responsibility for cache writes, which will lead to scan performance deterioration. This patch uses a thread pool for asynchronous writes, and the number of threads in the pool is determined by the new configuration 'data_cache_num_write_threads'. In asynchronous write mode, the IO thread only needs to copy data to the temporary buffer when storing data into the data cache. The additional memory consumption caused by temporary buffers can be limited, depending on the new configuration 'data_cache_write_buffer_limit'. Testing: - Add test cases for asynchronous data writing to the original DataCacheTest using different number of threads. - Add DataCacheTest,#OutOfWriteBufferLimit Used to test the limit of memory consumed by temporary buffers in the case of asynchronous writes Change-Id: I878f7486d485b6288de1a9145f49576b7155d312 --- M be/src/runtime/io/data-cache-test.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/data-cache.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/metrics.json 6 files changed, 366 insertions(+), 69 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/19475/5 -- To view, visit http://gerrit.cloudera.org:8080/19475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I878f7486d485b6288de1a9145f49576b7155d312 Gerrit-Change-Number: 19475 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11713: Switch to C++17
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19183 ) Change subject: IMPALA-11713: Switch to C++17 .. Patch Set 5: Rebased, will do a performance run -- To view, visit http://gerrit.cloudera.org:8080/19183 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadac41817fe5eaaa469a5f0e9f94056a409c14b9 Gerrit-Change-Number: 19183 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 16 Feb 2023 01:29:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11922 Verify JWKS URL server TLS certificate by default.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19503 ) Change subject: IMPALA-11922 Verify JWKS URL server TLS certificate by default. .. Patch Set 1: (7 comments) Thanks to work on this and added certificate verification to kudu::EasyCurl. curl_util.h/curl_util.cc were synced from Kudu repo, we need to port these changes to Kudu repo later. http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG@12 PS1, Line 12: nit: one extra space http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG@24 PS1, Line 24: nit: one extra space http://gerrit.cloudera.org:8080/#/c/19503/1//COMMIT_MSG@48 PS1, Line 48: add a Testing section http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/rpc/authentication.cc File be/src/rpc/authentication.cc: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/rpc/authentication.cc@1418 PS1, Line 1418: } Check jwks_ca_certificate is not empty if jwks_insecure_tls is set as false. http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util.h File be/src/util/jwt-util.h: http://gerrit.cloudera.org:8080/#/c/19503/1/be/src/util/jwt-util.h@64 PS1, Line 64: bool is_local_file do we still need this variable? http://gerrit.cloudera.org:8080/#/c/19503/1/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java: http://gerrit.cloudera.org:8080/#/c/19503/1/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@397 PS1, Line 397: It's better to give a certificate which does not match the certificate returned from server. http://gerrit.cloudera.org:8080/#/c/19503/1/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@420 PS1, Line 420: nit: extra spaces -- To view, visit http://gerrit.cloudera.org:8080/19503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f1e887fae39b5fb82fa9a40352e4b507b7d8d35 Gerrit-Change-Number: 19503 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 16 Feb 2023 00:05:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11920: [DOCS] Cleanup and update spill examples
Shajini Thayasingh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19504 ) Change subject: IMPALA-11920: [DOCS] Cleanup and update spill examples .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I624a607bda33ab47100e1540ff1d66c8d19a7329 Gerrit-Change-Number: 19504 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 23:38:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 23:39:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9052/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 23:34:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: [DOCS] Cleanup and update spill examples
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19504 ) Change subject: IMPALA-11920: [DOCS] Cleanup and update spill examples .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/700/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/19504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I624a607bda33ab47100e1540ff1d66c8d19a7329 Gerrit-Change-Number: 19504 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 23:30:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Michael Smith has removed Shajini Thayasingh from this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Removed reviewer Shajini Thayasingh. -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Hello Yida Wu, Shajini Thayasingh, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19496 to look at the new patch set (#5). Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. IMPALA-11920: Support spill to HDFS address by service ID Allows addressing HDFS (and Ozone) filesystems in `scratch_dirs` by a service identifier that doesn't include a port number. Examples - "hdfs://hdfs1/:10G" uses the root directory of HDFS with a 10G limit - "ofs://ozone1/tmp::" uses /tmp in Ozone with default limit/priority Updates `scratch_dirs` parsing to allow whitespace after each specifier, as in "hfs://hdfs1/ , /tmp". This is unambiguous and avoids failures for simple mistakes. Testing: - new backend test cases run with HDFS and Ozone - manually tested that Impala starts with --impalad_args=--scratch_dirs=ofs://localhost/tmp,/tmp creates impala-scratch in both locations Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc 2 files changed, 57 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/19496/5 -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11920: [DOCS] Cleanup and update spill examples
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19504 Change subject: IMPALA-11920: [DOCS] Cleanup and update spill examples .. IMPALA-11920: [DOCS] Cleanup and update spill examples Updates documentation to include examples with service identifier. Also fixes inconsistent use of ASCII quotes for example text, highlighting code and variable names, and normalizes descriptions between S3/HDFS/Ozone. Removes "priority" from remote descriptions as it is optional and does nothing. Change-Id: I624a607bda33ab47100e1540ff1d66c8d19a7329 --- M docs/topics/impala_disk_space.xml 1 file changed, 96 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/19504/1 -- To view, visit http://gerrit.cloudera.org:8080/19504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I624a607bda33ab47100e1540ff1d66c8d19a7329 Gerrit-Change-Number: 19504 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Patch Set 4: I think putting docs in the same patch as code changes is confusing the Jenkins auto-review jobs. Should I split them up? -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 23:17:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: [DOCS] Cleanup and update spill examples
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19504 ) Change subject: IMPALA-11920: [DOCS] Cleanup and update spill examples .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/700/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/19504 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I624a607bda33ab47100e1540ff1d66c8d19a7329 Gerrit-Change-Number: 19504 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 Feb 2023 23:19:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11922 Verify JWKS URL server TLS certificate by default.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19503 ) Change subject: IMPALA-11922 Verify JWKS URL server TLS certificate by default. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12382/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5f1e887fae39b5fb82fa9a40352e4b507b7d8d35 Gerrit-Change-Number: 19503 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 15 Feb 2023 23:14:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9051/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 15 Feb 2023 23:14:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11922 Verify JWKS URL server TLS certificate by default.
Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19503 Change subject: IMPALA-11922 Verify JWKS URL server TLS certificate by default. .. IMPALA-11922 Verify JWKS URL server TLS certificate by default. BREAKING CHANGE JWT Auth has an option to specify the location of the JSON Web Key Set (JWKS) using a URL. If that URL is accessed over https, the TLS certificate presented by the server is not verified. This means that Impala only requires the server to return a TLS certificate, whether or not Impala trusts the signing certificate chain. The implications of this setup is that a fully secure chain of trust cannot be established throughout the entire JWT authentication lifecycle and thus creates an attack vector where a bad actor could trick Impala into trusting an actor-controlled JWKS. The bad actor can then generate a JWT with any claims they chose and Impala will accept it. This change introduces: 1. verification of JWKS server TLS certificate by default 2. jwks_insecure_tls Impala startup flag 3. jwks_ca_certificate Impala startup flag 1. While previously, the JWKS URL was always called without verifying its TLS certificate, the default is to now to verify that cert. Thus, any cases where the JWKS was retrieved from an untrusted URL will now cause Impala to fail to start. 2. The new flag jwks_insecure_tls controls whether or not Impala verifies the TLS certificate presented by the JWKS server. It defaults to "false" meaning that the certificate will be verified. Setting this value to "true" will restore the previous behavior where untrusted TLS certificates are accepted. 3. The new flag jwks_ca_certificate enables specifying a pem bundle of certificates to trust when calling to the JWKS URL. Change-Id: I5f1e887fae39b5fb82fa9a40352e4b507b7d8d35 --- M be/src/kudu/util/curl_util.cc M be/src/kudu/util/curl_util.h M be/src/rpc/authentication.cc M be/src/service/impala-server.cc M be/src/util/jwt-util-internal.h M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc M be/src/util/jwt-util.h M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java 9 files changed, 200 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/19503/1 -- To view, visit http://gerrit.cloudera.org:8080/19503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5f1e887fae39b5fb82fa9a40352e4b507b7d8d35 Gerrit-Change-Number: 19503 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 48: (1 comment) http://gerrit.cloudera.org:8080/#/c/19033/48/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/19033/48/fe/src/main/java/org/apache/impala/planner/Planner.java@470 PS48, Line 470: blockingAwareCost nit: this result seems not used now. Add "TODO" comment -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 48 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 15 Feb 2023 21:00:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 15 Feb 2023 20:11:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
eubn...@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 7: I didn't notice that "commons-io.version" has been missing since IMPALA-11670. (https://github.com/apache/impala/commit/83c5e6e4098d8ed75de09a7e228d6ef10de2ee12) I fixed this error by adding it. -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 15 Feb 2023 20:09:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12380/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 15 Feb 2023 20:04:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
eubn...@gmail.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. IMPALA-10408: Support build using Apache components Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 --- M README-build.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M buildall.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M java/TableFlattener/pom.xml M java/pom.xml 9 files changed, 157 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/18977/7 -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
eubn...@gmail.com has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. IMPALA-10408: Support build using Apache components Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 --- M README-build.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M buildall.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M java/TableFlattener/pom.xml M java/pom.xml 9 files changed, 157 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/18977/6 -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18977/6/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/18977/6/bin/bootstrap_toolchain.py@614 PS6, Line 614: o flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 15 Feb 2023 19:45:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 48: (4 comments) http://gerrit.cloudera.org:8080/#/c/19033/48//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19033/48//COMMIT_MSG@325 PS48, Line 325: IMPALA-2805 As the comments in https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/Expr.java#L79-L81, it seems relative costs defined in IMPALA-2805 are not accurate. We may need to tune the numbers a little. http://gerrit.cloudera.org:8080/#/c/19033/48//COMMIT_MSG@346 PS48, Line 346: Testing: Could you attach the bench mark which show effective parallelism improvement? http://gerrit.cloudera.org:8080/#/c/19033/43/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java: http://gerrit.cloudera.org:8080/#/c/19033/43/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@263 PS43, Line 263: return deferredBatchQueueSize; > I intended this to be a serialization/deserialization cost per row. This assume the total cost for a row batch is 1. Is it right estimation? http://gerrit.cloudera.org:8080/#/c/19033/48/fe/src/main/java/org/apache/impala/planner/ScanNode.java File fe/src/main/java/org/apache/impala/planner/ScanNode.java: http://gerrit.cloudera.org:8080/#/c/19033/48/fe/src/main/java/org/apache/impala/planner/ScanNode.java@359 PS48, Line 359: 1.0f / getRowBatchSize(queryOptions); In ExchangeNode.estimateProcessingCostPerRow(), the cost per row is calculated as 1 / (getRowBatchSize(queryOptions) / avg-row-size). Show we do same? -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 48 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 15 Feb 2023 19:39:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19499 ) Change subject: IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53238612f4037572abb6d2cc913dd74ee830a9c9 Gerrit-Change-Number: 19499 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 15 Feb 2023 19:22:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/18729 )
Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables
..
Patch Set 33:
(2 comments)
Thanks to fix the issue for non unique primary key. It looks good to me, only a
few nits.
http://gerrit.cloudera.org:8080/#/c/18729/33/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18729/33/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3934
PS33, Line 3934: assigned
nit: created
http://gerrit.cloudera.org:8080/#/c/18729/33/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3940
PS33, Line 3940: List primaryColumnNames = new ArrayList<>();
: for (String primaryColumnName :
kuduTable.getPrimaryKeyColumnNames()) {
: if
(KuduUtil.getAutoIncrementingColumnName().equals(primaryColumnName)) continue;
: primaryColumnNames.add(primaryColumnName);
: }
nit: a little optimization to avoid checking auto-incrementing column for table
with unique key.
List primaryColumnNames = new
ArrayList(kuduTable.getPrimaryKeyColumnNames());
if (kuduTable.hasAutoIncrementingColumn()) {
primaryColumnNames.remove(KuduUtil.getAutoIncrementingColumnName());
}
--
To view, visit http://gerrit.cloudera.org:8080/18729
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42
Gerrit-Change-Number: 18729
Gerrit-PatchSet: 33
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Wed, 15 Feb 2023 18:20:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Hello Yida Wu, Shajini Thayasingh, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19496 to look at the new patch set (#4). Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. IMPALA-11920: Support spill to HDFS address by service ID Allows addressing HDFS (and Ozone) filesystems in `scratch_dirs` by a service identifier that doesn't include a port number. Examples - "hdfs://hdfs1/:10G" uses the root directory of HDFS with a 10G limit - "ofs://ozone1/tmp::" uses /tmp in Ozone with default limit/priority Updates `scratch_dirs` parsing to allow whitespace after each specifier, as in "hfs://hdfs1/ , /tmp". This is unambiguous and avoids failures for simple mistakes. Updates documentation to include examples with service identifier. Also fixes inconsistent use of ASCII quotes for example text, highlighting code and variable names, and normalizes descriptions between S3/HDFS/Ozone. Removes "priority" from remote descriptions as it is optional and does nothing. Testing: - new backend test cases run with HDFS and Ozone - manually tested that Impala starts with --impalad_args=--scratch_dirs=ofs://localhost/tmp,/tmp creates impala-scratch in both locations Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M docs/topics/impala_disk_space.xml 3 files changed, 153 insertions(+), 91 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/19496/4 -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/19496/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19496/3//COMMIT_MSG@17 PS3, Line 17: priority > Just to confirm that the priority is removed in the document for the remote Yes, it'll still be parsed in the code. http://gerrit.cloudera.org:8080/#/c/19496/3/be/src/runtime/tmp-file-mgr-test.cc File be/src/runtime/tmp-file-mgr-test.cc: http://gerrit.cloudera.org:8080/#/c/19496/3/be/src/runtime/tmp-file-mgr-test.cc@2240 PS3, Line 2240: hdfs > Maybe it is not related, but when I test with a space behind the url like - Ok. That happens because https://github.com/apache/impala/blob/feb4a76ed4cb5b688143eb21370f78ec93133c56/be/src/runtime/tmp-file-mgr.cc#L290 only does trim left. trim_copy would be fine there. http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml File docs/topics/impala_disk_space.xml: http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@298 PS3, Line 298: /dir1::0 > The alignment looks weird compared to the previous version, does this part It fixes the indentation in https://impala.apache.org/docs/build/asf-site-html/topics/impala_disk_space.html. All whitespace is included in a codeblock, and having the examples indented looks wrong to me. http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@486 PS3, Line 486: ozone1 > Just a question that what is the difference between Ozone service identifie Ozone Manager might be addressed via a hostname. https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/feature/om-ha.html discusses service IDs for an HA deployment. http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@488 PS3, Line 488: > nit. looks good to have a comma here Ack -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 17:50:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11802: Optimize count(*) queries for Iceberg V2 position delete tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19494 ) Change subject: IMPALA-11802: Optimize count(*) queries for Iceberg V2 position delete tables .. Patch Set 2: Code-Review+1 (2 comments) Awesome work, Penglin! Thank you so much? http://gerrit.cloudera.org:8080/#/c/19494/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19494/2//COMMIT_MSG@31 PS2, Line 31: ArithmeticExpr(ADD) Nice! This plan is much better than the one that I proposed in the description! http://gerrit.cloudera.org:8080/#/c/19494/2//COMMIT_MSG@44 PS2, Line 44: Testing: Could you please add a planner test for select count(*) from iceberg_v2_positional_not_all_data_files_have_delete_files; -- To view, visit http://gerrit.cloudera.org:8080/19494 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8172c805121bf91d23fe063f806493afe2f03d41 Gerrit-Change-Number: 19494 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Xiaoqing Gao Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 15 Feb 2023 16:21:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19499 ) Change subject: IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12379/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53238612f4037572abb6d2cc913dd74ee830a9c9 Gerrit-Change-Number: 19499 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 15 Feb 2023 14:38:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19499 ) Change subject: IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19499/1/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java File fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java: http://gerrit.cloudera.org:8080/#/c/19499/1/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java@232 PS1, Line 232: UdfExecutor createUdfExecutor(String jarFile, String udfClassPath, Type retType, > originalArgs is missing from the Javadoc comment Done -- To view, visit http://gerrit.cloudera.org:8080/19499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53238612f4037572abb6d2cc913dd74ee830a9c9 Gerrit-Change-Number: 19499 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 15 Feb 2023 14:21:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19499 ) Change subject: IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9050/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53238612f4037572abb6d2cc913dd74ee830a9c9 Gerrit-Change-Number: 19499 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 15 Feb 2023 14:22:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11920: Support spill to HDFS address by service ID
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19496 ) Change subject: IMPALA-11920: Support spill to HDFS address by service ID .. Patch Set 3: (6 comments) Thanks for the fix, it looks good, just some questions to understand the change. http://gerrit.cloudera.org:8080/#/c/19496/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19496/3//COMMIT_MSG@17 PS3, Line 17: priority Just to confirm that the priority is removed in the document for the remote scratch directory, however, can still be parsed in the code. http://gerrit.cloudera.org:8080/#/c/19496/3/be/src/runtime/tmp-file-mgr-test.cc File be/src/runtime/tmp-file-mgr-test.cc: http://gerrit.cloudera.org:8080/#/c/19496/3/be/src/runtime/tmp-file-mgr-test.cc@2240 PS3, Line 2240: hdfs Maybe it is not related, but when I test with a space behind the url like --impalad_args='--scratch_dirs="hdfs://localhost:8020/ , /tmp"', it will try to create a path on " hdfs://localhost:8020/ /impala-scratch", could you please take a look on the rtrim of the path and add a testcase for it if it is an issue? http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml File docs/topics/impala_disk_space.xml: http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@298 PS3, Line 298: /dir1::0 The alignment looks weird compared to the previous version, does this part show normally? http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@303 PS3, Line 303: /dir1:200GB : /dir1:200GB: Same as above http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@486 PS3, Line 486: ozone1 Just a question that what is the difference between Ozone service identifier and Ozone Manager? Is it no difference for the Impala to parse the Ozone scratch path? http://gerrit.cloudera.org:8080/#/c/19496/3/docs/topics/impala_disk_space.xml@488 PS3, Line 488: nit. looks good to have a comma here -- To view, visit http://gerrit.cloudera.org:8080/19496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167 Gerrit-Change-Number: 19496 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Shajini Thayasingh Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 15 Feb 2023 10:53:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18729 ) Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables .. Patch Set 33: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12378/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 Gerrit-Change-Number: 18729 Gerrit-PatchSet: 33 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 15 Feb 2023 10:15:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Xiaoqing Gao has posted comments on this change. (
http://gerrit.cloudera.org:8080/18729 )
Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables
..
Patch Set 32:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18729/32/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18729/32/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3930
PS32, Line 3930:
createTableParams.setPrimary_key_column_names(kuduTable.getPrimaryKeyColumnNames());
: List columns = new ArrayList<>();
: for (Column col : kuduTable.getColumns()) {
: columns.add(col.toThrift());
: }
> IMPALA-11809 was merged recently, which added support for non unique primar
I fixed it and added some tests. Thanks for the reminder.
--
To view, visit http://gerrit.cloudera.org:8080/18729
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42
Gerrit-Change-Number: 18729
Gerrit-PatchSet: 32
Gerrit-Owner: Xiaoqing Gao
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Xiaoqing Gao
Gerrit-Comment-Date: Wed, 15 Feb 2023 09:59:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Hello Quanlong Huang, Kurt Deschler, Alexey Serbin, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18729 to look at the new patch set (#33). Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables .. IMPALA-4052: CREATE TABLE LIKE for Kudu tables This commit implements cloning between Kudu tables, including clone the schema and hash partitions. But there is one limitation, cloning of Kudu tables with range paritions is not supported. For cloning range partitions, it's tracked by IMPALA-11912. Cloning Kudu tables from other types of tables is not implemented, because the table creation statements are different. Testing: - e2e tests - AnalyzeDDLTest tests Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 --- M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M testdata/datasets/functional/functional_schema_template.sql A testdata/workloads/functional-query/queries/QueryTest/kudu_create_table_like_table.test M tests/custom_cluster/test_kudu.py M tests/query_test/test_kudu.py 8 files changed, 232 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/18729/33 -- To view, visit http://gerrit.cloudera.org:8080/18729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 Gerrit-Change-Number: 18729 Gerrit-PatchSet: 33 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiaoqing Gao
[Impala-ASF-CR] IMPALA-11803: Impalad hit DCHECK when running union on empty table with MT DOP>1
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19474 ) Change subject: IMPALA-11803: Impalad hit DCHECK when running union on empty table with MT_DOP>1 .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/19474/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19474/2//COMMIT_MSG@7 PS2, Line 7: Impalad hit DCHECK when running union on : empty table with MT_DOP>1 > I've updated the description, should I change the title of the jira as well No, we just need to change the commit title. The JIRA title is ok to just describe the problem. The commit title would be nice to focus on how the issue is fixed, e.g. "Fix hitting DCHECK when running union on empty table with MT_DOP>1" BTW, the title should be in one line. http://gerrit.cloudera.org:8080/#/c/19474/2//COMMIT_MSG@11 PS2, Line 11: w > Done Not done yet in other lines.. http://gerrit.cloudera.org:8080/#/c/19474/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/19474/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@2095 PS3, Line 2095: useMtScanNode_ = queryOptions.mt_dop > 0; Could you add a comment like this? So we won't make the same mistake when adding codes in the future. // update 'useMtScanNode_' before any return cases. It's used in BE. -- To view, visit http://gerrit.cloudera.org:8080/19474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idbae5e1a78211327a214b2d936743bda767ae3c4 Gerrit-Change-Number: 19474 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 15 Feb 2023 08:56:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs
Peter Rozsa has posted comments on this change. (
http://gerrit.cloudera.org:8080/19499 )
Change subject: IMPALA-11911: Fix NULL argument handling in Hive GenericUDFs
..
Patch Set 1: Code-Review+1
(1 comment)
Nice!
http://gerrit.cloudera.org:8080/#/c/19499/1/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java
File fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java:
http://gerrit.cloudera.org:8080/#/c/19499/1/fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java@232
PS1, Line 232: Object originalArgs[], Object args[]) throws
ImpalaException, TException {
originalArgs is missing from the Javadoc comment
--
To view, visit http://gerrit.cloudera.org:8080/19499
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53238612f4037572abb6d2cc913dd74ee830a9c9
Gerrit-Change-Number: 19499
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Wed, 15 Feb 2023 08:27:17 +
Gerrit-HasComments: Yes
