[Impala-ASF-CR] IMPALA-11123: Optimize count(star) for ORC scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18327 ) Change subject: IMPALA-11123: Optimize count(star) for ORC scans .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18327 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091 Gerrit-Change-Number: 18327 Gerrit-PatchSet: 16 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 08:56:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11123: Optimize count(star) for ORC scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18327 ) Change subject: IMPALA-11123: Optimize count(star) for ORC scans .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8023/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18327 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091 Gerrit-Change-Number: 18327 Gerrit-PatchSet: 16 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 08:56:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11123: Optimize count(star) for ORC scans
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18327 ) Change subject: IMPALA-11123: Optimize count(star) for ORC scans .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18327 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091 Gerrit-Change-Number: 18327 Gerrit-PatchSet: 15 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 08:50:02 + Gerrit-HasComments: No
[native-toolchain-CR](CDH-7.1.8.x) IMPALA-11195: Disable SSL session renegotiation
Zoltan Borok-Nagy has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18382 ) Change subject: IMPALA-11195: Disable SSL session renegotiation .. IMPALA-11195: Disable SSL session renegotiation This patch disables TLS ciphers renegotiation for TLSv1.2 and prior protocol versions. Renegotiation is not possible in a TLSv1.3 connection. Addditonally it also brings Thrift 11 patches already existing in native-toolchain/master so 0005-IMPALA-11195-Disable-SSL-renegotiations.patch applies in a clean way and there's no inconsistency between the branches. In case of OpenSSL version 1.1.0h and newer, we are using SSL_OP_NO_RENEGOTIATION option to disable all renegotiations. In case of OpenSSL version prior to 1.1.0a, the undocumented flag SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS is used. The moot point is the version interval between 1.1.0a and 1.1.0g (inclusive): the SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag is no longer available from the application side, but SSL_OP_NO_RENEGOTIATION is not yet present. So, if a server binary has been compiled with OpenSSL in the specified version range, it's still advertising the renegotiation option, even if it's run against OpenSSL 1.1.0h or later versions. Change-Id: If3e12b6394134daf9c936268d4e60da6b4f3804e (cherry picked from commit be219b7c627f2db31c1928ee079160a2855ad3df) --- M buildall.sh A source/thrift/thrift-0.11.0-patches/0003-THRIFT-2087-Python-compiler-replace-non-utf-8-char-w.patch A source/thrift/thrift-0.11.0-patches/0004-THRIFT-5303-Fix-missing-error-handling-in-using-PyUn.patch A source/thrift/thrift-0.11.0-patches/0005-IMPALA-11195-Disable-SSL-renegotiations.patch A source/thrift/thrift-0.9.3-patches/0009-IMPALA-11195-Disable-SSL-renegotiations.patch 5 files changed, 165 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/82/18382/2 -- To view, visit http://gerrit.cloudera.org:8080/18382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: CDH-7.1.8.x Gerrit-MessageType: newpatchset Gerrit-Change-Id: If3e12b6394134daf9c936268d4e60da6b4f3804e Gerrit-Change-Number: 18382 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy
[native-toolchain-CR](CDH-7.1.8.x) IMPALA-11195: Disable SSL session renegotiation
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18382 Change subject: IMPALA-11195: Disable SSL session renegotiation .. IMPALA-11195: Disable SSL session renegotiation This patch disables TLS ciphers renegotiation for TLSv1.2 and prior protocol versions. Renegotiation is not possible in a TLSv1.3 connection. In case of OpenSSL version 1.1.0h and newer, we are using SSL_OP_NO_RENEGOTIATION option to disable all renegotiations. In case of OpenSSL version prior to 1.1.0a, the undocumented flag SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS is used. The moot point is the version interval between 1.1.0a and 1.1.0g (inclusive): the SSL3_FLAGS_NO_RENEGOTIATE_CIPHERS flag is no longer available from the application side, but SSL_OP_NO_RENEGOTIATION is not yet present. So, if a server binary has been compiled with OpenSSL in the specified version range, it's still advertising the renegotiation option, even if it's run against OpenSSL 1.1.0h or later versions. Change-Id: If3e12b6394134daf9c936268d4e60da6b4f3804e (cherry picked from commit be219b7c627f2db31c1928ee079160a2855ad3df) --- M buildall.sh A source/thrift/thrift-0.11.0-patches/0003-THRIFT-2087-Python-compiler-replace-non-utf-8-char-w.patch A source/thrift/thrift-0.11.0-patches/0004-THRIFT-5303-Fix-missing-error-handling-in-using-PyUn.patch A source/thrift/thrift-0.11.0-patches/0005-IMPALA-11195-Disable-SSL-renegotiations.patch A source/thrift/thrift-0.9.3-patches/0009-IMPALA-11195-Disable-SSL-renegotiations.patch 5 files changed, 165 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/82/18382/1 -- To view, visit http://gerrit.cloudera.org:8080/18382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: CDH-7.1.8.x Gerrit-MessageType: newchange Gerrit-Change-Id: If3e12b6394134daf9c936268d4e60da6b4f3804e Gerrit-Change-Number: 18382 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 13: Code-Review+1 Looks good to me -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 05 Apr 2022 09:06:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11038: Zipping unnest from view
Gabor Kaszab has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18094 ) Change subject: IMPALA-11038: Zipping unnest from view .. IMPALA-11038: Zipping unnest from view IMPALA-10920 introduced zipping unnest functionality for arrays that are in a table. This patch improves that support further by accepting inputs from views as well. Testing: - Added planner tests to verify which execution node handles the predicates on unnested items. - E2E tests for both unnesting syntaxes (select list and from clause) to cover when the source of the unnested arrays is not a table but a view. Also tested multi-level views and filtering the unnested items on different levels. Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a Reviewed-on: http://gerrit.cloudera.org:8080/18094 Tested-by: Impala Public Jenkins Reviewed-by: Gabor Kaszab --- M be/src/exec/subplan-node.cc M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/zipping-unnest.test M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-from-view.test M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-select-list.test 19 files changed, 945 insertions(+), 49 deletions(-) Approvals: Impala Public Jenkins: Verified Gabor Kaszab: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a Gerrit-Change-Number: 18094 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11038: Zipping unnest from view
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/18094 ) Change subject: IMPALA-11038: Zipping unnest from view .. Patch Set 9: Code-Review+2 Carry +2 from Csaba -- To view, visit http://gerrit.cloudera.org:8080/18094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a Gerrit-Change-Number: 18094 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 05 Apr 2022 07:56:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue There are some checks in OrcStringColumnReader::ReadValue() that we can determine outside the scope of this method. They should be optimized since this is a critical method that will be executed for each row (and for each string column). With these checks, the method is too complex to be inlined in OrcBatchedReader::ReadValueBatch() by the compiler. This patch templates OrcStringColumnReader::ReadValue() with two parameters, one for the target slot type (i.e. STRING/CHAR/VARCHAR), and the other one for whether the column is dictionary encoded. Also adds an ALWAYS_INLINE marker to force inlining it. OrcStringColumnReader::ReadValueBatch() will call a template version of ReadValue() based on the slot type and the orc batch encoded state. Ran a single node perf test on TPCH(30) on my dev box using 3 impalad instances. There are some improvements and no significant regressions: +--++-++ | Query| Avg(s) | Base Avg(s) | Delta(Avg) | +--++-++ | TPCH-Q19 | 5.62 | 6.07| I -7.41% | | TPCH-Q6 | 2.56 | 2.78| I -7.77% | | TPCH-Q4 | 3.85 | 4.25| I -9.42% | | TPCH-Q12 | 4.25 | 4.99| I -14.78% | +--++-++ Base commit: ff21728 File Format: orc/snap/block Iterations: 30 Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Reviewed-on: http://gerrit.cloudera.org:8080/18366 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h 2 files changed, 95 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 08:38:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11123: Optimize count(star) for ORC scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18327 ) Change subject: IMPALA-11123: Optimize count(star) for ORC scans .. Patch Set 16: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18327 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091 Gerrit-Change-Number: 18327 Gerrit-PatchSet: 16 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 13:27:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11123: Optimize count(star) for ORC scans
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18327 ) Change subject: IMPALA-11123: Optimize count(star) for ORC scans .. IMPALA-11123: Optimize count(star) for ORC scans This patch provides count(star) optimization for ORC scans, similar to the work done in IMPALA-5036 for Parquet scans. We use the stripes num rows statistics when computing the count star instead of materializing empty rows. The aggregate function changed from a count to a special sum function initialized to 0. This count(star) optimization is disabled for the full ACID table because the scanner might need to read and validate the 'currentTransaction' column in table's special schema. This patch drops 'parquet' from names related to the count star optimization. It also improves the count(star) operation in general by serving the result just from the file's footer stats for both Parquet and ORC. We unify the optimized count star and zero slot scan functions into HdfsColumnarScanner. The following table shows a performance comparison before and after the patch. primitive_count_star query target tpch10_parquet.lineitem table (10GB scale TPC-H). Meanwhile, count_star_parq and count_star_orc query is a modified primitive_count_star query that targets tpch_parquet.lineitem and tpch_orc_def.lineitem table accordingly. +---+--+---++-++++---++-+---+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +---+--+---++-++++---++-+---+ | tpch_parquet | count_star_parq | parquet / none / none | 0.06 | 0.07| -10.45% | 2.87%| * 25.51% * | 9 | -1.47% | -1.26 | -1.22 | | tpch_orc_def | count_star_orc | orc / def / none | 0.06 | 0.08| -22.37% | 6.22%| * 30.95% * | 9 | -1.85% | -1.16 | -2.14 | | TARGETED-PERF(10) | primitive_count_star | parquet / none / none | 0.06 | 0.08| I -30.40% | 2.68%| * 29.63% * | 9 | I -7.20% | -2.42 | -3.07 | +---+--+---++-++++---++-+---+ Testing: - Add PlannerTest.testOrcStatsAgg - Add TestAggregationQueries::test_orc_count_star_optimization - Exercise count(star) in TestOrc::test_misaligned_orc_stripes - Pass core tests Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091 Reviewed-on: http://gerrit.cloudera.org:8080/18327 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/orc-column-readers.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-upper-lower-bound-metrics.test M testdata/workloads/functional-query/queries/QueryTest/mixed-format.test A testdata/workloads/functional-query/queries/QueryTest/orc-stats-agg.test M testdata/workloads/functional-query/queries/QueryTest/parquet-stats-agg.test M testdata/workloads/functional-query/queries/QueryTest/partition-key-scans.test M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test M testdata/workloads/functional-query/queries/QueryTest/runtime_filters_mt_dop.test M testdata/workloads/functional-query/queries/QueryTest/scanners.test M tests/custom_cluster/test_executor_groups.py M tests/custom_cluster/test_query_retries.py M tests/query_test/test_aggregation.py M tests/query_test/test_scanners.py M tests/util/test_file_parser.py 31 files changed, 1,046 insertions(+), 256 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18327 To
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10398/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 05 Apr 2022 13:31:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. - Show remote UDS address for KRPC inbound connections on rpcz Web page. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 582 insertions(+), 139 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/5 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 1: Hi Csaba, thanks for picking this up! We hit similar issue in the past at IMPALA-9851, where the fix is to truncate the message to 128kb. Should we defensively add similar truncation/check in RuntimeProfileBase::AddInfoStringInternal? -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 16:54:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18383 Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. IMPALA-11023: Raise error when delete file is found in an Iceberg table Iceberg V2 DeleteFiles are skipped during scans and the whole content of the DataFiles are returned. This commit adds an extra check to prevent scanning tables that have delete files to avoid unexpected results till merge on read is supported. Testing: - Added e2e test. Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e --- M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/0-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 14 files changed, 194 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18383/1 -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate
[Impala-ASF-CR] IMPALA-11162: Support GenericUDFs for Hive
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/18295 )
Change subject: IMPALA-11162: Support GenericUDFs for Hive
..
Patch Set 4:
(35 comments)
Thanks, and sorry for the late review.
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java
File
fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java:
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java@65
PS4, Line 65: are passed in
How can they not be passed in? If you mean "are not NULL", it would be better
to write that in my opinion, because 'not passing in' a parameter suggests to
me that there is an overload that doesn't take those parameters.
Also, in checkValidFunction(), the check is always performed (regardless of the
types being NULL), isn't it?
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java@128
PS4, Line 128: getGenericUDFInstance
'constructGenericUDFInstance' would be better as we already have a
'getGenericUDFInstance' which does a very different thing (is a getter).
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java@139
PS4, Line 139: catch (InstantiationException e) {
: throw new CatalogException("Unable to call create UDF
instance.", e);
: } catch (IllegalAccessException e) {
: throw new CatalogException("Unable to call create UDF
instance.", e);
: } catch (InvocationTargetException e) {
: throw new CatalogException("Unable to call create UDF
instance.", e);
These exceptions are handled identically, so catching them in the same clause
could simplify the code (and also generate smaller bytecode):
catch
(InstantiationException|IllegalAccessException|CatalogException|CatalogException
e) {
throw new CatalogException("Unable to call create UDF instance.", e);
}
https://docs.oracle.com/javase/8/docs/technotes/guides/language/catch-multiple.html
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java@154
PS4, Line 154: !returnOI.getTypeName().equals("void")
Does it mean that if 'genericUDF_.initialize(parameterOIs)' returns 'void' we
accept it even if 'retType_' is an int or some other valid type? Shouldn't we
only accept 'void' if 'retType_' specifically indicates that we expect a void
return type? I know that 'Type' doesn't have a void value, but it could be
indicated by 'retType_' being NULL or 'retType_.isInvalid()' being true.
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutorGeneric.java
File
fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutorGeneric.java:
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutorGeneric.java@67
PS4, Line 67: @SuppressWarnings("restriction")
Just curious, why is this warning suppression needed?
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutorGeneric.java@76
PS4, Line 76:
Nit: missing 'is'.
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfExecutorGeneric.java@79
PS4, Line 79: deferredObjects_
I think 'parameters_' would be more descriptive name. 'deferredObjects_'
actually refers to the type, not what they are used for in the code. If you are
worried that it could be confused with 'inputArgs_' in the base class, the name
could also be 'deferredParameters_' or something like that.
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
File fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java@132
PS4, Line 132: " + "
Nit: no need to separate the string literals here.
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/test/java/org/apache/impala/hive/executor/TestGenericUdf.java
File fe/src/test/java/org/apache/impala/hive/executor/TestGenericUdf.java:
http://gerrit.cloudera.org:8080/#/c/18295/4/fe/src/test/java/org/apache/impala/hive/executor/TestGenericUdf.java@50
PS4, Line 50: This class is a copy of the TestGenericUdf class in the FE.
This is actually in the FE. Either this comment should be removed from this
file (and only kept in
java/test-hive-udfs/src/main/java/org/apache/impala/TestGenericUdf.java), or if
it is important that the two files be exactly the same, the comment should be
adjusted so that it is appropriate in both folders.
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/18383 )
Change subject: IMPALA-11023: Raise error when delete file is found in an
Iceberg table
..
Patch Set 1:
(1 comment)
Thanks for fixing this!
http://gerrit.cloudera.org:8080/#/c/18383/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/18383/1/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@556
PS1, Line 556: fileScanTasks
Seems like we don't need this variable, the for-loop could just remain:
for (FileScanTask task : scan.planFiles()) {
--
To view, visit http://gerrit.cloudera.org:8080/18383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e
Gerrit-Change-Number: 18383
Gerrit-PatchSet: 1
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 05 Apr 2022 13:18:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18385 Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile Before this patch the message was added to the profile in Open(), which can be called multiple times in subplans. Moved it to Close(), which is only called once in the lifetime of a Node/Aggregator. This can be also useful if async_codegen=1, as the codegen may not be finished yet during Open(). A drawback of this is that this info won't be visible when the Node is still active, but I don't think that it is a very useful info in a still running query. Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f --- M be/src/exec/aggregator.cc M be/src/exec/exec-node.cc 2 files changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/18385/1 -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10400/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 05 Apr 2022 17:02:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10399/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 05 Apr 2022 18:03:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18386 ) Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. Patch Set 1: Thank Wenzhe! -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 06 Apr 2022 00:26:05 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 24142695
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18365 ) Change subject: Bump up CDP_BUILD_NUMBER to 24142695 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10402/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5afc45a410159527e98d15ceee38e0c604b6ce73 Gerrit-Change-Number: 18365 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 06 Apr 2022 00:37:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18386 ) Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10401/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 05 Apr 2022 23:48:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 24142695
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18365 ) Change subject: Bump up CDP_BUILD_NUMBER to 24142695 .. Bump up CDP_BUILD_NUMBER to 24142695 We started to see that the E2E test of test_hive_with_ranger_setup failed due to Hive Metastore not being able to be started correctly. This patch bumps up CDP_BUILD_NUMBER associated with which the script that starts Hive could output more debugging information when Hive could not be started. Change-Id: I5afc45a410159527e98d15ceee38e0c604b6ce73 --- M bin/impala-config.sh 1 file changed, 12 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/18365/2 -- To view, visit http://gerrit.cloudera.org:8080/18365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5afc45a410159527e98d15ceee38e0c604b6ce73 Gerrit-Change-Number: 18365 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18386 Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. IMPALA-11220: Bump ORC version to 1.7.0-p8 This patch bumps ORC version to 1.7.0-p8 to contain the improvement of ORC-1137. Test: - Ran exhaustive tests. Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/18386/1 -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18386 ) Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 06 Apr 2022 00:02:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18386 ) Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8024/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 06 Apr 2022 00:26:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18386 ) Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. IMPALA-11220: Bump ORC version to 1.7.0-p8 This patch bumps ORC version to 1.7.0-p8 to contain the improvement of ORC-1137. Test: - Ran exhaustive tests. Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Reviewed-on: http://gerrit.cloudera.org:8080/18386 Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Wenzhe Zhou: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11220: Bump ORC version to 1.7.0-p8
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18386 ) Change subject: IMPALA-11220: Bump ORC version to 1.7.0-p8 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1700e8fca0d28dba0df66ee2169a4842e2280603 Gerrit-Change-Number: 18386 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 06 Apr 2022 04:55:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
lipeng...@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 4: (2 comments) done http://gerrit.cloudera.org:8080/#/c/18373/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18373/3//COMMIT_MSG@10 PS3, Line 10: fail > nit: fail Done http://gerrit.cloudera.org:8080/#/c/18373/3/fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java File fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java: http://gerrit.cloudera.org:8080/#/c/18373/3/fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java@27 PS3, Line 27: import org.slf4j.Logger; > nit: move this to line 41. We try to maintain imports in order. Done -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 02:51:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
lipeng...@sensorsdata.cn has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies When Ranger authorization is enabled, analyzing a LOAD DATA statement may fail when checking access to the source file if the HDFS permissions is more restrict than the Ranger policies. This patch provides a quick fix, when Ranger authz is enabled, we invoke method hadoop.fs.FileSystem.access(Path path, FsAction mode) to check the actual access permission first. Testing: - existing tests - add FsPermissionCheckerTest.java to fe ut Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c --- M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java A fe/src/test/java/org/apache/impala/util/FsPermissionCheckerTest.java 3 files changed, 121 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/18373/4 -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18373/3/fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java File fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java: http://gerrit.cloudera.org:8080/#/c/18373/3/fe/src/main/java/org/apache/impala/util/FsPermissionChecker.java@27 PS3, Line 27: import org.apache.hadoop.security.AccessControlException; nit: move this to line 41. We try to maintain imports in order. -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 02:19:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18373 ) Change subject: IMPALA-10272: LOAD DATA should respect Ranger-HDFS policies .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10403/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0acd83e8e237ea6a1d5fe1e9e440db0dd68b09c Gerrit-Change-Number: 18373 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Apr 2022 03:10:44 + Gerrit-HasComments: No
