[native-toolchain-CR] IMPALA-9999: Build the toolchain with GCC 10
Joe McDonnell has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18806 ) Change subject: IMPALA-: Build the toolchain with GCC 10 .. IMPALA-: Build the toolchain with GCC 10 This switches the toolchain to build with GCC 10. This required several version changes and patches due to new warnings or changes in default settings: 1. LLVM is patched to fix compilation in lli (type mismatch on char) 2. crcutil is upgraded to the latest to handle the two digit GCC version. The new version incorporates the two patches previously used. 3. libunwind is upgraded to 1.5.1 due to GCC 10 switch to -fno-common by default. The new version incorporates a couple of the previous patches. 4. Breakpad is upgraded to a recent commit (from late March 2022), which uses a newer version of lss that can compile with GCC 10. 5. Flatbuffers is upgraded to 1.12.0, which fixes a constness casting issue 6. The TPC-DS library is patched to use -fcommon, as GCC 10 switched to -fno-common by default. 7. GDB fails to compile on ARM due to the switch to -fno-common, so this adds newer GDB versions without that issue. This upgrade GCC to 10.4 and binutils to 2.35.1. This changes GCC's build command to build GCC using link time optimization. This should speed up GCC's execution, but it has no other impact. Building GCC with LTO requires GCC compilation to use an updated binutils, so this modifies the build script to build binutils first and use that for GCC. GCC 10.4 requires a patch to libstdc++ to fix missing noexcepts, otherwise Clang compilation of Impala fails. Testing: - This has built on all supported platforms, and Impala can build and operate with this toolchain. Change-Id: I2fbfcdcb497ad60772bf0a7837e4043ae75bcfaa --- M buildall.sh M functions.sh M init-compiler.sh M init.sh A source/breakpad/breakpad-e09741c609dcd5f5274d40182c5e2cc9a002d5ba-patches/0001-Add-basic-support-for-dwz-dwarf-extension.patch A source/breakpad/breakpad-e09741c609dcd5f5274d40182c5e2cc9a002d5ba-patches/0002-Build-breakpad-e09741c6-on-ppc64le.patch M source/gcc/build.sh A source/gcc/gcc-10.4.0-patches/0001-libstdc-Fix-inconsistent-noexcept-specific-for-valar.patch M source/gdb/build.sh A source/libunwind/libunwind-1.5.0-patches/0001-libunwind-trace-cache-destructor.patch A source/llvm/llvm-5.0.1-patches/0005-PATCH-Fix-lli-compilation-on-gcc8.patch A source/tpc-ds/tpc-ds-2.1.0-patches/0001-PATCH-Fix-compilation-on-gcc10-fcommon.patch 12 files changed, 710 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/06/18806/5 -- To view, visit http://gerrit.cloudera.org:8080/18806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2fbfcdcb497ad60772bf0a7837e4043ae75bcfaa Gerrit-Change-Number: 18806 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-10262: RPM/DEB Packaging Support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18939 ) Change subject: IMPALA-10262: RPM/DEB Packaging Support .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11351/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64419fd400fe8d233dac016b6306157fe9461d82 Gerrit-Change-Number: 18939 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 14 Sep 2022 03:44:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18978 ) Change subject: IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18978 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie04ff243c6b82c1a06c489da74353f2d8afe423a Gerrit-Change-Number: 18978 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 14 Sep 2022 03:40:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18940 ) Change subject: IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11350/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d05bb4ec1f70f5387bb21fbe23f62c05941af18 Gerrit-Change-Number: 18940 Gerrit-PatchSet: 6 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 14 Sep 2022 03:25:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10262: RPM/DEB Packaging Support
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18939 ) Change subject: IMPALA-10262: RPM/DEB Packaging Support .. Patch Set 3: (7 comments) Thanks for your feedback! http://gerrit.cloudera.org:8080/#/c/18939/2/CMakeLists.txt File CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/18939/2/CMakeLists.txt@27 PS2, Line 27: option(BUILD_PACKAGES "Build deployment packages") > typo in 'installater'? changed to "deployment". http://gerrit.cloudera.org:8080/#/c/18939/2/CMakeLists.txt@608 PS2, Line 608: > Is this will strip the final 'bin/impalad'? can we make it optional? Yeah, added an env STRIP_DEPLOYMENT_IMPALAD to control this. http://gerrit.cloudera.org:8080/#/c/18939/2/bin/deploy/common.sh File bin/deploy/common.sh: http://gerrit.cloudera.org:8080/#/c/18939/2/bin/deploy/common.sh@1 PS2, Line 1: > Is it better to rename it to impala-env.sh? make it's function more clear, Yeah, make sense. http://gerrit.cloudera.org:8080/#/c/18939/2/bin/deploy/start-catalogd.sh File bin/deploy/start-catalogd.sh: http://gerrit.cloudera.org:8080/#/c/18939/2/bin/deploy/start-catalogd.sh@29 PS2, Line 29: > maybe better to assign IMPALA_HOME to current directory using '$(cd $(dirna I tend to make IMPALA_HOME configurable so if you want to deploy different versions of Impala, you can just switch to a new location. http://gerrit.cloudera.org:8080/#/c/18939/2/bin/deploy/start-catalogd.sh@31 PS2, Line 31: > I have some suggestions here: These are pretty good points! But they are feature requests. I think we can add them in follow-up JIRAs so other guys can also join this work. http://gerrit.cloudera.org:8080/#/c/18939/2/fe/src/test/resources/deploy/catalogd_flags File fe/src/test/resources/deploy/catalogd_flags: PS2: > files below 'fe/src/test/resources/deploy' have little relation to fe, mayb Done http://gerrit.cloudera.org:8080/#/c/18939/2/fe/src/test/resources/deploy/core-site.xml File fe/src/test/resources/deploy/core-site.xml: http://gerrit.cloudera.org:8080/#/c/18939/2/fe/src/test/resources/deploy/core-site.xml@1 PS2, Line 1: > can we read xxx-site.xml by envirement variables first? I'm not sure if users need different sets of configuration in practise. Let's support this in a follow-up JIRA as well. -- To view, visit http://gerrit.cloudera.org:8080/18939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64419fd400fe8d233dac016b6306157fe9461d82 Gerrit-Change-Number: 18939 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 14 Sep 2022 03:24:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10262: RPM/DEB Packaging Support
Hello Impala Public Jenkins, Xiang Yang, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18939 to look at the new patch set (#3). Change subject: IMPALA-10262: RPM/DEB Packaging Support .. IMPALA-10262: RPM/DEB Packaging Support This patch bases on a previous patch contributed by Shant Hovsepian: https://gerrit.cloudera.org/c/16612/ It adds a new option, -package, to buildall.sh for building a package for the current OS type (e.g. CentOS/Ubuntu). You can also use "make/ninja package" to build the package. Scripts for launching the services and the required configuration files are also added. Tests: - Built on Ubuntu 16.04 and CentOS 7.5 using ./buildall.sh -noclean -skiptests -ninja -release -package - Deployed the RPM package on a CDP cluster. Verifed the scripts. Change-Id: I64419fd400fe8d233dac016b6306157fe9461d82 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/service/CMakeLists.txt M bin/bootstrap_system.sh M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M bin/rat_exclude_files.txt M buildall.sh A package/bin/impala-env.sh A package/bin/start-catalogd.sh A package/bin/start-impalad.sh A package/bin/start-statestore.sh A package/conf/catalogd_flags A package/conf/core-site.xml A package/conf/fair-scheduler.xml A package/conf/hdfs-site.xml A package/conf/hive-site.xml A package/conf/impalad_flags A package/conf/llama-site.xml A package/conf/statestore_flags 20 files changed, 352 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/18939/3 -- To view, visit http://gerrit.cloudera.org:8080/18939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I64419fd400fe8d233dac016b6306157fe9461d82 Gerrit-Change-Number: 18939 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES
Baike Xia has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18940 ) Change subject: IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES .. IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES Add TBLPROPERTIES support to the view, here are some examples: CREATE VIEW [IF NOT EXISTS] [database_name.]view_name [(column_name [COMMENT 'column_comment'][, ...])] [COMMENT 'view_comment'] [TBLPROPERTIES (property_name = property_value, ...)] AS select_statement; ALTER VIEW [database_name.]view_name SET TBLPROPERTIES (property_name = property_value, ...); ALTER VIEW [database_name.]view_name UNSET TBLPROPERTIES (property_name, ...); Change-Id: I8d05bb4ec1f70f5387bb21fbe23f62c05941af18 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/AlterViewSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java A fe/src/main/java/org/apache/impala/analysis/AlterViewUnSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java 12 files changed, 334 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/18940/6 -- To view, visit http://gerrit.cloudera.org:8080/18940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8d05bb4ec1f70f5387bb21fbe23f62c05941af18 Gerrit-Change-Number: 18940 Gerrit-PatchSet: 6 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18940 ) Change subject: IMPALA-11420: Support CREATE/ALTER VIEW SET/UNSET TBLPROPERTIES .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8569/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d05bb4ec1f70f5387bb21fbe23f62c05941af18 Gerrit-Change-Number: 18940 Gerrit-PatchSet: 6 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 14 Sep 2022 03:05:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11557: Fix memory leak in BlockingRowBatchQueue
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18960 ) Change subject: IMPALA-11557: Fix memory leak in BlockingRowBatchQueue .. Patch Set 2: Code-Review+2 Carry Csaba's +1. Thanks for fixing this, Xianqing! -- To view, visit http://gerrit.cloudera.org:8080/18960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I656316b6575ce74a03b83fcd45e772c763835d56 Gerrit-Change-Number: 18960 Gerrit-PatchSet: 2 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 14 Sep 2022 02:04:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11557: Fix memory leak in BlockingRowBatchQueue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18960 ) Change subject: IMPALA-11557: Fix memory leak in BlockingRowBatchQueue .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I656316b6575ce74a03b83fcd45e772c763835d56 Gerrit-Change-Number: 18960 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 14 Sep 2022 02:05:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11557: Fix memory leak in BlockingRowBatchQueue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18960 ) Change subject: IMPALA-11557: Fix memory leak in BlockingRowBatchQueue .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8568/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I656316b6575ce74a03b83fcd45e772c763835d56 Gerrit-Change-Number: 18960 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 14 Sep 2022 02:05:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11578: Exclude locality test for remote FS
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18980 ) Change subject: IMPALA-11578: Exclude locality test for remote FS .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 Gerrit-Change-Number: 18980 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 14 Sep 2022 01:00:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18978 ) Change subject: IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8567/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18978 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie04ff243c6b82c1a06c489da74353f2d8afe423a Gerrit-Change-Number: 18978 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 13 Sep 2022 22:36:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11578: Exclude locality test for remote FS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18980 ) Change subject: IMPALA-11578: Exclude locality test for remote FS .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11349/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 Gerrit-Change-Number: 18980 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 13 Sep 2022 21:28:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11578: Exclude locality test for remote FS
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18980 ) Change subject: IMPALA-11578: Exclude locality test for remote FS .. Patch Set 2: Running the test suite with Ozone. I'll update when it passes. -- To view, visit http://gerrit.cloudera.org:8080/18980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 Gerrit-Change-Number: 18980 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 13 Sep 2022 21:10:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11578: Exclude locality test for remote FS
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18980 to look at the new patch set (#2). Change subject: IMPALA-11578: Exclude locality test for remote FS .. IMPALA-11578: Exclude locality test for remote FS Exclude test_scheduler_locality when the filesystem can only be remote. Updates SkipOfNotHdfsMinicluster to also run on Ozone, enabling several dozen more tests for Ozone. Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 --- M tests/common/skip.py M tests/custom_cluster/test_scheduler_locality.py 2 files changed, 6 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/18980/2 -- To view, visit http://gerrit.cloudera.org:8080/18980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 Gerrit-Change-Number: 18980 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18970 ) Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py@437 PS2, Line 437: assert cnt_fail <= 5 > Ok, with data loaded it still fails but makes a bit more sense Restarted my impala cluster and ran it again, and this time it passes. -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Sep 2022 20:37:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11578: Exclude locality test for remote FS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18980 ) Change subject: IMPALA-11578: Exclude locality test for remote FS .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11348/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 Gerrit-Change-Number: 18980 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 13 Sep 2022 18:35:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11418: A statement that returns at most one row need not to spool results
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18711 ) Change subject: IMPALA-11418: A statement that returns at most one row need not to spool results .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8566/ -- To view, visit http://gerrit.cloudera.org:8080/18711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd4d73c21106048df68a270cf03d4abd56bd3aac Gerrit-Change-Number: 18711 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 13 Sep 2022 18:34:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18970 )
Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py@437
PS2, Line 437: assert cnt_fail <= 5
> With Ozone I get
Ok, with data loaded it still fails but makes a bit more sense
TestHdfsScannerSkew.test_mt_dop_skew_lpt[protocol: beeswax | exec_option:
{'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none]
tests/query_test/test_scanners.py:436: in test_mt_dop_skew_lpt
cnt_fail += count_intra_node_skew(profile)
tests/query_test/test_scanners.py:391: in count_intra_node_skew
assert len(lines) == 7 # Averaged fragment + 6 fragment
E assert 3 == 7
E+ where 3 = len(['- BytesRead: 662.80 MB (694994180)', '- BytesRead:
680.03 MB (713059192)', '- BytesRead: 645.57 MB (676929169)'])
--
To view, visit http://gerrit.cloudera.org:8080/18970
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99
Gerrit-Change-Number: 18970
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 13 Sep 2022 18:17:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11578: Exclude locality test for remote FS
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18980 Change subject: IMPALA-11578: Exclude locality test for remote FS .. IMPALA-11578: Exclude locality test for remote FS Exclude test_scheduler_locality when the filesystem can only be remote. Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 --- M tests/common/skip.py M tests/custom_cluster/test_scheduler_locality.py 2 files changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/18980/1 -- To view, visit http://gerrit.cloudera.org:8080/18980 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie6198421f21bc2520773ecbb34ffaf65969ebc43 Gerrit-Change-Number: 18980 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 21: (1 comment) http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG@45 PS20, Line 45: the maximum bytes of the read buffer memory are no more than 10% > nit. May mention that the memory borrowed from the reserved memory will be Done -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 21 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 13 Sep 2022 17:58:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Yida Wu has posted comments on this change. (
http://gerrit.cloudera.org:8080/17979 )
Change subject: IMPALA-10791 Add batch reading for remote temporary files
..
Patch Set 21:
(8 comments)
Thanks Qifan for the review.
http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG@32
PS20, Line 32: and disable this block, because it may be good to avoid
: duplicated reads from the remote fs for the same content.
:
> nit. suggest to present the positive case first. That is, moving it before
Done
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/runtime/tmp-file-mgr-test.cc
File be/src/runtime/tmp-file-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/runtime/tmp-file-mgr-test.cc@2183
PS20, Line 2183: WaitForDiskFileStatus(file1->DiskFile(),
DiskFileStatus::PERSISTED);
> nit. 'we use' -> 'we did use'.
Done
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/runtime/tmp-file-mgr-test.cc@2200
PS20, Line 2200: SSERT_OK(file_group.Read(handl
> To improve the readability, it looks like various IF tests inside this bloc
Added a helper function SetReadBufferSizeHelper().
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc@1070
PS13, Line 1070:
> nit exists
Done
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc@1078
PS13, Line 1078: [read_buffer_block, tmp_file = this](const Status&
fetch_status) {
> nit. Can this line be moved above 1074? That is, if remote batch reading is
The function firstly check whether the local buffer exists (even batch reading
is disabled, we could still have the disk buffer), if not, and batch reading
enabled, we will try to see if there is an available buffer in memory. So we
are not able to move the line ahead.
Also, this function is changed a lot in part 2, it may be better to keep
optimizing in part 2 if there is no safety issue here.
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc@1104
PS13, Line 1104: ile* TmpFileRemote::GetR
> Looks like this function does more than what the function name implies. You
Added a comment in the tmp-file-mgr.cc. Already a comment in the header of
IncrementReadPageCount() in the tmp-file-mgr-internal.h. I think it is more
efficient, because otherwise we need to fetch the lock to double check whether
all the pages are read every time after the increment. Also it ensures that
only one thread knows the "all_read" status and does the buffer removing job.
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/mem-info.cc
File be/src/util/mem-info.cc:
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/mem-info.cc@265
PS20, Line 265: (process_avail_mem) *process_a
> may just say
Done
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/metrics.h
File be/src/util/metrics.h:
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/metrics.h@272
PS20, Line 272: /// The hwm value is also updated atomically.
> May add "Also return the updated current value." here.
Done
--
To view, visit http://gerrit.cloudera.org:8080/17979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b
Gerrit-Change-Number: 17979
Gerrit-PatchSet: 21
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Tue, 13 Sep 2022 17:57:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11347/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 21 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 13 Sep 2022 17:55:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18970 ) Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. Patch Set 2: > Patch Set 2: > > (1 comment) Oh, turns out I didn't have data loaded. Let me reload and try again. -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Sep 2022 17:43:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Yida Wu has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. IMPALA-10791 Add batch reading for remote temporary files The patch adds a feature to batch read from a remote temporary file in order to improve the reading performance for the spilled remote data. Originally, the design is to use the local disk file as the buffer for batch read from the remote file. But in practice, it doesn't help to improve the performance. Therefore, the design is changed to use the memory as the read buffer. Currently, each TmpFileRemote has two DiskFile, one is for the remote, and one is for the local buffer. The patch adds MemBlocks to the local buffer file. Each local buffer file is divided into several MemBlocks evenly. Moreover, in order to guarantee a single page not being cut into two parts in different blocks, the block size could be a little different to each other in practice. The default block size is the minimum value between the default file size and MAX_REMOTE_READ_MEM_BLOCK_THRESHOLD_BYTES, which is 16MB. When pinning a page, the system will detect if there is enough memory for the block that holds the page. If yes, the block will be stored in the memory until all of the pages in the block are read or the query ends. If not, we will go reading the page directly and disable this block, because it may be good to avoid duplicated reads from the remote fs for the same content. One challenge of the read buffer is where to get the extra memory for it, because when impala starts to spill data, it means the process lacks of memory to use. By default, impala process will reserve 20% of the total system memory as unused memory, and here we will use this unused memory for the read buffer because it is reasonable to use it for the emergency case like spilling and the memory of the read buffer will be returned immediately after the use. For system reliability consideration, we set a restriction that, the maximum bytes of the read buffer memory are no more than 10% of the total system memory and 50% of the unused memory. Also, if the unused memory is less than 5% of the total system memory, the read buffer will be disabled. Two start options have been added for the new feature. 1. remote_batch_read. Default is false. If set true, the batch read is enabled. 2. remote_read_memory_buffer_size. Default is 1G. The maximum memory that can be used by the read buffer. The number is also restricted by the process memory limit, which can not exceed 10% of the process memory limit. Added metrics ScratchReadsUseMem/ScratchBytesReadUseMem/ ScratchBytesReadUseLocalDisk to the query profile. The patch also increases the MAX_REMOTE_TMPFILE_SIZE_THRESHOLD_MB from 256 to 512. Tests: Ran core and exhaustive tests. Added and ran TmpFileMgrTest::TestBatchReadingFromRemote. Added e2e test test_scratch_dirs_batch_reading. Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b --- M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/disk-file-test.cc M be/src/runtime/io/disk-file.cc M be/src/runtime/io/disk-file.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/mem-info.cc M be/src/util/mem-info.h M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_scratch_disk.py 19 files changed, 1,666 insertions(+), 156 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/17979/21 -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 21 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11511: Add build options for reducing binary sizes
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18962 )
Change subject: IMPALA-11511: Add build options for reducing binary sizes
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18962/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18962/1//COMMIT_MSG@9
PS1, Line 9: Impala's build produces has dozens of C++ binaries
"produces has" doesn't read right.
http://gerrit.cloudera.org:8080/#/c/18962/1/be/CMakeLists.txt
File be/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/18962/1/be/CMakeLists.txt@220
PS1, Line 220: SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g1")
Will this end up adding it in Release mode? Is the next line sufficient?
--
To view, visit http://gerrit.cloudera.org:8080/18962
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04a20258a86053d8f3972b9c7c81cd5bec1bbb66
Gerrit-Change-Number: 18962
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 13 Sep 2022 17:03:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18970 )
Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py@437
PS2, Line 437: assert cnt_fail <= 5
With Ozone I get
TestHdfsScannerSkew.test_mt_dop_skew_lpt[protocol: beeswax | exec_option:
{'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none]
tests/query_test/test_scanners.py:436: in test_mt_dop_skew_lpt
cnt_fail += count_intra_node_skew(profile)
tests/query_test/test_scanners.py:391: in count_intra_node_skew
assert len(lines) == 7 # Averaged fragment + 6 fragment
E assert 2 == 7
E+ where 2 = len(['- BytesRead: 0', '- BytesRead: 0'])
I can dig into that a bit, I have no theory yet why they'd be different.
--
To view, visit http://gerrit.cloudera.org:8080/18970
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99
Gerrit-Change-Number: 18970
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 13 Sep 2022 16:58:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9999: Switch to GCC 10.4
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18134 ) Change subject: IMPALA-: Switch to GCC 10.4 .. Patch Set 12: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe6857b822925226d39fd4d6413457ef6bbaabec Gerrit-Change-Number: 18134 Gerrit-PatchSet: 12 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 13 Sep 2022 16:36:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18970 ) Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11346/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Sep 2022 15:34:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18970 ) Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/18970/2/tests/query_test/test_scanners.py@378 PS2, Line 378: a flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Sep 2022 15:13:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18970 ) Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. Patch Set 1: (3 comments) Thanks for the comments. http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py@401 PS1, Line 401: se > typo Done http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py@405 PS1, Line 405: + > flake8: E226 missing whitespace around arithmetic operator Done http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py@410 PS1, Line 410: print "Intra-node bytes read ratio:", ratio > Is this intentionally left here? Yeah, I also mentioned it in the commit message. So if the test fails we'll know what were the ratios. -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 13 Sep 2022 15:12:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18970 to look at the new patch set (#2). Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. IMPALA-11572: deflake test_mt_dop_skew_lpt test_mt_dop_skew_lpt was flaky. Also, it calculated the min(bytes_read) / max(bytes_read) globally across all fragment insteances, not just among the intra-node fragment instances. To deflake the test, this test: * calculate intra-node min(bytes_read) / max(bytes_read) ratios instead of global ones * print out the ratios so we'll know the numbers when the test fails * eliminate compression codec test dimension which is not used anyway Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 --- M tests/query_test/test_scanners.py 1 file changed, 22 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18970/2 -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11572: deflake test mt dop skew lpt
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18970 ) Change subject: IMPALA-11572: deflake test_mt_dop_skew_lpt .. Patch Set 1: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py@401 PS1, Line 401: se typo http://gerrit.cloudera.org:8080/#/c/18970/1/tests/query_test/test_scanners.py@410 PS1, Line 410: print "Intra-node bytes read ratio:", ratio Is this intentionally left here? -- To view, visit http://gerrit.cloudera.org:8080/18970 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I823542c21fe8f10f43a501fe4175da883eaf2f99 Gerrit-Change-Number: 18970 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 13 Sep 2022 15:06:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18927 ) Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function. .. IMPALA-11528: Catalogd should start up with a corrupt Hive function. This commit handles the case for a specific kind of corrupt function within the Hive Metastore in the following situation: A valid Hive SQL function gets created in HMS. This UDF is written in Java and must derive from the "UDF" class. After creating this function in Impala, we then replace the underlying jar file with a class that does NOT derive from the "UDF" class. In this scenario, catalogd should reject the function and still start up gracefully. Before this commit, catalogd wasn't coming up. The reason for this was because the Hive function FunctionUtils.getUDFClassType() has a dependency on UDAF and was throwing a LinkageError exception, so we need to include the UDAF class in the shaded jar. Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112 Reviewed-on: http://gerrit.cloudera.org:8080/18927 Tested-by: Impala Public Jenkins Reviewed-by: Tamas Mate --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M java/pom.xml M java/shaded-deps/hive-exec/pom.xml A java/test-corrupt-hive-udfs/pom.xml A java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptUdf.java M testdata/bin/copy-udfs-udas.sh M tests/custom_cluster/test_permanent_udfs.py 8 files changed, 153 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Verified Tamas Mate: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112 Gerrit-Change-Number: 18927 Gerrit-PatchSet: 11 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-11528: Catalogd should start up with a corrupt Hive function.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18927 ) Change subject: IMPALA-11528: Catalogd should start up with a corrupt Hive function. .. Patch Set 10: Submitting as the exhaustive tests were run successfully. -- To view, visit http://gerrit.cloudera.org:8080/18927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I54e7a1df6d018ba6cf5ecf32dc9946edf86e2112 Gerrit-Change-Number: 18927 Gerrit-PatchSet: 10 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Steve Carlin Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 13 Sep 2022 14:48:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/17979 )
Change subject: IMPALA-10791 Add batch reading for remote temporary files
..
Patch Set 20:
(9 comments)
Looks great!
http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG@32
PS20, Line 32: If the system decides to fetch a block, the block will be
: stored in the memory until all of the pages in the block are read
: or the query ends.
nit. suggest to present the positive case first. That is, moving it before "if
not, we will go" at line 29.
http://gerrit.cloudera.org:8080/#/c/17979/20//COMMIT_MSG@45
PS20, Line 45: the read buffer will be disabled.
nit. May mention that the memory borrowed from the reserved memory will be
returned immediately after use.
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/runtime/tmp-file-mgr-test.cc
File be/src/runtime/tmp-file-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/runtime/tmp-file-mgr-test.cc@2183
PS20, Line 2183: // Check the metrics that we use the read buffer for reading.
nit. 'we use' -> 'we did use'.
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/runtime/tmp-file-mgr-test.cc@2200
PS20, Line 2200: f (buffer_size == small_size)
To improve the readability, it looks like various IF tests inside this block
can be avoided if a helper function testCapacityForReadBuffer(int64_t
buffer_size) can be written that tests one buffer size only.
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc@1070
PS13, Line 1070:
nit exists
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc@1078
PS13, Line 1078: [read_buffer_block, tmp_file = this](const Status&
fetch_status) {
nit. Can this line be moved above 1074? That is, if remote batch reading is
disabled, return null immediately. No need to do the work from 1074 to 1077.
http://gerrit.cloudera.org:8080/#/c/17979/13/be/src/runtime/tmp-file-mgr.cc@1104
PS13, Line 1104: ile* TmpFileRemote::GetR
Looks like this function does more than what the function name implies. You may
need to add a comment at least. Is there a strong need to do this check?
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/mem-info.cc
File be/src/util/mem-info.cc:
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/mem-info.cc@265
PS20, Line 265: (process_avail_mem != nullptr)
may just say
if (process_avail_mem) *process_avail_mem = avail_mem;
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/metrics.h
File be/src/util/metrics.h:
http://gerrit.cloudera.org:8080/#/c/17979/20/be/src/util/metrics.h@272
PS20, Line 272: /// The hwm value is also updated atomically.
May add "Also return the updated current value." here.
--
To view, visit http://gerrit.cloudera.org:8080/17979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b
Gerrit-Change-Number: 17979
Gerrit-PatchSet: 20
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Tue, 13 Sep 2022 14:21:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11563: Optimized /etc/sysconfig/clock to find the time zone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18958 ) Change subject: IMPALA-11563: Optimized /etc/sysconfig/clock to find the time zone .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18958 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f80fd1817d072f8dadf288025cb9534191ca458 Gerrit-Change-Number: 18958 Gerrit-PatchSet: 2 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Tue, 13 Sep 2022 13:50:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: For alter table, add column operation is optimized
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: For alter table, add column operation is optimized .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 3 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Tue, 13 Sep 2022 13:42:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7942 (part 1): Add query hints for table cardinalities
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/18829 ) Change subject: IMPALA-7942 (part 1): Add query hints for table cardinalities .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/18829/5/testdata/workloads/functional-planner/queries/PlannerTest/table-cardinality-hint.test File testdata/workloads/functional-planner/queries/PlannerTest/table-cardinality-hint.test: http://gerrit.cloudera.org:8080/#/c/18829/5/testdata/workloads/functional-planner/queries/PlannerTest/table-cardinality-hint.test@11 PS5, Line 11: 1 Maybe in the future we could allow short-hand notation such as 100K, 10G etc. Copied from https://gerrit.cloudera.org/#/c/18023/, by Qifan Chen. -- To view, visit http://gerrit.cloudera.org:8080/18829 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f0c773f4e67782a1428db64062f68afbd257af7 Gerrit-Change-Number: 18829 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 13 Sep 2022 13:02:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7942 (part 1): Add query hints for table cardinalities
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/18829 ) Change subject: IMPALA-7942 (part 1): Add query hints for table cardinalities .. Patch Set 5: (4 comments) > (5 comments) > > Looks good! > > Complete the code section and have a few comments. > > I have not got a chance to review the test section. If it is > similar to the one in the parent JIRA of this JIRA, would it be > possible to address any concerns there in this JIRA? Hi Qifan, thanks for continued interest in this feature. This patch is split from https://gerrit.cloudera.org/#/c/18023. We will focus on table cardinality hint in this patch, and reduce previous patch to focus on selectivity hint. I've already minimize test cases in this patch. Remove unnecessary plan to keeping test file small. Such as PARALLEL and DISTRIBUTED plan, not exists in test file. Only 61 lines currently. If you are free, you can read it quickly and simply. http://gerrit.cloudera.org:8080/#/c/18829/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18829/5//COMMIT_MSG@9 PS5, Line 9: Currently, We need execute 'COMPUTE STATS' manually to compute : table stats info. Stats is very useful for query planning. : Without these stats, query plan maybe worse. In order to solve : this probelm, this patch adds a new query hint: 'TABLE_NUM_ROWS', > nit. Rewording. Done By the way, 'valid' -> 'invalid'? http://gerrit.cloudera.org:8080/#/c/18829/5/fe/src/main/java/org/apache/impala/analysis/TableRef.java File fe/src/main/java/org/apache/impala/analysis/TableRef.java: http://gerrit.cloudera.org:8080/#/c/18829/5/fe/src/main/java/org/apache/impala/analysis/TableRef.java@176 PS5, Line 176: tableNumRowsHint_ > This new data member in class TableRef implies that two table ref objects m Do you mean different 'table_num_row' value for same table in sql will cause different cardinality is a problem? We should raise error for this situation? Such as: select count(1) from functional_parquet.alltypes t1/* +TABLE_NUM_ROWS(1000) */,functional_parquet.alltypes t2/* +TABLE_NUM_ROWS(2000) */ where t1.id = t2.id; http://gerrit.cloudera.org:8080/#/c/18829/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/18829/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1547 PS5, Line 1547: * Currently, we provide a table hint 'TABLE_NUM_ROWS' to set table rows manually if :* table has no stats or has corrupt stats. If the table has valid stats, this hint :* will be ignored. > In my past experience with very large data warehouse, that the table num ro Thanks for adivce, Qifan. I agree that we can extend for valid stats later on if necessary. http://gerrit.cloudera.org:8080/#/c/18829/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java: http://gerrit.cloudera.org:8080/#/c/18829/5/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@379 PS5, Line 379: no > nit. has no stats Done -- To view, visit http://gerrit.cloudera.org:8080/18829 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f0c773f4e67782a1428db64062f68afbd257af7 Gerrit-Change-Number: 18829 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 13 Sep 2022 13:00:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11418: A statement that returns at most one row need not to spool results
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18711 ) Change subject: IMPALA-11418: A statement that returns at most one row need not to spool results .. Patch Set 9: Code-Review+2 LGTM. Carry Riza's +1. -- To view, visit http://gerrit.cloudera.org:8080/18711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd4d73c21106048df68a270cf03d4abd56bd3aac Gerrit-Change-Number: 18711 Gerrit-PatchSet: 9 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 13 Sep 2022 12:45:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11418: A statement that returns at most one row need not to spool results
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18711 ) Change subject: IMPALA-11418: A statement that returns at most one row need not to spool results .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd4d73c21106048df68a270cf03d4abd56bd3aac Gerrit-Change-Number: 18711 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 13 Sep 2022 12:46:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11418: A statement that returns at most one row need not to spool results
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18711 ) Change subject: IMPALA-11418: A statement that returns at most one row need not to spool results .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8566/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd4d73c21106048df68a270cf03d4abd56bd3aac Gerrit-Change-Number: 18711 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 13 Sep 2022 12:46:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11557: Fix memory leak in BlockingRowBatchQueue
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18960 ) Change subject: IMPALA-11557: Fix memory leak in BlockingRowBatchQueue .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I656316b6575ce74a03b83fcd45e772c763835d56 Gerrit-Change-Number: 18960 Gerrit-PatchSet: 2 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 13 Sep 2022 12:29:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18977 ) Change subject: IMPALA-10408: Support build using Apache components .. Patch Set 1: Thanks for your patch! I see you uploaded several patches in the same topic. In case you don't know that, you can update the patch by "git commit --amend" and then "git push asf-gerrit HEAD:refs/for/master". Gerrit will use the Change-Id to match the patch. -- To view, visit http://gerrit.cloudera.org:8080/18977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399 Gerrit-Change-Number: 18977 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 13 Sep 2022 12:28:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: For alter table, add column operation is optimized
Jian Zhang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18953 )
Change subject: IMPALA-11565: For alter table, add column operation is optimized
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18953/3/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java:
http://gerrit.cloudera.org:8080/#/c/18953/3/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java@87
PS3, Line 87: if (ifNotExists_) {
: if (col != null) {
: if (!col.getType().equals(c.getType())) {
: throw new AnalysisException(String.format("Error
adding column %s " +
: "from table %s: type not match", colName,
t.getName()));
: } else {
: iterator.remove();
: }
: }
: } else if (col != null) {
: throw new AnalysisException("Column already exists: " +
colName);
: }
how about simplifying the precondition check to the following to improve the
code readability:
```cpp
if (col != nil) {
if (!ifNotExists) {
throw new AnalysisException("Column already exists: " + colName);
}
// handle the case that ifNotExists is true
if (!col.getType().equals(c.getType())) {
throw new AnalysisException(String.format("Error adding column %s " +
"from table %s: type not match", colName, t.getName()));
}
// remove the column to be added which is existed.
iterator.remove();
}
```
http://gerrit.cloudera.org:8080/#/c/18953/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18953/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1029
PS3, Line 1029: if (params.getAdd_cols_params() != null
: && params.getAdd_cols_params().getColumnsSize()
!= 0) {
could you add a coment about the reason for adding this validation, is it a
bug-fix?
--
To view, visit http://gerrit.cloudera.org:8080/18953
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517
Gerrit-Change-Number: 18953
Gerrit-PatchSet: 3
Gerrit-Owner: Baike Xia
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jian Zhang
Gerrit-Comment-Date: Tue, 13 Sep 2022 11:28:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11565: For alter table, add column operation is optimized
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: For alter table, add column operation is optimized .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11345/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 3 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Tue, 13 Sep 2022 09:21:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11563: Optimized /etc/sysconfig/clock to find the time zone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18958 ) Change subject: IMPALA-11563: Optimized /etc/sysconfig/clock to find the time zone .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8565/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18958 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f80fd1817d072f8dadf288025cb9534191ca458 Gerrit-Change-Number: 18958 Gerrit-PatchSet: 2 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Tue, 13 Sep 2022 09:02:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: For alter table, add column operation is optimized
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: For alter table, add column operation is optimized .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8564/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 3 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Tue, 13 Sep 2022 09:00:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: For alter table, add column operation is optimized
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/18953 )
Change subject: IMPALA-11565: For alter table, add column operation is optimized
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18953/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/18953/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@473
PS3, Line 473: AnalysisError("alter table functional.alltypes add if not
exists columns (int_col tinyint)",
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/18953
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517
Gerrit-Change-Number: 18953
Gerrit-PatchSet: 3
Gerrit-Owner: Baike Xia
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jian Zhang
Gerrit-Comment-Date: Tue, 13 Sep 2022 09:00:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11565: For alter table, add column operation is optimized
Baike Xia has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: For alter table, add column operation is optimized .. IMPALA-11565: For alter table, add column operation is optimized For alter table, add if not exists column operation, if the columns already exist and are of the same type, no operation is performed; If the type is different, an error is reported. Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 --- M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java 4 files changed, 34 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18953/3 -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 3 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang
[Impala-ASF-CR] IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18978 ) Change subject: IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11344/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18978 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie04ff243c6b82c1a06c489da74353f2d8afe423a Gerrit-Change-Number: 18978 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 13 Sep 2022 08:51:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18978 Change subject: IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates .. IMPALA-11580: Fix memory leak in legacy catalog mode when applying incremental partition updates In the legacy catalog mode, catalogd propagates incremental metadata updates at the partition level. While applying the updates, impalad reuses the existing partition objects and moves them to a new HdfsTable object. However, the partition objects are immutable, which means their reference to the old table object remains unchanged. JVM GC cannot collect the stale table objects since they still have active reference from the partitions, which results in memory leak. This patch fixes the issue by recreating a new partition object based on the existing partition object with the new table field. Tests: - Verified locally that after applying the patch, I don’t see the number of live HdfsTable objects keeps bumping. Change-Id: Ie04ff243c6b82c1a06c489da74353f2d8afe423a --- M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java 2 files changed, 11 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/18978/1 -- To view, visit http://gerrit.cloudera.org:8080/18978 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie04ff243c6b82c1a06c489da74353f2d8afe423a Gerrit-Change-Number: 18978 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-11463: Support IF NOT EXISTS in alter table add columns for kudu table
Xiang Yang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18812 )
Change subject: IMPALA-11463: Support IF NOT EXISTS in alter table add columns
for kudu table
..
Patch Set 4:
(2 comments)
> Patch Set 2:
>
> (2 comments)
http://gerrit.cloudera.org:8080/#/c/18812/2/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18812/2/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@519
PS2, Line 519: if (!oldColumns.contains(column.getColumnName())) {
> should we raise an error when oldColumns.contains(column.getColumnName()) a
Hi jian, impala already have check here:
https://github.com/apache/impala/blob/4.1.0/fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java#L84.
http://gerrit.cloudera.org:8080/#/c/18812/2/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
File testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test:
http://gerrit.cloudera.org:8080/#/c/18812/2/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test@352
PS2, Line 352: CATCH
> could you add another test without the `if not exists` clause:
Done
--
To view, visit http://gerrit.cloudera.org:8080/18812
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If624b3bd12eeb1a848e9fc4b6099c47bcf80c9f6
Gerrit-Change-Number: 18812
Gerrit-PatchSet: 4
Gerrit-Owner: Xiang Yang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jian Zhang
Gerrit-Reviewer: Xiang Yang
Gerrit-Comment-Date: Tue, 13 Sep 2022 06:40:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11463: Support IF NOT EXISTS in alter table add columns for kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18812 ) Change subject: IMPALA-11463: Support IF NOT EXISTS in alter table add columns for kudu table .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18812 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If624b3bd12eeb1a848e9fc4b6099c47bcf80c9f6 Gerrit-Change-Number: 18812 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Tue, 13 Sep 2022 06:32:12 + Gerrit-HasComments: No
