[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5786/ -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 05:49:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9669: Fix wrong table types for GET TABLES in LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15887 ) Change subject: IMPALA-9669: Fix wrong table types for GET_TABLES in LocalCatalog .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6007/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15887 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2180c603f061838347936f718cd4a0257d82e633 Gerrit-Change-Number: 15887 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 May 2020 04:19:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9570: [DOCS] add memory management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15836 ) Change subject: IMPALA-9570: [DOCS] add memory management .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Gerrit-Change-Number: 15836 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 08 May 2020 04:18:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9570: [DOCS] add memory management
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15836 ) Change subject: IMPALA-9570: [DOCS] add memory management .. IMPALA-9570: [DOCS] add memory management add memory management and fix broken links. Incorporated review changes. Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Reviewed-on: http://gerrit.cloudera.org:8080/15836 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docs/impala_keydefs.ditamap M docs/topics/impala_udf.xml 2 files changed, 10 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Gerrit-Change-Number: 15836 Gerrit-PatchSet: 4 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9541: [DOCS] add steps to dynamically change log levels
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15867 ) Change subject: IMPALA-9541: [DOCS] add steps to dynamically change log levels .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/15867/1/docs/topics/impala_logging.xml File docs/topics/impala_logging.xml: http://gerrit.cloudera.org:8080/#/c/15867/1/docs/topics/impala_logging.xml@44 PS1, Line 44: The Impala logs record information about: Just curious: there are many formatting changes like these. Are they nessasary, i.e. the old codes violating any code styles? http://gerrit.cloudera.org:8080/#/c/15867/1/docs/topics/impala_logging.xml@195 PS1, Line 195: By default this : page displays the current running log level of the service. The current log levels are not shown in this page. But users can query the log4j log level of "root" or "org.apache.impala" by using the "Get Java Log Level" button. http://gerrit.cloudera.org:8080/#/c/15867/1/docs/topics/impala_logging.xml@204 PS1, Line 204: Log Levels. Can we mention that this is a one character info and their meaning are "I=INFO, W=WARNING, E=ERROR, F=FATAL" ? http://gerrit.cloudera.org:8080/#/c/15867/1/docs/topics/impala_logging.xml@208 PS1, Line 208: PID/TID of the process/thread. I think there are no PID for process. http://gerrit.cloudera.org:8080/#/c/15867/1/docs/topics/impala_logging.xml@211 PS1, Line 211: user-specified remove this word? Logs are generated by the system(impala). -- To view, visit http://gerrit.cloudera.org:8080/15867 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic72fbfe3218530f5a1ffa0159ce714e411401bcd Gerrit-Change-Number: 15867 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 04:16:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9570: [DOCS] add memory management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15836 ) Change subject: IMPALA-9570: [DOCS] add memory management .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/366/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Gerrit-Change-Number: 15836 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 08 May 2020 03:57:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9570: [DOCS] add memory management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15836 ) Change subject: IMPALA-9570: [DOCS] add memory management .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Gerrit-Change-Number: 15836 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 08 May 2020 03:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9570: [DOCS] add memory management
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15836 ) Change subject: IMPALA-9570: [DOCS] add memory management .. Patch Set 2: Code-Review+2 Thanks for updating the doc. -- To view, visit http://gerrit.cloudera.org:8080/15836 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e8b6d0c3fe2e1746831665b3d3ae98a0beaa1e7 Gerrit-Change-Number: 15836 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 08 May 2020 03:54:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9597: Eliminate redundant Ranger audits for column masking
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15854 ) Change subject: IMPALA-9597: Eliminate redundant Ranger audits for column masking .. Patch Set 3: Code-Review+1 (2 comments) The solution looks good to me. Commented a minor optimization that may save some codes. http://gerrit.cloudera.org:8080/#/c/15854/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/15854/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@184 PS3, Line 184: ((RangerAuthorizationContext) authzCtx).applyDeduplicatedStashedAuditEvents(); We can merge these two calls into one if we use a Map to track the stashed events. http://gerrit.cloudera.org:8080/#/c/15854/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java: http://gerrit.cloudera.org:8080/#/c/15854/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java@40 PS3, Line 40: private List stashedAuditEvents_; Can we use a Map for this? Then we don't need to call deduplicateStashedAuthzEvents and don't need two lists here. -- To view, visit http://gerrit.cloudera.org:8080/15854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42d60130fba93d63fbc36949f2bf746b7ae2497d Gerrit-Change-Number: 15854 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 08 May 2020 03:52:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9669: Fix wrong table types for GET TABLES in LocalCatalog
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15887 Change subject: IMPALA-9669: Fix wrong table types for GET_TABLES in LocalCatalog .. IMPALA-9669: Fix wrong table types for GET_TABLES in LocalCatalog Coordinator can be in two modes: legacy mode or LocalCatalog mode. Before IMPALA-8606, GET_TABLES required all tables to be loaded in LocalCatalog-mode coordinator’s cache, which is a performance regression compared to legacy mode coordinators. IMPALA-8606 changes the behavior to only load the table names and create LocalIncompleteTable for each table, which boosts the performance but results in all views being returned with the default table type (TABLE). Besides this, all returned comments are empty even if the table/view is loaded. This is a regression since in legacy coordinators, loaded tables/views are shown with correct table types and comments. In fact, if the hmsTable object of a table/view is loaded in local cache, we can extract its exact table type and comment. This patch fixes the problem by adding an additional lookup in the local cache for the table meta (hmsTable) of each table when creating the LocalIncompleteTables. When getting the table type and comment from a LocalIncompleteTable, use the loaded hmsTable. Tests - Add tests in test_hs2.test_get_tables - Run CORE tests Change-Id: I2180c603f061838347936f718cd4a0257d82e633 --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M tests/hs2/test_hs2.py 7 files changed, 90 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/15887/1 -- To view, visit http://gerrit.cloudera.org:8080/15887 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2180c603f061838347936f718cd4a0257d82e633 Gerrit-Change-Number: 15887 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-9727: Fix HBaseScanNode explain formatting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15749 ) Change subject: IMPALA-9727: Fix HBaseScanNode explain formatting .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5788/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15749 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30fad791408a1f7e35e9b3f2e6cb4958952dd567 Gerrit-Change-Number: 15749 Gerrit-PatchSet: 4 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 08 May 2020 03:19:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9727: Fix HBaseScanNode explain formatting
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15749 ) Change subject: IMPALA-9727: Fix HBaseScanNode explain formatting .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15749 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30fad791408a1f7e35e9b3f2e6cb4958952dd567 Gerrit-Change-Number: 15749 Gerrit-PatchSet: 4 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 08 May 2020 03:18:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9727: Fix HBaseScanNode explain formatting
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15749 ) Change subject: IMPALA-9727: Fix HBaseScanNode explain formatting .. Patch Set 3: Test failed due to IMPALA-9729. Rerun the GVO. -- To view, visit http://gerrit.cloudera.org:8080/15749 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30fad791408a1f7e35e9b3f2e6cb4958952dd567 Gerrit-Change-Number: 15749 Gerrit-PatchSet: 3 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 08 May 2020 03:17:58 + Gerrit-HasComments: No
[Impala-ASF-CR] Only build ext-py directories tracked by git for make shell tarball.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15886 ) Change subject: Only build ext-py directories tracked by git for make_shell_tarball.sh .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6006/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb70ef2d5048d5cfeb260ce62c34f04835c7132d Gerrit-Change-Number: 15886 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 May 2020 03:12:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7833 Audit and fix string builtins for long string handling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15864 ) Change subject: IMPALA-7833 Audit and fix string builtins for long string handling .. Patch Set 2: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5785/ -- To view, visit http://gerrit.cloudera.org:8080/15864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93a53845f04e61ff446b363c78db1e49cbd5dc49 Gerrit-Change-Number: 15864 Gerrit-PatchSet: 2 Gerrit-Owner: Akos Kovacs Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 03:05:08 + Gerrit-HasComments: No
[Impala-ASF-CR] Only build ext-py directories tracked by git for make shell tarball.sh
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15886 Change subject: Only build ext-py directories tracked by git for make_shell_tarball.sh .. Only build ext-py directories tracked by git for make_shell_tarball.sh When versions change in shell/ext-py/*, there can be leftover directories when developers rebase. These leftover directories are usually empty and unbuildable, so make_shell_tarball.sh will fail with an error message like: Creating an egg for .../Impala/shell/ext-py/bitarray-0.9.0 Traceback (most recent call last): File "", line 1, in IOError: [Errno 2] No such file or directory: 'setup.py' ... This changes the build logic to only build directories that are tracked in git. When a version of an ext-py package changes, the directory for the old version may stick around, but it is gone from the git repository and won't be built. The downside is that when a developer is adding a new package version, it won't be built until it is added in git. Testing: - Added an empty directory in shell/ext-py that was not tracked in git. Verified it is not built (and would fail before). Change-Id: Ibb70ef2d5048d5cfeb260ce62c34f04835c7132d --- M shell/make_shell_tarball.sh 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/15886/1 -- To view, visit http://gerrit.cloudera.org:8080/15886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibb70ef2d5048d5cfeb260ce62c34f04835c7132d Gerrit-Change-Number: 15886 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6002/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 02:54:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6001/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 02:48:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15755 ) Change subject: IMPALA-9676 Add aarch64 compile options for clang .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6005/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 Gerrit-Change-Number: 15755 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 May 2020 02:45:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9630 Keep blocking queue cache line aligned on aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15705 ) Change subject: IMPALA-9630 Keep blocking queue cache line aligned on aarch64 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6004/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 Gerrit-Change-Number: 15705 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 02:39:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 29: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6003/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 29 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 May 2020 02:30:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
zhaoren...@hotmail.com has uploaded a new patch set (#29). ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's crc32 instructions with ARM's instructions Replace Intel's popcntq instruction with ARM's mechanism Replace Intel's pcmpestri and pcmpestrm instructions with ARM mechanism Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/benchmarks/bswap-benchmark.cc M be/src/benchmarks/int-hash-benchmark.cc M be/src/codegen/CMakeLists.txt M be/src/codegen/gen_ir_descriptions.py M be/src/exec/delimited-text-parser.inline.h M be/src/exec/filter-context.cc M be/src/kudu/util/block_bloom_filter.cc M be/src/kudu/util/group_varint-inl.h M be/src/kudu/util/group_varint-test.cc A be/src/kudu/util/sse2neon.h M be/src/util/bit-util-test.cc M be/src/util/bit-util.cc M be/src/util/bit-util.h M be/src/util/bloom-filter-ir.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/cpu-info.cc M be/src/util/hash-util-ir.cc M be/src/util/hash-util.h M be/src/util/sse-util.h A be/src/util/sse2neon.h 23 files changed, 3,855 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/15531/29 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 29 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang
zhaoren...@hotmail.com has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/15755 ) Change subject: IMPALA-9676 Add aarch64 compile options for clang .. IMPALA-9676 Add aarch64 compile options for clang Add signed-char and armv8a and crc compile options to clang Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 --- M be/CMakeLists.txt 1 file changed, 7 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/15755/6 -- To view, visit http://gerrit.cloudera.org:8080/15755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 Gerrit-Change-Number: 15755 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9630 Keep blocking queue cache line aligned on aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/15705 ) Change subject: IMPALA-9630 Keep blocking queue cache line aligned on aarch64 .. IMPALA-9630 Keep blocking queue cache line aligned on aarch64 On aarch64, the DNCHECK_NE in BlockingQueue construct function will fail. So here use a different method to keep the 'put' class members and 'get' class members cache line aligned. Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 --- M be/src/common/compiler-util.h M be/src/exec/hdfs-scanner.h M be/src/util/aligned-new.h M be/src/util/blocking-queue.h 4 files changed, 13 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15705/8 -- To view, visit http://gerrit.cloudera.org:8080/15705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 Gerrit-Change-Number: 15705 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 8: Hi, Tomas, this has been fixed as your request -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 02:00:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/15718 )
Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64
..
IMPALA-9645 Port LLVM codegen to adapt aarch64
On aarch64, the Lowered type of struct {bool, int128} is form
{ {i8}, {i128} }. No padding add. This is different with x86-64,
which is { {i8}, {15*i8}, {i128} } with padding add automatically.
And here also add some type conversion between x86 and aarch64 data types.
And also add some aarch64 cpu's feature.
Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/codegen-anyval.h
M be/src/codegen/llvm-codegen.cc
M be/src/exec/text-converter.cc
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-fn-call.cc
6 files changed, 141 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15718/8
--
To view, visit http://gerrit.cloudera.org:8080/15718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Gerrit-Change-Number: 15718
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/15718 )
Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64
..
IMPALA-9645 Port LLVM codegen to adapt aarch64
On aarch64, the Lowered type of struct {bool, int128} is form
{ {i8}, {i128} }. No padding add. This is different with x86-64,
which is { {i8}, {15*i8}, {i128} } with padding add automatically.
And here also add some type conversion between x86 and aarch64 data types.
And also add some aarch64 cpu's feature.
Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/codegen-anyval.h
M be/src/codegen/llvm-codegen.cc
M be/src/exec/text-converter.cc
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-fn-call.cc
6 files changed, 132 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15718/7
--
To view, visit http://gerrit.cloudera.org:8080/15718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Gerrit-Change-Number: 15718
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9736: fix mt dop not supported error
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15884 ) Change subject: IMPALA-9736: fix mt_dop not supported error .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6000/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8eb8106f86c47a14cc951c4a77966fe51b5c30e3 Gerrit-Change-Number: 15884 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 May 2020 00:55:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5786/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 00:38:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 5: Code-Review+2 rebased -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 00:38:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9729: consistent GetExecSummary() behaviour
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15868 ) Change subject: IMPALA-9729: consistent GetExecSummary() behaviour .. Patch Set 2: Verified+1 I'm going to carry the verification because no code changes and this fixes some flakiness. -- To view, visit http://gerrit.cloudera.org:8080/15868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Gerrit-Change-Number: 15868 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 May 2020 00:31:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9729: consistent GetExecSummary() behaviour
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15868 ) Change subject: IMPALA-9729: consistent GetExecSummary() behaviour .. IMPALA-9729: consistent GetExecSummary() behaviour Before IMPALA-9380, GetExecSummary() for archived queries without a coordinator returned a blank TExecSummary, but for in-flight queries it returned a "not found" error. This was masked by the synchronous query unregistration, but now has made an impala-shell test flaky, because the close() RPC returns to the shell before the query is archived, so the following GetExecSummary() call may go down either code path. This change makes the behaviour consistent for both code paths and preserves impala-shell behaviour. Testing: Looped TestImpalaShell.test_summary to make sure flakiness was fixed. Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Reviewed-on: http://gerrit.cloudera.org:8080/15868 Reviewed-by: Thomas Tauber-Marshall Tested-by: Tim Armstrong --- M be/src/service/impala-server.cc M common/thrift/ImpalaService.thrift 2 files changed, 7 insertions(+), 1 deletion(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/15868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Gerrit-Change-Number: 15868 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9585: update mt dop docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15883 ) Change subject: IMPALA-9585: update mt_dop docs .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/593/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/15883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f Gerrit-Change-Number: 15883 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shajini Thayasingh Gerrit-Comment-Date: Fri, 08 May 2020 00:26:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 2): Refactor parts of TExecPlanFragmentInfo to protobuf
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15844 ) Change subject: IMPALA-9692 (part 2): Refactor parts of TExecPlanFragmentInfo to protobuf .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5999/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a8e46767b257bbf677171ac2f4efb1b623ba41b Gerrit-Change-Number: 15844 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 08 May 2020 00:16:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Bankim Bhavsar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15683 )
Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
..
Patch Set 11:
(2 comments)
Apologize for not taking a look earlier. Lot of the changes looked specific to
Impala.
I mainly looked at Impala BBF code that was updated to invoke one from
kudu-util.
http://gerrit.cloudera.org:8080/#/c/15683/11/be/src/util/bloom-filter.cc
File be/src/util/bloom-filter.cc:
http://gerrit.cloudera.org:8080/#/c/15683/11/be/src/util/bloom-filter.cc@71
PS11, Line 71: if (directory_in_size == 0)
Could you clarify with a comment in what case would directory_in_size be 0
since this Init variant should really be used for deserialization case.
http://gerrit.cloudera.org:8080/#/c/15683/11/be/src/util/bloom-filter.cc@201
PS11, Line 201: if (is_allocated_) {
: LOG(DFATAL) << "Each call to AllocateBuffer must have a
corresponding call "
: << "to FreeBuffer.";
: Close(); // Ensure that any previously allocated memory is
released.
: }
Same instance of allocator can be used for multiple BBFs. So this doesn't look
right to me.
With that in mind, I don't see need for is_allocated_member variable.
ImpalaBloomFilterBufferAllocator should be a simple wrapper on top of Impala's
default buffer pool and should be implemented as a singleton.
--
To view, visit http://gerrit.cloudera.org:8080/15683
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754
Gerrit-Change-Number: 15683
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Bankim Bhavsar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 08 May 2020 00:11:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9585: update mt dop docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15883 ) Change subject: IMPALA-9585: update mt_dop docs .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/593/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/15883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f Gerrit-Change-Number: 15883 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shajini Thayasingh Gerrit-Comment-Date: Fri, 08 May 2020 00:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9585: update mt dop docs
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15883 to look at the new patch set (#2). Change subject: IMPALA-9585: update mt_dop docs .. IMPALA-9585: update mt_dop docs Updated to reflect changes in IMPALA-9099 and IMPALA-9736. Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f --- M docs/topics/impala_mt_dop.xml 1 file changed, 27 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/15883/2 -- To view, visit http://gerrit.cloudera.org:8080/15883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f Gerrit-Change-Number: 15883 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9736: fix mt dop not supported error
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15884 Change subject: IMPALA-9736: fix mt_dop not supported error .. IMPALA-9736: fix mt_dop not supported error The error was not accurate, because joins are now supported. Also updated it to refer to DML statements instead of table sinks to be more user-appropriate. Change-Id: I8eb8106f86c47a14cc951c4a77966fe51b5c30e3 --- M fe/src/main/java/org/apache/impala/planner/Planner.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test 2 files changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/15884/1 -- To view, visit http://gerrit.cloudera.org:8080/15884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8eb8106f86c47a14cc951c4a77966fe51b5c30e3 Gerrit-Change-Number: 15884 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9585: update mt dop docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15883 ) Change subject: IMPALA-9585: update mt_dop docs .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/592/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/15883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f Gerrit-Change-Number: 15883 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 23:33:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9692 (part 2): Refactor parts of TExecPlanFragmentInfo to protobuf
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15844 to look at the new patch set (#3). Change subject: IMPALA-9692 (part 2): Refactor parts of TExecPlanFragmentInfo to protobuf .. IMPALA-9692 (part 2): Refactor parts of TExecPlanFragmentInfo to protobuf The new admission control service will be written in protobuf, so there are various admission control related structures currently stored in Thrift that it would be convenient to convert to protobuf, to minimize the amount of converting back and forth that needs to be done. This patch converts some portions of TExecPlanFragmentInfo to protobuf. TExecPlanFragmentInfo is sent as a sidecar with the Exec() rpc, so the refactored parts are now just directly included in the ExecQueryFInstancesRequestPB. The portions that are converted are those that are part of the QuerySchedule, in particular the TPlanFragmentDestination, TScanRangeParams, and TJoinBuildInput. This patch is just a refactor and doesn't contain any functional changes. One notable related change is that DataSink::CreateSink() has two parameters removed - TPlanFragmentCtx (which no longer exists) and TPlanFragmentInstanceCtx. These variables and the new PB eqivalents are available via the RuntimeState that was already being passed in as another parameter and don't need to be individually passed in. Testing: - Passed a full run of existing tests. Change-Id: I3a8e46767b257bbf677171ac2f4efb1b623ba41b --- M be/src/benchmarks/expr-benchmark.cc M be/src/benchmarks/hash-benchmark.cc M be/src/codegen/llvm-codegen-test.cc M be/src/exec/blocking-join-node.cc M be/src/exec/data-sink.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-table-sink.cc M be/src/exec/hbase-table-sink.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M be/src/exec/nested-loop-join-builder.cc M be/src/exec/nested-loop-join-builder.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exec/scan-node.h M be/src/exprs/expr-codegen-test.cc M be/src/rpc/CMakeLists.txt M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/query-state.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/test-env.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/fe-support.cc M be/src/util/container-util.h M be/src/util/uid-util.h M common/protobuf/CMakeLists.txt M common/protobuf/common.proto M common/protobuf/control_service.proto A common/protobuf/planner.proto M common/thrift/ImpalaInternalService.thrift 50 files changed, 623 insertions(+), 375 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/15844/3 -- To view, visit http://gerrit.cloudera.org:8080/15844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3a8e46767b257bbf677171ac2f4efb1b623ba41b Gerrit-Change-Number: 15844 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 4: gvo failed due to IMPALA-9729, will try again -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 23:18:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9729: consistent GetExecSummary() behaviour
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15868 ) Change subject: IMPALA-9729: consistent GetExecSummary() behaviour .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Gerrit-Change-Number: 15868 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 07 May 2020 23:16:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9585: update mt dop docs
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15883 Change subject: IMPALA-9585: update mt_dop docs .. IMPALA-9585: update mt_dop docs Updated to reflect changes in IMPALA-9099 Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f --- M docs/topics/impala_mt_dop.xml 1 file changed, 27 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/15883/1 -- To view, visit http://gerrit.cloudera.org:8080/15883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f Gerrit-Change-Number: 15883 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15881 ) Change subject: Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" .. Patch Set 1: Verified+1 The tests that were failing on Centos 7 pass with the revert. Going forward with this. -- To view, visit http://gerrit.cloudera.org:8080/15881 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Gerrit-Change-Number: 15881 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 23:15:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15881 ) Change subject: Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" .. Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" The fix for IMPALA-9718 introduced test failures on Centos 7. See IMPALA-9735. This reverts commit 75d98b4b081df95b58d7388da39bb1ec7c2f4f67. Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Reviewed-on: http://gerrit.cloudera.org:8080/15881 Reviewed-by: Tim Armstrong Tested-by: Joe McDonnell --- M LICENSE.txt M shell/make_shell_tarball.sh A shell/pkg_resources.py 3 files changed, 2,702 insertions(+), 0 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/15881 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Gerrit-Change-Number: 15881 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9585: update mt dop docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15883 ) Change subject: IMPALA-9585: update mt_dop docs .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/592/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/15883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc7511fede5f9b36ae8250d3acf8d0061b48106f Gerrit-Change-Number: 15883 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 23:15:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5783/ -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 23:01:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9597: Eliminate redundant Ranger audits for column masking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15854 ) Change subject: IMPALA-9597: Eliminate redundant Ranger audits for column masking .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5998/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42d60130fba93d63fbc36949f2bf746b7ae2497d Gerrit-Change-Number: 15854 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 07 May 2020 22:32:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9597: Eliminate redundant Ranger audits for column masking
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15854 ) Change subject: IMPALA-9597: Eliminate redundant Ranger audits for column masking .. Patch Set 3: Hi Csaba and Quanlong, I just found that I forgot to change my commit message accordingly. For the sentence "Specifically, instead of updating the List of AuthzAuditEvent's of the corresponding RangerBufferAuditHandler immediately after the deduplication, we clear this List first and add the deduplicated events back to the List only if there is no AuthorizationException.", I would like to change that to "Specifically, we stash the audit events after the analysis of the query and will deduplicate the stashed events and add back those deduplicated events only if the authorization of the query is successful." I will do this when addressing your comments in the next iteration. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/15854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42d60130fba93d63fbc36949f2bf746b7ae2497d Gerrit-Change-Number: 15854 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 07 May 2020 22:16:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 22:14:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support Impala 4 moved to using CDP versions for components, which involves adopting Hive 3. This removes the old code supporting CDH components and Hive 2. Specifically, it does the following: 1. Remove USE_CDP_HIVE and default to the values from USE_CDP_HIVE=true. USE_CDP_HIVE now has no effect on the Impala environment. This also means that bin/jenkins/build-all-flag-combinations.sh no longer include USE_CDP_HIVE=false as a configuration. 2. Remove USE_CDH_KUDU and default to getting Impala from the native toolchain. 3. Ban IMPALA_HIVE_MAJOR_VERSION<3 and remove related code, including the IMPALA_HIVE_MAJOR_VERSION=2 maven profile in fe/pom.xml. There is a fair amount of code that still references the Hive major version. Upstream Hive is now working on Hive 4, so there is a high likelihood that we'll need some code to deal with that transition. This leaves some code (such as maven profiles) and test logic in place. Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Reviewed-on: http://gerrit.cloudera.org:8080/15869 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M README-build.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M fe/pom.xml D fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java D fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M testdata/bin/create-load-data.sh M testdata/bin/run-hive-server.sh M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 11 files changed, 81 insertions(+), 1,108 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7833 Audit and fix string builtins for long string handling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15864 ) Change subject: IMPALA-7833 Audit and fix string builtins for long string handling .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5785/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93a53845f04e61ff446b363c78db1e49cbd5dc49 Gerrit-Change-Number: 15864 Gerrit-PatchSet: 2 Gerrit-Owner: Akos Kovacs Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 22:10:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9597: Eliminate redundant Ranger audits for column masking
Fang-Yu Rao has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/15854 ) Change subject: IMPALA-9597: Eliminate redundant Ranger audits for column masking .. IMPALA-9597: Eliminate redundant Ranger audits for column masking After IMPALA-9350, Impala is able to produce the corresponding Ranger audits when a query involves policies of column masking. However, redundant audit events could be produced due to the fact that the analysis of the TableRef containing a column involved in a column masking policy could be performed more than once for a query that has to be analyzed more than once. For example, a query consisting of a WithClause or a query that requires a rewrite operation followed by a re-analysis phase would result in RangerImpalaPlugin#evalDataMaskPolicies() being invoked multiple times, each producing an audit log entry for the same column. Moreover, for a query involving column masking policies, the corresponding audit log entries will still be generated even though there is an AuthorizationException thrown in the authorization phase. This patch fixes those two issues described above by adding a post-processing step after the analysis of a query to deduplicate the List of AuthzAuditEvent's for column masking policies. Specifically, instead of updating the List of AuthzAuditEvent's of the corresponding RangerBufferAuditHandler immediately after the deduplication, we clear this List first and add the deduplicated events back to the List only if there is no AuthorizationException. Testing: - Added three test cases in RangerAuditLogTest#testAuditsForColumnMasking() to make sure the issues above are resolved. - Verified that this patch passes the FE tests in the DEBUG build. Change-Id: I42d60130fba93d63fbc36949f2bf746b7ae2497d --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 9 files changed, 217 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15854/3 -- To view, visit http://gerrit.cloudera.org:8080/15854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I42d60130fba93d63fbc36949f2bf746b7ae2497d Gerrit-Change-Number: 15854 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9199: Add support for single query retries on cluster membership changes
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/14824 )
Change subject: IMPALA-9199: Add support for single query retries on cluster
membership changes
..
Patch Set 24:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/coordinator.cc@937
PS24, Line 937: RETURN_IF_ERROR(
This isn't valid. (and you have similar issues elsewhere)
As far as I can tell, the only way that this returns is if we decide to retry
the query and then Thread::Create returns an error, but even in that case we
need to still finish this function and eg. call UpdateExecState below with the
original error status.
In general, errors related to failing to retry aren't ever going to be overall
query errors, and we'll always need to finish whatever processing we were doing
for the original query, so probably all we need to do is log any errors from
TryQueryRetry and move on. It may even be best to log the errors from within
TryQueryRetry and not have it even return anything.
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h
File be/src/runtime/query-driver.h:
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@1
PS24, Line 1: // Licensed to the Apache Software Foundation (ASF) under one
The distinction isn't that clear, but it might make more sense to put this in
/be/src/service, since right now its:
impala-server (src/service) -> query-driver (src/runtime) ->
client-request-state (src/service) -> coordinator (src/runtime)
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@39
PS24, Line 39: QueryHandle
I think this name is confusing, esp. since there's already a QueryHandle in
beeswax. Its really more of a ClientRequestStateHandle, though that's wordy.
Maybe CRSHandle? ClientRequestHandle?
It might even be worth renaming ClientRequestState, since if anything the
QueryDriver is what really holds the current state of the client request. Maybe
QueryInstance? Not a huge deal though
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@82
PS24, Line 82: /// *Transparent* Query Retries:
Not a huge deal in this patch since it's still hidden behind a flag and
experimental, but I think that we should be cautious about saying that this is
completely transparent, since it does change what clients see, potentially in
ways that could break existing clients (eg. if you call GetRuntimeProfile() and
then do an assert that the returned profile has the same query_id as what you
requested, turning this on could cause that assert to fail).
Obviously at a minimum we need to test this with all the usual known clients
(impala-shell, impyla, jdbc/odbc, Hue, etc.), and we'll probably be making
changes to at least some of those clients for the sake of observability around
this (eg. impala-shell might want to print a message about the query getting
retried)
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@136
PS24, Line 136: Status TryQueryRetry(ClientRequestState*
client_request_state, Status* status,
The way you're both returning a Status and also returning some status
information in a Status out parameter seems confusing, and in fact it looks
like at most of the call sites of this function the Status out parameter is
never actually referenced again after the call so any info added to it is
usually dropped.
I think its probably better to just not have 'status' be an out parameter and
return use the returned status for everything, or possibly not return any
status info at all (see my other comments)
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@189
PS24, Line 189: A shared_ptr is used to allow asynchronous deletion
?
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@194
PS24, Line 194: A shared_ptr is used to allow
: /// asynchronous deletion.
?
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.h@204
PS24, Line 204: std::unique_ptr retry_exec_request_;
Now that this is just move()-ed from 'exec_request_' I'm not sure why its still
needed.
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.cc
File be/src/runtime/query-driver.cc:
http://gerrit.cloudera.org:8080/#/c/14824/24/be/src/runtime/query-driver.cc@97
PS24, Line 97: if (!client_request_state->fetched_rows()) {
This function would be more readable if you turned these into:
if (reason_not_to_retry) {
LOG << reason;
return;
}
if (another_reason) {
...
(after doing that, I personally would get rid of RetryAsync() as 1) its only
called in one place 2) a lot of its code is just DCHECK-ing things that the ifs
here enforce, so that could just be removed and 3) the names
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15881 ) Change subject: Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5997/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15881 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Gerrit-Change-Number: 15881 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 21:04:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/llvm-codegen.cc@114 PS6, Line 114: ,crc,neon,fp-armv8,crypto I am not sure about the right approach on this one. Do we want to enable these cpu attrs by default? -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 20:55:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15881 ) Change subject: Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15881 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Gerrit-Change-Number: 15881 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 20:47:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 6: (7 comments) Found some formatting and comment consistency nits. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.h File be/src/codegen/codegen-anyval.h: http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.h@90 PS6, Line 90: #ifdef __aarch64__ nit: It is possible that clang did not catch this. The preprocessor directives are generally not indented in the Impala code base. This reoccurs multiple places. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.h@91 PS6, Line 91: // add conversion function here which convert val type between aarch64 and x86 nit: This could simply describe the method instead the intent behind it, for example: Conversion function which converts val type between aarch64 and x86. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.h@94 PS6, Line 94: nit: Missing function description. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.h@101 PS6, Line 101: /// Same as above but wraps the result in a CodegenAnyVal. nit: By inserting the above two methods this comment became obsolete. Maybe ChangeRetType and CreateCallWithChangeRetType should come after this method. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.cc File be/src/codegen/codegen-anyval.cc: http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.cc@185 PS6, Line 185: // add conversion function here which convert val type between aarch64 and x86 nit: This comment is not necessary, the description is available in the header. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/codegen/codegen-anyval.cc@270 PS6, Line 270: nit: Empty line. http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/exec/text-converter.cc File be/src/exec/text-converter.cc: http://gerrit.cloudera.org:8080/#/c/15718/6/be/src/exec/text-converter.cc@310 PS6, Line 310: // For Decimal values, the return type generated by Clang is struct type rather than : // integer so casting is necessary nit: With the new condition this comment became obsolete, could you update this comment. -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 20:25:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15881 ) Change subject: Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5784/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15881 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Gerrit-Change-Number: 15881 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 20:17:45 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15881 )
Change subject: Revert "IMPALA-9718: Delete pkg_resources from
IMPALA_HOME/shell/"
..
Patch Set 1:
(443 comments)
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py
File shell/pkg_resources.py:
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@23
PS1, Line 23: import sys, os, zipimport, time, re, imp, types
flake8: E402 module level import not at top of file
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@23
PS1, Line 23: ,
flake8: E401 multiple imports on one line
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@24
PS1, Line 24: from urlparse import urlparse, urlunparse
flake8: E402 module level import not at top of file
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@32
PS1, Line 32: from os import utime
flake8: E402 module level import not at top of file
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@40
PS1, Line 40: from os import open as os_open
flake8: E402 module level import not at top of file
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@41
PS1, Line 41: from os.path import isdir, split
flake8: E402 module level import not at top of file
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@52
PS1, Line 52: def _bypass_ensure_directory(name, mode=0777):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@68
PS1, Line 68: def get_supported_platform():
flake8: E303 too many blank lines (8)
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@81
PS1, Line 81: ;
flake8: E702 multiple statements on one line (semicolon)
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: E303 too many blank lines (21)
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'set_extraction_path' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'resource_string' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'resource_filename' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'resource_isdir' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'cleanup_resources' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'resource_stream' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'resource_listdir' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@109
PS1, Line 109: __all__ = [
flake8: F822 undefined name u'resource_exists' in __all__
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@111
PS1, Line 111:
flake8: E241 multiple spaces after ','
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@126
PS1, Line 126: ,
flake8: E231 missing whitespace after ','
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@126
PS1, Line 126: ,
flake8: E231 missing whitespace after ','
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@126
PS1, Line 126: ,
flake8: E231 missing whitespace after ','
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@150
PS1, Line 150: class ResolutionError(Exception):
flake8: E302 expected 2 blank lines, found 0
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@153
PS1, Line 153: +
flake8: E226 missing whitespace around arithmetic operator
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@155
PS1, Line 155: class VersionConflict(ResolutionError):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@158
PS1, Line 158: class DistributionNotFound(ResolutionError):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@161
PS1, Line 161: class UnknownExtra(ResolutionError):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@163
PS1, Line 163: _provider_factories = {}
flake8: E305 expected 2 blank lines after class or function definition, found 0
http://gerrit.cloudera.org:8080/#/c/15881/1/shell/pkg_resources.py@166
PS1, Line 166:
flake8: E221 multiple spaces before operator
[Impala-ASF-CR] Revert "IMPALA-9718: Delete pkg resources from IMPALA HOME/shell/"
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15881 Change subject: Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" .. Revert "IMPALA-9718: Delete pkg_resources from IMPALA_HOME/shell/" The fix for IMPALA-9718 introduced test failures on Centos 7. See IMPALA-9735. This reverts commit 75d98b4b081df95b58d7388da39bb1ec7c2f4f67. Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e --- M LICENSE.txt M shell/make_shell_tarball.sh A shell/pkg_resources.py 3 files changed, 2,702 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/15881/1 -- To view, visit http://gerrit.cloudera.org:8080/15881 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id09c55435f432a8626a45079f58860d6e27ac55e Gerrit-Change-Number: 15881 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-7833 Audit and fix string builtins for long string handling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15864 ) Change subject: IMPALA-7833 Audit and fix string builtins for long string handling .. Patch Set 2: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5780/ -- To view, visit http://gerrit.cloudera.org:8080/15864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93a53845f04e61ff446b363c78db1e49cbd5dc49 Gerrit-Change-Number: 15864 Gerrit-PatchSet: 2 Gerrit-Owner: Akos Kovacs Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 20:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15676 ) Change subject: IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15676 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifac41ffb3e1742ffb6a969cb1c368d6d93c23357 Gerrit-Change-Number: 15676 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 07 May 2020 19:45:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/15676 ) Change subject: IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter .. IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter Ported BlockBloomFilter related source files from Kudu upstream to Impala be/src/kudu/util. The git hash of Kudu to take these files is 389d4f1e1c. Testing: Passed core tests. Change-Id: Ifac41ffb3e1742ffb6a969cb1c368d6d93c23357 --- M be/src/kudu/util/CMakeLists.txt M be/src/kudu/util/block_bloom_filter-test.cc M be/src/kudu/util/block_bloom_filter.cc M be/src/kudu/util/block_bloom_filter.h A be/src/kudu/util/block_bloom_filter.proto M be/src/kudu/util/block_bloom_filter_avx2.cc M be/src/kudu/util/hash.proto M be/src/kudu/util/hash_util-test.cc M be/src/kudu/util/hash_util.h M be/src/kudu/util/memory/arena-test.cc M be/src/kudu/util/memory/arena.cc M be/src/kudu/util/memory/arena.h 12 files changed, 785 insertions(+), 101 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/15676/7 -- To view, visit http://gerrit.cloudera.org:8080/15676 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifac41ffb3e1742ffb6a969cb1c368d6d93c23357 Gerrit-Change-Number: 15676 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5995/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 07 May 2020 18:47:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Wenzhe Zhou has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu Defined the BloomFilter class as the wrapper of kudu::BlockBloomFilter. impala::BloomFilter build runtime bloom filter in kudu::BlockBloomFilter APIs with FastHash as default hash algorithm. Removed the duplicated functions from impala::BloomFillter class. Pushed down bloom filter to Kudu through Kudu clinet API. Added a new query option to set enabled runtime filter types, which only affect Kudu scan node now. By default, both bloom filter and min-max filter will be enabled for Kudu. Added new test cases in PlannerTest and end-end runtime_filters test for pushing down bloom filter to Kudu. Updated bloom-filter-benchmark for the bloom-filter implementation change. Testing: - Passed test_kudu.py - Passed end-end test_runtime_filters.py. - Passed frontend Planner tests. - Ran single_node_perf_run.py on TPC-H with scale as 30 for parquet and Kudu. Verified that new hash function and bloom-filter implementation don't cause regressions for HDFS bloom filters. - Ran bloom-filter-benchmark and verified there is no regression due to bloom-filter implementation changes. Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 --- M be/CMakeLists.txt M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/codegen/gen_ir_descriptions.py M be/src/exec/filter-context.cc M be/src/exec/kudu-scanner.cc M be/src/runtime/raw-value-ir.cc M be/src/runtime/raw-value.h M be/src/runtime/raw-value.inline.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter-ir.cc M be/src/runtime/runtime-filter.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/bloom-filter-ir.cc M be/src/util/bloom-filter-test.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/hash-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/kudu-update.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test A testdata/workloads/functional-query/queries/QueryTest/all_runtime_filters.test M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test M tests/query_test/test_runtime_filters.py 34 files changed, 1,344 insertions(+), 620 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/15683/11 -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/15683 )
Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
..
Patch Set 9:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/15683/9/be/CMakeLists.txt
File be/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15683/9/be/CMakeLists.txt@234
PS9, Line 234: "-DKUDU_HEADERS_NO_STUBS"
> nit formatting: trailing whitespace and you can wrap this into the rest of
will fix as suggested.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/filter-context.cc
File be/src/exec/filter-context.cc:
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/filter-context.cc@91
PS9, Line 91: //local_bloom_filter->Insert(val, expr_eval->root().type());
> I guess this was left by accident?
Right, will remove these two lines.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/filter-context.cc@106
PS9, Line 106: // An example of the generated code for TPCH-Q2: RF002 ->
n_regionkey
> This needs to be updated (though of course it will only change slightly). I
will add LlvmCodeGen::Print(*fn) at the end of function to dump the codegen
functions, then update the sample in the functions' comments.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/kudu-scanner.cc
File be/src/exec/kudu-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/kudu-scanner.cc@232
PS9, Line 232: }
: else if
> nit: formatting ('else if' should go on the same line as prior '}')
Will fix it.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/kudu-scanner.cc@234
PS9, Line 234: continue;
> Brief comment, eg. 'This filter won't actually remove any rows so we don't
will add comments
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/kudu-scanner.cc@236
PS9, Line 236: else
> nit: it doesn't actually change how the code works, but it would be better
will remove "else"
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/kudu-scanner.cc@238
PS9, Line 238: if (filter != nullptr) {
> I think we can save ourselves some duplication and an indention level by ch
Agree, will change the code.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/exec/kudu-scanner.cc@239
PS9, Line 239: auto it = ctx.filter->filter_desc().planid_to_target_ndx.find(
: scan_node_->id());
: const TRuntimeFilterTargetDesc _desc =
: ctx.filter->filter_desc().targets[it->second];
: const string _name = target_desc.kudu_col_name;
: DCHECK(col_name != "");
> This is duplicated below, I think we can eliminate the duplication by movin
That's right. Will remove the duplication.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h
File be/src/util/bloom-filter.h:
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h@53
PS9, Line 53: class BloomFilterBufferAllocator;
> I don't think this is necessary
will remove it.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h@73
PS9, Line 73: BloomFilterBufferAllocator
> nit: maybe name this ImpalaBloomFilterBufferAllocator to ensure there's no
Will rename it as suggested
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h@91
PS9, Line 91: std::shared_ptr
Clone() const override {
> If this is something that's really only used in Kudu's internal testing and
Will add LOG(FATAL) and simplify the logic of allocate/close functions.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h@111
PS9, Line 111: /// A BloomFilter stores sets of items and offers a query
operation indicating whether or
> This class comment could probably use some updating to reflect the fact tha
Will update the comments.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h@171
PS9, Line 171: void Insert(void* val, const ColumnType& col_type) noexcept;
> Is this version of Insert() used anywhere? Same with the equivalent Find()
Will remove them
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.h@241
PS9, Line 241: HashAlgorithm hash_algorithm_;
> As far as I can tell, this isn't actually getting used anywhere, and I don'
Will remove it.
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.cc
File be/src/util/bloom-filter.cc:
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.cc@205
PS9, Line 205: if (is_allocated_) Close();
> I suspect its probably supposed to be guaranteed that FreeBuffer() is alway
Will add LOG(DFATAL).
http://gerrit.cloudera.org:8080/#/c/15683/9/be/src/util/bloom-filter.cc@226
PS9, Line 226: Close(); // Ensure that any previously allocated memory is
released.
> I wonder if this is really necessary - I would hope that Kudu provides a gu
Will LOG(DFALTAL) if is allocated.
[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5783/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 17:43:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9716: Add jitter to the exponential backoff in status reporting
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15860 ) Change subject: IMPALA-9716: Add jitter to the exponential backoff in status reporting .. Patch Set 4: Code-Review+2 gvo failed due to maven failing to download some dependencies. retrying -- To view, visit http://gerrit.cloudera.org:8080/15860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id05c224517aa606057117328f480dfa98676b923 Gerrit-Change-Number: 15860 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 17:43:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9722: Consolidate avg size calculation in PerColumnStats
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15878 ) Change subject: IMPALA-9722: Consolidate avg_size calculation in PerColumnStats .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae1efb1c568c67dff6f25887c2ea2b8b249eea4b Gerrit-Change-Number: 15878 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 17:20:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5782/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 16:59:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 16:59:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 16:23:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9722: Consolidate avg size calculation in PerColumnStats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15878 ) Change subject: IMPALA-9722: Consolidate avg_size calculation in PerColumnStats .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5991/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iae1efb1c568c67dff6f25887c2ea2b8b249eea4b Gerrit-Change-Number: 15878 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 16:11:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5994/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 07 May 2020 16:11:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9729: consistent GetExecSummary() behaviour
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15868 ) Change subject: IMPALA-9729: consistent GetExecSummary() behaviour .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5993/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Gerrit-Change-Number: 15868 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 07 May 2020 16:08:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15818 ) Change subject: IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5992/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d Gerrit-Change-Number: 15818 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 May 2020 15:58:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9199: Add support for single query retries on cluster membership changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14824 ) Change subject: IMPALA-9199: Add support for single query retries on cluster membership changes .. Patch Set 24: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5990/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14824 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e4a0e72a9bf8ec10b91639aefd81bef17886ddd Gerrit-Change-Number: 14824 Gerrit-PatchSet: 24 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 07 May 2020 15:50:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15869 Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support Impala 4 moved to using CDP versions for components, which involves adopting Hive 3. This removes the old code supporting CDH components and Hive 2. Specifically, it does the following: 1. Remove USE_CDP_HIVE and default to the values from USE_CDP_HIVE=true. USE_CDP_HIVE now has no effect on the Impala environment. This also means that bin/jenkins/build-all-flag-combinations.sh no longer include USE_CDP_HIVE=false as a configuration. 2. Remove USE_CDH_KUDU and default to getting Impala from the native toolchain. 3. Ban IMPALA_HIVE_MAJOR_VERSION<3 and remove related code, including the IMPALA_HIVE_MAJOR_VERSION=2 maven profile in fe/pom.xml. There is a fair amount of code that still references the Hive major version. Upstream Hive is now working on Hive 4, so there is a high likelihood that we'll need some code to deal with that transition. This leaves some code (such as maven profiles) and test logic in place. Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 --- M README-build.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M fe/pom.xml D fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java D fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M testdata/bin/create-load-data.sh M testdata/bin/run-hive-server.sh M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 11 files changed, 81 insertions(+), 1,108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/15869/3 -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9731: Remove USE CDP HIVE�lse and Hive 2 support
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15869 ) Change subject: IMPALA-9731: Remove USE_CDP_HIVE=false and Hive 2 support .. Patch Set 3: GVO failure is IMPALA-9729, not related to this. -- To view, visit http://gerrit.cloudera.org:8080/15869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85e849beaf4e19dda4092874185462abd2ec608 Gerrit-Change-Number: 15869 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 07 May 2020 15:42:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3926: part 2: avoid setting LD LIBRARY PATH
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14494 ) Change subject: IMPALA-3926: part 2: avoid setting LD_LIBRARY_PATH .. Patch Set 25: Thanks for the review! Glad to get this fixed finally! -- To view, visit http://gerrit.cloudera.org:8080/14494 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I61c83e6cca6debb87a12135e58ee501244bc9603 Gerrit-Change-Number: 14494 Gerrit-PatchSet: 25 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 15:40:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9729: consistent GetExecSummary() behaviour
Hello Thomas Tauber-Marshall, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15868 to look at the new patch set (#2). Change subject: IMPALA-9729: consistent GetExecSummary() behaviour .. IMPALA-9729: consistent GetExecSummary() behaviour Before IMPALA-9380, GetExecSummary() for archived queries without a coordinator returned a blank TExecSummary, but for in-flight queries it returned a "not found" error. This was masked by the synchronous query unregistration, but now has made an impala-shell test flaky, because the close() RPC returns to the shell before the query is archived, so the following GetExecSummary() call may go down either code path. This change makes the behaviour consistent for both code paths and preserves impala-shell behaviour. Testing: Looped TestImpalaShell.test_summary to make sure flakiness was fixed. Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b --- M be/src/service/impala-server.cc M common/thrift/ImpalaService.thrift 2 files changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15868/2 -- To view, visit http://gerrit.cloudera.org:8080/15868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Gerrit-Change-Number: 15868 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9729: consistent GetExecSummary() behaviour
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15868 Change subject: IMPALA-9729: consistent GetExecSummary() behaviour .. IMPALA-9729: consistent GetExecSummary() behaviour Before IMPALA-9380, GetExecSummary() for archived queries without a coordinator returned a blank TExecSummary, but for in-flight queries it returned a "not found" error. This was masked by the synchronous query unregistration, but now has made an impala-shell test flaky, because the close() RPC returns to the shell before the query is archived, so the following GetExecSummary() call may go down either code path. This change makes the behaviour consistent for both code paths and preserves impala-shell behaviourp. Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b --- M be/src/service/impala-server.cc M common/thrift/ImpalaService.thrift 2 files changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15868/1 -- To view, visit http://gerrit.cloudera.org:8080/15868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If2b2ab118c4769e80284b2a3390577dd5f5e888b Gerrit-Change-Number: 15868 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15818 ) Change subject: IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15818/5/be/src/exec/acid-metadata-utils.h File be/src/exec/acid-metadata-utils.h: http://gerrit.cloudera.org:8080/#/c/15818/5/be/src/exec/acid-metadata-utils.h@26 PS5, Line 26: ValidWriteIdList > Added unit tests. Added thrift object TValidWriteIdList. -- To view, visit http://gerrit.cloudera.org:8080/15818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d Gerrit-Change-Number: 15818 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 May 2020 15:31:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15818 to look at the new patch set (#7). Change subject: IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list .. IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list Minor compactions can compact several delta directories into a single delta directory. The current directory filtering algorithm had to be modified to handle minor compacted directories and prefer those over plain delta directories. This happens in the Frontend, mostly in AcidUtils.java. Hive Streaming Ingestion writes similar delta directories, but they might contain rows Impala cannot see based on its valid write id list. E.g. we can have the following delta directory: full_acid/delta_001_010/ # minWriteId: 1 # maxWriteId: 10 This delta dir contains rows with write ids between 1 and 10. But maybe we are only allowed to see write ids less than 5. Therefore we need to check the ACID write id column (named originalTransaction) to determine which rows are valid. Delta directories written by Hive Streaming don't have a visibility txn id, so we can recognize them based on the directory name. If there's a visibilityTxnId and it is committed => every row is valid. If there's no visibilityTxnId then it was created via Hive Streaming, therefore we need to validate rows. Fortunately Hive Streaming writes rows with different write ids into different ORC stripes, therefore we don't need to validate the write id per row. If we had statistics, we could validate per stripe, but since Hive Streaming doesn't write statistics we validate the write id per ORC row batch (an alternative could be to do a 2-pass read, first we'd read a single value from each stripe's 'currentTransaction' field, then we'd read the stripe if the write id is valid). Testing * the frontend logic is tested in AcidUtilsTest * the backend row validation is tested in test_acid_row_validation Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d --- M be/src/exec/CMakeLists.txt A be/src/exec/acid-metadata-utils-test.cc A be/src/exec/acid-metadata-utils.cc A be/src/exec/acid-metadata-utils.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/orc-metadata-utils.cc M be/src/exec/orc-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M common/thrift/CatalogObjects.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M testdata/bin/generate-schema-statements.py M testdata/data/README A testdata/data/streaming.orc M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/acid-row-validation-0.test A testdata/workloads/functional-query/queries/QueryTest/acid-row-validation-1.test A testdata/workloads/functional-query/queries/QueryTest/acid-row-validation-2.test M testdata/workloads/functional-query/queries/QueryTest/acid.test M testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test A tests/query_test/test_acid_row_validation.py A tests/util/acid_txn.py 39 files changed, 1,398 insertions(+), 188 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15818/7 -- To view, visit http://gerrit.cloudera.org:8080/15818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d Gerrit-Change-Number: 15818 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9722: Consolidate avg size calculation in PerColumnStats
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15878 Change subject: IMPALA-9722: Consolidate avg_size calculation in PerColumnStats .. IMPALA-9722: Consolidate avg_size calculation in PerColumnStats This change refactors the 'avg_size' calculation, 'total_width' will be storing the sum of row widths till a Finalize call calculates the average column size. Testing: - Added unit test to verify the aggregation result Change-Id: Iae1efb1c568c67dff6f25887c2ea2b8b249eea4b --- M be/src/exec/incr-stats-util-test.cc M be/src/exec/incr-stats-util.cc M be/src/exec/incr-stats-util.h 3 files changed, 27 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/15878/1 -- To view, visit http://gerrit.cloudera.org:8080/15878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iae1efb1c568c67dff6f25887c2ea2b8b249eea4b Gerrit-Change-Number: 15878 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-9199: Add support for single query retries on cluster membership changes
Hello Thomas Tauber-Marshall, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14824
to look at the new patch set (#24).
Change subject: IMPALA-9199: Add support for single query retries on cluster
membership changes
..
IMPALA-9199: Add support for single query retries on cluster membership changes
Adds the core logic for transparently retrying queries that fail due to
cluster membership changes (IMPALA-9124).
Query retries are triggered if (1) a node has been removed from the
cluster membership by a statestore update (rather than cancelling all
queries running on the removed node, queries are retried), or (2) if a
query fails and as a result, blacklists a node. Either event is
considered a cluster membership change as it affects what nodes a query
will be scheduled on. The assumption is that a retry of the query with
the updated cluster membership will succeed.
A query retry is modelled as a brand new query, with its own query id.
This simplifies the implementation and the resulting runtime profiles
when queries are retried.
Core Features:
* Retries are transparent to the user; no modification to client
libraries are necessary to support query retries
* Retried queries skip all fe/ parsing, planning, authorization, etc.
* Retries are configurable ('retry_failed_queries') and are off by
default
Implementation:
* When a query is retried, the original query is cancelled, the new
query is created, registered, and started, and then the original query
is closed
* A new layer of abstraction between the ImpalaServer and
ClientRequestState has been added; it is called the QueryDriver
* Each ClientRequestState is treated as a single attempt of a query, and
the QueryDriver owns all ClientRequestStates for a query
* ClientRequestState has a new state object called RetryState; a
ClientRequestState can either be NOT_RETRIED, RETRYING, or RETRIED
* The QueryDriver owns the TExecRequest for the query as well, it is
re-used for each query retry
Observability:
* Users can tell if a query is retried using runtime profiles and the
Impala Web UI
* Runtime profiles of queries that fail and then are retried will have:
* "Retry Status: RETRIED"
* "Retry Cause: [the error that triggered the retry]"
* "Retried Query Id: [the query id of the retried query]"
* Runtime profiles of the retried query (e.g. the second attempt of the
query) will include:
* "Original Query Id: [the query id of the original query]"
* The Impala Web UI will list all retried queries as being in the
"RETRIED" state
Testing:
* Added E2E tests in test_query_retries.py; looped tests for a few days
* Added a stress test query_retries_stress_runner.py that runs concurrent
streams of a TPC-{H,DS} workload and randomly kills impalads
* Ran the stress test with various configurations: tpch on parquet,
tpcds on parquet, tpch 30 GB on parquet (one stream), tpcds 30 GB on
parquet (one stream), tpch on text, tpcds on text
* Ran exhaustive tests
* Ran exhaustive tests with 'retry_failed_queries' set to true, no
unexpected failures
* Ran 30 GB TPC-DS workload on a 3 node cluster, randomly restarted
impalads, and manually verified that queries were retried
* Manually tested retries work with various clients, specifically the
impala-shell and Hue
* Ran core tests and query retry stress test against an ASAN build
* Ran concurrent_select.py to stress query cancellation
* Ran be/ tests against a TSAN build, filed IMPALA-9730 as a follow up
Limitations:
* There are several limitations that are listed out in the parent JIRA
Change-Id: I2e4a0e72a9bf8ec10b91639aefd81bef17886ddd
---
M be/src/benchmarks/process-wide-locks-benchmark.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
A be/src/runtime/query-driver.cc
A be/src/runtime/query-driver.h
M be/src/service/CMakeLists.txt
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/control-service.cc
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
R be/src/service/query-driver-map.cc
A be/src/service/query-driver-map.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/testutil/impalad-query-executor.cc
M be/src/testutil/impalad-query-executor.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/generate_error_codes.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
A tests/custom_cluster/test_query_retries.py
A tests/stress/query_retries_stress_runner.py
28 files changed, 2,458 insertions(+), 510 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/14824/24
--
To view, visit
[Impala-ASF-CR] IMPALA-7833 Audit and fix string builtins for long string handling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15864 ) Change subject: IMPALA-7833 Audit and fix string builtins for long string handling .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5780/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93a53845f04e61ff446b363c78db1e49cbd5dc49 Gerrit-Change-Number: 15864 Gerrit-PatchSet: 2 Gerrit-Owner: Akos Kovacs Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 14:45:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15818 ) Change subject: IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5989/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d Gerrit-Change-Number: 15818 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 May 2020 13:22:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/15818 )
Change subject: IMPALA-9512: Full ACID Milestone 2: Validate rows against the
valid write id list
..
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15818/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15818/5//COMMIT_MSG@35
PS5, Line 35: but since Hive Streaming doesn't write
: statistics we validate the write id per ORC row batch.
> I think there is still a way to optimize this without statistics:
I added a sentence, but I'm a bit skeptic about the performance gain of the
proposed approach. Filling an array with the same values should be pretty fast
on modern CPUs.
And creating another OrcRowReader, OrcRowReaderOptions, single-element orc
batches for each stripes would also have some overhead.
So it's hard to tell without measurements, but I wouldn't complicate the code
until we run into perf issues.
http://gerrit.cloudera.org:8080/#/c/15818/5/be/src/exec/acid-metadata-utils.h
File be/src/exec/acid-metadata-utils.h:
http://gerrit.cloudera.org:8080/#/c/15818/5/be/src/exec/acid-metadata-utils.h@26
PS5, Line 26: ValidWriteIdList
> optional: Some unit tests would be nice for this class as it hides some rea
Added unit tests.
A thrift ValidWriteIdList needs extra conversions between the thrift structure
and the Java ValidWriteIdList. But I agree that it'd make the code a bit safer
since we wouldn't depend on a plain string format. I'll add that in my next PS.
http://gerrit.cloudera.org:8080/#/c/15818/3/be/src/exec/hdfs-orc-scanner.cc
File be/src/exec/hdfs-orc-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/15818/3/be/src/exec/hdfs-orc-scanner.cc@517
PS3, Line 517: tch* row_batch) {
> The handling of this case is still not clear to me.
Good catch. Besides the current write id column we also selected the column
with the "maximum column id" from the table. It's basically the last column, or
the deepest child of the last column if last column is nested.
Now we only select the current write id column.
--
To view, visit http://gerrit.cloudera.org:8080/15818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d
Gerrit-Change-Number: 15818
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 07 May 2020 12:51:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15818 to look at the new patch set (#6). Change subject: IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list .. IMPALA-9512: Full ACID Milestone 2: Validate rows against the valid write id list Minor compactions can compact several delta directories into a single delta directory. The current directory filtering algorithm had to be modified to handle minor compacted directories and prefer those over plain delta directories. This happens in the Frontend, mostly in AcidUtils.java. Hive Streaming Ingestion writes similar delta directories, but they might contain rows Impala cannot see based on its valid write id list. E.g. we can have the following delta directory: full_acid/delta_001_010/ # minWriteId: 1 # maxWriteId: 10 This delta dir contains rows with write ids between 1 and 10. But maybe we are only allowed to see write ids less than 5. Therefore we need to check the ACID write id column (named originalTransaction) to determine which rows are valid. Delta directories written by Hive Streaming don't have a visibility txn id, so we can recognize them based on the directory name. If there's a visibilityTxnId and it is committed => every row is valid. If there's no visibilityTxnId then it was created via Hive Streaming, therefore we need to validate rows. Fortunately Hive Streaming writes rows with different write ids into different ORC stripes, therefore we don't need to validate the write id per row. If we had statistics, we could validate per stripe, but since Hive Streaming doesn't write statistics we validate the write id per ORC row batch (an alternative could be to do a 2-pass read, first we'd read a single value from each stripe's 'currentTransaction' field, then we'd read the stripe if the write id is valid). Testing * the frontend logic is tested in AcidUtilsTest * the backend row validation is tested in test_acid_row_validation Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d --- M be/src/exec/CMakeLists.txt A be/src/exec/acid-metadata-utils-test.cc A be/src/exec/acid-metadata-utils.cc A be/src/exec/acid-metadata-utils.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/orc-metadata-utils.cc M be/src/exec/orc-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M testdata/bin/generate-schema-statements.py M testdata/data/README A testdata/data/streaming.orc M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/acid-row-validation-0.test A testdata/workloads/functional-query/queries/QueryTest/acid-row-validation-1.test A testdata/workloads/functional-query/queries/QueryTest/acid-row-validation-2.test M testdata/workloads/functional-query/queries/QueryTest/acid.test M testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test A tests/query_test/test_acid_row_validation.py A tests/util/acid_txn.py 32 files changed, 1,306 insertions(+), 178 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15818/6 -- To view, visit http://gerrit.cloudera.org:8080/15818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5ed74585a2d73ebbcee763b0545be4412926299d Gerrit-Change-Number: 15818 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5987/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 10:22:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9630 Keep blocking queue cache line aligned on aarch64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15705 ) Change subject: IMPALA-9630 Keep blocking queue cache line aligned on aarch64 .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5986/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 Gerrit-Change-Number: 15705 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 10:21:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15755 ) Change subject: IMPALA-9676 Add aarch64 compile options for clang .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5988/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 Gerrit-Change-Number: 15755 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 09:58:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 27: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5985/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 27 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 May 2020 09:56:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/15718 ) Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64 .. Patch Set 6: Hi, Tim, this has been fixed as your request -- To view, visit http://gerrit.cloudera.org:8080/15718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11 Gerrit-Change-Number: 15718 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 09:29:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9630 Keep blocking queue cache line aligned on aarch64
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/15705 ) Change subject: IMPALA-9630 Keep blocking queue cache line aligned on aarch64 .. Patch Set 7: Hi, Tim, this has been fixed as your request -- To view, visit http://gerrit.cloudera.org:8080/15705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 Gerrit-Change-Number: 15705 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 May 2020 09:30:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9645 Port LLVM codegen to adapt aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/15718 )
Change subject: IMPALA-9645 Port LLVM codegen to adapt aarch64
..
IMPALA-9645 Port LLVM codegen to adapt aarch64
On aarch64, the Lowered type of struct {bool, int128} is form
{ {i8}, {i128} }. No padding add. This is different with x86-64,
which is { {i8}, {15*i8}, {i128} } with padding add automatically.
And here also add some type conversion between x86 and aarch64 data types.
And also add some aarch64 cpu's feature.
Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
---
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/codegen-anyval.h
M be/src/codegen/llvm-codegen.cc
M be/src/exec/text-converter.cc
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-fn-call.cc
6 files changed, 131 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15718/6
--
To view, visit http://gerrit.cloudera.org:8080/15718
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f30ee84ea9bf5245da88154632bb69079103d11
Gerrit-Change-Number: 15718
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9630 Keep blocking queue cache line aligned on aarch64
zhaoren...@hotmail.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/15705 ) Change subject: IMPALA-9630 Keep blocking queue cache line aligned on aarch64 .. IMPALA-9630 Keep blocking queue cache line aligned on aarch64 On aarch64, the DNCHECK_NE in BlockingQueue construct function will fail. So here use a different method to keep the 'put' class members and 'get' class members cache line aligned. Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 --- M be/src/common/compiler-util.h M be/src/exec/hdfs-scanner.h M be/src/util/aligned-new.h M be/src/util/blocking-queue.h 4 files changed, 13 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15705/7 -- To view, visit http://gerrit.cloudera.org:8080/15705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia169e15d3a225f1e4780e671b8cce680b176c171 Gerrit-Change-Number: 15705 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9676 Add aarch64 compile options for clang
zhaoren...@hotmail.com has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/15755 ) Change subject: IMPALA-9676 Add aarch64 compile options for clang .. IMPALA-9676 Add aarch64 compile options for clang Add signed-char and armv8a and crc compile options to clang Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 --- M be/CMakeLists.txt 1 file changed, 7 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/15755/4 -- To view, visit http://gerrit.cloudera.org:8080/15755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69a5ff64bbd4427dd87ec6e884251e76d6a73122 Gerrit-Change-Number: 15755 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
