[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 3: The build was aborted. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 07:55:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8361: Propagate predicates of outer-joined InlineView
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15047 ) Change subject: IMPALA-8361: Propagate predicates of outer-joined InlineView .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5283/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c23a45aeb5dd1aa06a95c9aa8628ecbe37ef2c1 Gerrit-Change-Number: 15047 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 05 Feb 2020 07:54:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8361: Propagate predicates of outer-joined InlineView
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15047 ) Change subject: IMPALA-8361: Propagate predicates of outer-joined InlineView .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5282/ -- To view, visit http://gerrit.cloudera.org:8080/15047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c23a45aeb5dd1aa06a95c9aa8628ecbe37ef2c1 Gerrit-Change-Number: 15047 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 05 Feb 2020 07:11:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15151 ) Change subject: IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idefba98ecd65efbd47b1618291330795ef13b910 Gerrit-Change-Number: 15151 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 05 Feb 2020 06:23:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15151 ) Change subject: IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate .. IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate The test uses a thread pool of 3 slots to run 3 concurrent CreateTable statements. Each thread creates a client and closes it at the end of the task(run_create_table_if_not_exists). The first 2 thread tasks start at the same time. Then after one second, the 3rd thread task starts. Ideally the 3 tasks should be executed in 3 different threads of the thread pool. However, if any of the first 2 tasks finish in 1s, the 3rd task could be executed by reusing the thread. Since the client in the thread is closed, the task fails. This patch removes the close operation at the end of the task. Connection of the thread can be reused and will be closed when the thread terminates. Also fix the same problem in test_concurrent_ddls.py. Tests: - Run test_concurrent_ddls.py and test_concurrent_kudu_create.py. In the logs of impalad, observe that connections are being reused and closed at the end of the tests. Change-Id: Idefba98ecd65efbd47b1618291330795ef13b910 Reviewed-on: http://gerrit.cloudera.org:8080/15151 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_concurrent_ddls.py M tests/custom_cluster/test_concurrent_kudu_create.py 2 files changed, 4 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idefba98ecd65efbd47b1618291330795ef13b910 Gerrit-Change-Number: 15151 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-9352 [DOCS] Document Impala column masking support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15162 ) Change subject: IMPALA-9352 [DOCS] Document Impala column masking support .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/539/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/15162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60a1453d9ff4b25ba9e75dfd9fa2dc41006e32e Gerrit-Change-Number: 15162 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Feb 2020 04:54:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9352 [DOCS] Document Impala column masking support
kh...@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15162 Change subject: IMPALA-9352 [DOCS] Document Impala column masking support .. IMPALA-9352 [DOCS] Document Impala column masking support Added the Ranger Column Masking section consisting of an intro, how to enable column masking, and built-in mask types. Change-Id: Ia60a1453d9ff4b25ba9e75dfd9fa2dc41006e32e --- M docs/topics/impala_authorization.xml 1 file changed, 88 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/15162/1 -- To view, visit http://gerrit.cloudera.org:8080/15162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia60a1453d9ff4b25ba9e75dfd9fa2dc41006e32e Gerrit-Change-Number: 15162 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] IMPALA-9352 [DOCS] Document Impala column masking support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15162 ) Change subject: IMPALA-9352 [DOCS] Document Impala column masking support .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/539/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/15162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60a1453d9ff4b25ba9e75dfd9fa2dc41006e32e Gerrit-Change-Number: 15162 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Feb 2020 04:48:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 05 Feb 2020 04:18:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. IMPALA-9330: Fix column masking in nested tables + enable column masking by default Column masking policies on primitive columns of a table which contains nested types (though they won't be masked) will cause query failures. To be specifit, if tableA(id int, int_array array) has a masking policy on column "id", all queries on "tableA" will fail, e.g. select id from tableA; select t.id, a.item from tableA t, t.int_array a; Column masking is implemented by wrapping the underlying table/view with a table masking view. However, as we don't support nested types in SelectList, the table masking view can't expose nested columns of the masked table, which causes collection refs not being resolved correctly. This patch fixes the issue by 2 steps: 1) Expose nested columns of the underlying table in the output Type of the table masking view (see InlineViewRef#createTupleDescriptor()). So nested Paths in the original query block can be resolved. 2) For such kind of Paths, resolved them again inside the table masking view. So they can point to the underlying table as what they mean (see Analyzer#resolvePathWithMasking()). TupleDescriptor of such kind of table masking view won't be materialized since the view is simple enough that its query plan is just a ScanNode of the underlying table. The whole query plan can be stitched as if the table is not masked. Note that one day when we support nested columns in SelectList, we may don't need these 2 hacks. This patch also adds some TRACE level loggings to improve debuggability, and enables column masking by default. Test changes in TestRanger.test_column_masking: - Add column masking policy on a table containing nested types. - Add queries on the masked tables. Some queries are borrowed from existing tests for nested types. Tests: - Run CORE tests. Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Reviewed-on: http://gerrit.cloudera.org:8080/15108 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M tests/authorization/test_ranger.py 16 files changed, 538 insertions(+), 24 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9358: Query slowdown with inline views and hundreds of columns
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/15157 )
Change subject: IMPALA-9358: Query slowdown with inline views and hundreds of
columns
..
Patch Set 1:
(2 comments)
Thanks for making this quick-fix. Comments are more for my understanding than
about the code change.
This change seems small enough but did you have a chance to run any EE/FE tests
that touch this path as a sanity check that nothing went there?
http://gerrit.cloudera.org:8080/#/c/15157/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15157/1//COMMIT_MSG@25
PS1, Line 25: Performance testing with a query with 1000 expressions of the
Not a blocking comment, feel free to skip it but would it be possible to
include the performance improvement you observed here?
http://gerrit.cloudera.org:8080/#/c/15157/1/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
File fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java:
http://gerrit.cloudera.org:8080/#/c/15157/1/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java@77
PS1, Line 77: for (int i = hint; i < lhs_.size(); ++i) {
: if (lhsExpr.equals(lhs_.get(i))) return rhs_.get(i);
: }
Just trying to understand this. Is it safe to assume that most equality
searches end in this for loop?
--
To view, visit http://gerrit.cloudera.org:8080/15157
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77423d9c10e1edbb505cb210b5c072281b5d7cfc
Gerrit-Change-Number: 15157
Gerrit-PatchSet: 1
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 05 Feb 2020 03:55:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5126/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 02:18:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 3: > Does ignore_noninstrumented_modules work on Linux? Looking at the > LLVM source, https://reviews.llvm.org/D61708 doesn't look like it's > been merged. > > FWIW, it didn't exist (and certainly didn't work on Linux) when > Kudu's TSAN support was added, which is why we went the other > direction and recompile all of our dependencies with > -fsanitize=thread if doing a TSAN build. @Adar, tried to dig through the LLVM code, but couldn't find any evidence that this has been fixed. FWIW I've tested it multiple times locally and the flag works, and I ran it on Jenkins and the flag works as expected. -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 02:09:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8361: Propagate predicates of outer-joined InlineView
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15047 ) Change subject: IMPALA-8361: Propagate predicates of outer-joined InlineView .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5282/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c23a45aeb5dd1aa06a95c9aa8628ecbe37ef2c1 Gerrit-Change-Number: 15047 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 05 Feb 2020 01:48:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5125/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 01:43:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15151 ) Change subject: IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idefba98ecd65efbd47b1618291330795ef13b910 Gerrit-Change-Number: 15151 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 05 Feb 2020 01:34:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15151 ) Change subject: IMPALA-9289: Fix flakiness in TestConcurrentKuduCreate .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5281/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idefba98ecd65efbd47b1618291330795ef13b910 Gerrit-Change-Number: 15151 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 05 Feb 2020 01:34:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/15116/4/be/src/common/init.cc File be/src/common/init.cc: http://gerrit.cloudera.org:8080/#/c/15116/4/be/src/common/init.cc@423 PS4, Line 423: // https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual#non-instrumented-code), line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 01:32:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Hello Adar Dembo, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15116 to look at the new patch set (#4). Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. IMPALA-5904: Add full_tsan option and fix several TSAN bugs This patch adds an additional build flag -full_tsan in addition to the existing -tsan build flag. -full_tsan is equivalent to the current -tsan behavior, and -tsan is changed to set the ignore_noninstrumented_modules flag to true. ignore_noninstrumented_modules causes TSAN to ignore any modules that are not TSAN-instrumented. This is necessary to get TSAN to play nicely with Java, since Java is not TSAN-instrumented (see https://wiki.openjdk.java.net/display/tsan/Main and JDK-8208520). While this might decrease the number of issues surfaced by TSAN, it drastically decreases the amount of noise produced by TSAN because the JVM is not running TSAN-instrumented code. Without this flag set to true, almost every single backend test fails with the error: WARNING: ThreadSanitizer: data race (pid=12939) Write of size 1 at 0x7fcbe379c4c6 by thread T31: #0 strncpy /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (unifiedbetests+0x1b2a4ad) #1 (libjvm.so+0x90e706) This patch fixes various TSAN bugs (e.g. data races) reported while running backend tests and E2E against a TSAN build (it does not make Impala completely TSAN-clean). This patch makes the following changes: * Fixes several bugs involving issues with updating shared variables between threads * Fixes a few race conditions in test classes * Where possible, existing locks are used to fix any data races; in cases where the locking logic is non-trivial, atomics are used * There are a few places where variables are marked as 'volatile' presumably for synchronization purposes; TSAN flags these 'volatile' variables as unsafe, and according to https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md#Rconc-volatile using 'volatile' for synchronization is dangerous; in these cases, the 'volatile' variables are changed to 'atomic' variables * This patch adds a suppression file (bin/tsan-suppresions.txt) similar to the UBSAN suppresion file (bin/ubsan-suppresions.txt) Testing: * Ran exhaustive tests * Ran core tests w/ ASAN build * Manually re-ran backend tests against a TSAN build and made sure the reported errors are gone Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/common/init.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node.cc M be/src/rpc/thrift-thread.cc M be/src/rpc/thrift-thread.h M be/src/runtime/bufferpool/reservation-tracker.cc M be/src/runtime/bufferpool/reservation-tracker.h M be/src/runtime/bufferpool/system-allocator.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-test.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-stress.h M be/src/runtime/io/scan-range.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-test.cc M be/src/util/runtime-profile-test.cc M be/src/util/stopwatch.h M bin/run-backend-tests.sh A bin/tsan-suppressions.txt M buildall.sh M tests/common/environ.py M tests/webserver/test_web_pages.py 29 files changed, 240 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/15116/4 -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15116/3/be/src/runtime/bufferpool/reservation-tracker.cc File be/src/runtime/bufferpool/reservation-tracker.cc: http://gerrit.cloudera.org:8080/#/c/15116/3/be/src/runtime/bufferpool/reservation-tracker.cc@328 PS3, Line 328: if (tracker->reservation_.Load() + bytes > tracker->reservation_limit_.Load()) return false; > line too long (96 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 01:32:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/15116/2/be/src/runtime/bufferpool/reservation-tracker.h File be/src/runtime/bufferpool/reservation-tracker.h: http://gerrit.cloudera.org:8080/#/c/15116/2/be/src/runtime/bufferpool/reservation-tracker.h@322 PS2, Line 322: > We should probably change this too for consistency (I think the same consid Done http://gerrit.cloudera.org:8080/#/c/15116/1/be/src/runtime/bufferpool/system-allocator.cc File be/src/runtime/bufferpool/system-allocator.cc: http://gerrit.cloudera.org:8080/#/c/15116/1/be/src/runtime/bufferpool/system-allocator.cc@124 PS1, Line 124: #if defined(ADDRESS_SANITIZER) || defined(THREAD_SANITIZER) > I assigned IMPALA-6259 to you since you're fixing it whoops, I take that back. looks like it was fixed for ASAN in LLVM 5.0.0 but not fixed for TSAN until LLVM 6.0.0 - https://reviews.llvm.org/D35690 and https://github.com/llvm/llvm-project/commit/132689243e03b4e817aaac82d352a029216e9fee I've updated IMPALA-6259 and the code comments -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 01:30:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15116 ) Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/15116/3/be/src/common/init.cc File be/src/common/init.cc: http://gerrit.cloudera.org:8080/#/c/15116/3/be/src/common/init.cc@423 PS3, Line 423: // https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual#non-instrumented-code), line too long (95 > 90) http://gerrit.cloudera.org:8080/#/c/15116/3/be/src/runtime/bufferpool/reservation-tracker.cc File be/src/runtime/bufferpool/reservation-tracker.cc: http://gerrit.cloudera.org:8080/#/c/15116/3/be/src/runtime/bufferpool/reservation-tracker.cc@328 PS3, Line 328: if (tracker->reservation_.Load() + bytes > tracker->reservation_limit_.Load()) return false; line too long (96 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 05 Feb 2020 01:28:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5904: Add full tsan option and fix several TSAN bugs
Hello Adar Dembo, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15116 to look at the new patch set (#3). Change subject: IMPALA-5904: Add full_tsan option and fix several TSAN bugs .. IMPALA-5904: Add full_tsan option and fix several TSAN bugs This patch adds an additional build flag -full_tsan in addition to the existing -tsan build flag. -full_tsan is equivalent to the current -tsan behavior, and -tsan is changed to set the ignore_noninstrumented_modules flag to true. ignore_noninstrumented_modules causes TSAN to ignore any modules that are not TSAN-instrumented. This is necessary to get TSAN to play nicely with Java, since Java is not TSAN-instrumented (see https://wiki.openjdk.java.net/display/tsan/Main and JDK-8208520). While this might decrease the number of issues surfaced by TSAN, it drastically decreases the amount of noise produced by TSAN because the JVM is not running TSAN-instrumented code. Without this flag set to true, almost every single backend test fails with the error: WARNING: ThreadSanitizer: data race (pid=12939) Write of size 1 at 0x7fcbe379c4c6 by thread T31: #0 strncpy /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (unifiedbetests+0x1b2a4ad) #1 (libjvm.so+0x90e706) This patch fixes various TSAN bugs (e.g. data races) reported while running backend tests and E2E against a TSAN build (it does not make Impala completely TSAN-clean). This patch makes the following changes: * Fixes several bugs involving issues with updating shared variables between threads * Fixes a few race conditions in test classes * Where possible, existing locks are used to fix any data races; in cases where the locking logic is non-trivial, atomics are used * There are a few places where variables are marked as 'volatile' presumably for synchronization purposes; TSAN flags these 'volatile' variables as unsafe, and according to https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md#Rconc-volatile using 'volatile' for synchronization is dangerous; in these cases, the 'volatile' variables are changed to 'atomic' variables * This patch adds a suppression file (bin/tsan-suppresions.txt) similar to the UBSAN suppresion file (bin/ubsan-suppresions.txt) Testing: * Ran exhaustive tests * Ran core tests w/ ASAN build * Manually re-ran backend tests against a TSAN build and made sure the reported errors are gone Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/common/init.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node.cc M be/src/rpc/thrift-thread.cc M be/src/rpc/thrift-thread.h M be/src/runtime/bufferpool/reservation-tracker.cc M be/src/runtime/bufferpool/reservation-tracker.h M be/src/runtime/bufferpool/system-allocator.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-test.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-stress.h M be/src/runtime/io/scan-range.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-test.cc M be/src/util/runtime-profile-test.cc M be/src/util/stopwatch.h M bin/run-backend-tests.sh A bin/tsan-suppressions.txt M buildall.sh M tests/common/environ.py M tests/webserver/test_web_pages.py 29 files changed, 238 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/15116/3 -- To view, visit http://gerrit.cloudera.org:8080/15116 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3d7ef5c228afd5882e145e6f53885b355d6c25a0 Gerrit-Change-Number: 15116 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9358: Query slowdown with inline views and hundreds of columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15157 ) Change subject: IMPALA-9358: Query slowdown with inline views and hundreds of columns .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5124/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77423d9c10e1edbb505cb210b5c072281b5d7cfc Gerrit-Change-Number: 15157 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 05 Feb 2020 01:26:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15154 ) Change subject: IMPALA-8712: Make ExecQueryFInstances async .. Patch Set 2: (14 comments) http://gerrit.cloudera.org:8080/#/c/15154/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15154/2//COMMIT_MSG@23 PS2, Line 23: Removing this thread pool has potential performance implications, as : it means that the Exec() parameters are serialized in serialize rather : than in parallel (with the level of parallelism determined by the size : of the thread pool, which was configurable by an Advanced flag and : defaulted to 12). do we have a runtime counter to measure this overhead? http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h File be/src/runtime/coordinator-backend-state.h: http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h@82 PS2, Line 82: Returns True nit: "Returns true" http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h@96 PS2, Line 96: rpc_complete_barrier nit: needs to be updated http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h@102 PS2, Line 102: Exec nit: should the docs of this method be updated to indicate that this method is now asynchronous? http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h@108 PS2, Line 108: WaitOnExec might want to mention some more docs about thread safety - e.g. is a client allowed to call Exec from one thread and then call WaitOnExec from another thread? http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h@369 PS2, Line 369: Only needs to be protected by : /// 'lock_' until Exec() has finished, after which it is constant. same as below http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.h@375 PS2, Line 375: Only needs : /// to be protected by 'lock_' until Exec() has finished, after which it is constant. i believe this isn't thread safe. even if it is no longer set after Exec() has finished, the reader still needs to acquire the lock - the reader thread could have cached the exec_done_ in its CPU cache, which means it won't see an updates from the writer http://igoro.com/archive/volatile-keyword-in-c-memory-model-explained/ has some details, specifically the part about thread caches these could be made atomic if we don't want to acquire the lock each time they are read. http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc@a452 PS2, Line 452: : : : : : : : : : : : : looks like dead code, but are any of the TODOs still relevant? http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc@173 PS2, Line 173: exec_done_cv_.NotifyAll(); i believe we generally try to notify any waiting threads after the lock has been released. that way the waiting threads can acquire the lock immediately, rather than waking up, and then going back to sleep because the lock is still acquired - IMPALA-6125 also will there be multiple threads waiting on this CV? http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc@193 PS2, Line 193: exec_done_cv_.NotifyAll(); nit: won't this be called twice, once in SetExecError and again when method exits? http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc@228 PS2, Line 228: exec_done_ = true; is this thread safe? this gets set without holding onto the lock, and its possible for Coordinator::UpdateBackendExecStatus or UpdateFilter to call WaitOnExec from a different thread http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc@236 PS2, Line 236: SetExecError same issue as above, this sets exec_done_ without holding the lock http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator-backend-state.cc@853 PS2, Line 853: DCHECK(exec_done_) should be run after acquiring the lock, or made atomic http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/15154/2/be/src/runtime/coordinator.cc@427 PS2, Line 427: if (exec_rpcs_complete_.Load()) return; is the idea here that this method should only ever be executed once? whats the thread safety guarantee on this method suppose to be? -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit
[Impala-ASF-CR] IMPALA-9358: Query slowdown with inline views and hundreds of columns
Kurt Deschler has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15157 Change subject: IMPALA-9358: Query slowdown with inline views and hundreds of columns .. IMPALA-9358: Query slowdown with inline views and hundreds of columns IMPALA-8386 introduced an expensive precondition check using the function ExprSubstitutionMap.checkComposedFrom(). This check has significant performance impact on statements that contain inline views with hundreds of columns. Most of the cost is in the get() calls used to find expressions in the local substitution map. The fix is to add a get_hint() call that uses the current loop index as a starting point to search for expressions. This leverages the fact that expressions have identical positions in both substitution maps in most common cases. A more generic approach would be to accelerate expression equality search using hash functions but that would be a much riskier fix and Impala currently lacks the infrasturucture to so. Testing: Performance testing with a query with 1000 expressions of the following form: with a as (select c1 c1, c1 c2, c1 c3, ... from t) select c1, c2, c3, ... from a; Change-Id: I77423d9c10e1edbb505cb210b5c072281b5d7cfc --- M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java 1 file changed, 17 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/15157/1 -- To view, visit http://gerrit.cloudera.org:8080/15157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I77423d9c10e1edbb505cb210b5c072281b5d7cfc Gerrit-Change-Number: 15157 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5280/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 23:27:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 23:27:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. Patch Set 12: Code-Review+2 Carry on Csaba's +2 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 23:26:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9335 (part 1): Rebase copied Kudu source code
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15143 ) Change subject: IMPALA-9335 (part 1): Rebase copied Kudu source code .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0190f15b96563f5edc40065fa4690c467c50f83 Gerrit-Change-Number: 15143 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 04 Feb 2020 23:03:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9335 (part 1): Rebase copied Kudu source code
Thomas Tauber-Marshall has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/15143 )
Change subject: IMPALA-9335 (part 1): Rebase copied Kudu source code
..
IMPALA-9335 (part 1): Rebase copied Kudu source code
Removed be/src/kudu/{util|security|rpc} and copied back over from Kudu
commit: fc4ab691a502067bc4d5bdff30507cac7feb7cfe
Change-Id: Ia0190f15b96563f5edc40065fa4690c467c50f83
Reviewed-on: http://gerrit.cloudera.org:8080/15143
Reviewed-by: Joe McDonnell
Tested-by: Thomas Tauber-Marshall
---
M be/src/kudu/rpc/CMakeLists.txt
M be/src/kudu/rpc/acceptor_pool.cc
M be/src/kudu/rpc/acceptor_pool.h
M be/src/kudu/rpc/blocking_ops.h
M be/src/kudu/rpc/client_negotiation.cc
M be/src/kudu/rpc/connection.cc
M be/src/kudu/rpc/connection.h
M be/src/kudu/rpc/constants.h
M be/src/kudu/rpc/inbound_call.cc
M be/src/kudu/rpc/inbound_call.h
M be/src/kudu/rpc/messenger.cc
M be/src/kudu/rpc/messenger.h
M be/src/kudu/rpc/mt-rpc-test.cc
M be/src/kudu/rpc/negotiation-test.cc
M be/src/kudu/rpc/negotiation.h
M be/src/kudu/rpc/outbound_call.cc
M be/src/kudu/rpc/outbound_call.h
M be/src/kudu/rpc/protoc-gen-krpc.cc
M be/src/kudu/rpc/proxy.h
M be/src/kudu/rpc/reactor.cc
M be/src/kudu/rpc/reactor.h
M be/src/kudu/rpc/remote_method.h
M be/src/kudu/rpc/response_callback.h
M be/src/kudu/rpc/result_tracker.cc
M be/src/kudu/rpc/retriable_rpc.h
M be/src/kudu/rpc/rpc-bench.cc
M be/src/kudu/rpc/rpc-test-base.h
M be/src/kudu/rpc/rpc-test.cc
M be/src/kudu/rpc/rpc.cc
M be/src/kudu/rpc/rpc.h
M be/src/kudu/rpc/rpc_context.cc
M be/src/kudu/rpc/rpc_context.h
M be/src/kudu/rpc/rpc_controller.cc
M be/src/kudu/rpc/rpc_controller.h
M be/src/kudu/rpc/rpc_header.proto
M be/src/kudu/rpc/rpc_introspection.proto
M be/src/kudu/rpc/rpc_service.h
M be/src/kudu/rpc/rpc_sidecar.h
M be/src/kudu/rpc/rpc_stub-test.cc
A be/src/kudu/rpc/rpc_verification_util.cc
A be/src/kudu/rpc/rpc_verification_util.h
M be/src/kudu/rpc/rpcz_store.cc
M be/src/kudu/rpc/sasl_common.cc
M be/src/kudu/rpc/sasl_common.h
M be/src/kudu/rpc/sasl_helper.h
M be/src/kudu/rpc/serialization.cc
M be/src/kudu/rpc/serialization.h
M be/src/kudu/rpc/server_negotiation.cc
M be/src/kudu/rpc/service_if.cc
M be/src/kudu/rpc/service_if.h
M be/src/kudu/rpc/service_pool.cc
M be/src/kudu/rpc/service_pool.h
M be/src/kudu/rpc/service_queue.h
M be/src/kudu/rpc/transfer.cc
M be/src/kudu/rpc/transfer.h
M be/src/kudu/security/CMakeLists.txt
M be/src/kudu/util/CMakeLists.txt
M be/src/kudu/util/async_util-test.cc
M be/src/kudu/util/async_util.h
M be/src/kudu/util/bitmap-test.cc
M be/src/kudu/util/bitmap.cc
M be/src/kudu/util/bitmap.h
A be/src/kudu/util/bitset-test.cc
A be/src/kudu/util/bitset.h
A be/src/kudu/util/block_bloom_filter-test.cc
A be/src/kudu/util/block_bloom_filter.cc
A be/src/kudu/util/block_bloom_filter.h
A be/src/kudu/util/block_bloom_filter_avx2.cc
R be/src/kudu/util/block_cache_metrics.cc
A be/src/kudu/util/block_cache_metrics.h
M be/src/kudu/util/bloom_filter.h
M be/src/kudu/util/cache-bench.cc
M be/src/kudu/util/cache-test.cc
M be/src/kudu/util/cache.cc
M be/src/kudu/util/cache.h
M be/src/kudu/util/cache_metrics.h
A be/src/kudu/util/char_util-test.cc
A be/src/kudu/util/char_util.cc
A be/src/kudu/util/char_util.h
A be/src/kudu/util/cloud/instance_detector-test.cc
A be/src/kudu/util/cloud/instance_detector.cc
A be/src/kudu/util/cloud/instance_detector.h
A be/src/kudu/util/cloud/instance_metadata.cc
A be/src/kudu/util/cloud/instance_metadata.h
M be/src/kudu/util/compression/compression-test.cc
M be/src/kudu/util/compression/compression_codec.cc
M be/src/kudu/util/countdown_latch-test.cc
M be/src/kudu/util/countdown_latch.h
A be/src/kudu/util/curl_util-test.cc
M be/src/kudu/util/curl_util.cc
M be/src/kudu/util/curl_util.h
M be/src/kudu/util/debug/trace_event_impl.cc
M be/src/kudu/util/debug/trace_logging.h
M be/src/kudu/util/easy_json.cc
M be/src/kudu/util/env-test.cc
M be/src/kudu/util/env.cc
M be/src/kudu/util/env.h
M be/src/kudu/util/env_posix.cc
M be/src/kudu/util/env_util.cc
M be/src/kudu/util/env_util.h
M be/src/kudu/util/faststring-test.cc
M be/src/kudu/util/faststring.cc
M be/src/kudu/util/faststring.h
M be/src/kudu/util/fault_injection.cc
M be/src/kudu/util/fault_injection.h
M be/src/kudu/util/file_cache-stress-test.cc
M be/src/kudu/util/file_cache-test.cc
M be/src/kudu/util/file_cache.cc
M be/src/kudu/util/file_cache.h
A be/src/kudu/util/file_cache_metrics.cc
A be/src/kudu/util/file_cache_metrics.h
M be/src/kudu/util/flag_tags-test.cc
M be/src/kudu/util/flag_tags.h
M be/src/kudu/util/flags-test.cc
M be/src/kudu/util/flags.cc
M be/src/kudu/util/flags.h
A be/src/kudu/util/hash.proto
M be/src/kudu/util/hash_util-test.cc
M be/src/kudu/util/hash_util.h
M be/src/kudu/util/hdr_histogram-test.cc
M be/src/kudu/util/hdr_histogram.cc
M be/src/kudu/util/hdr_histogram.h
M be/src/kudu/util/high_water_mark.h
M be/src/kudu/util/init.cc
M be/src/kudu/util/init.h
M be/src/kudu/util/int128_util.h
M
[Impala-ASF-CR] IMPALA-9335 (part 2): Fix rebased KRPC to compile
Thomas Tauber-Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15144 ) Change subject: IMPALA-9335 (part 2): Fix rebased KRPC to compile .. IMPALA-9335 (part 2): Fix rebased KRPC to compile This patch applies various fixes to Impala and to the copied Kudu source code in be/src/kudu/* to allow everything to compile. Some highlights of the changes made: - Various Kudu files were removed from compilation due to issues like relying on libraries that Impala does not provide. The linking of some executable is also changed for similar reasons. - The Kudu Cache implementation changed to support unique_ptr, allowing us to remove various uses of MakeScopeExitTrigger. - Some flags that have a DEFINE in both Kudu and Impala are modified to change one of the DEFINEs to a DECLARE. This patch was in part based on the patches that were applied the last time we rebased the Kudu code in IMPALA-7006, and I ensured that all changes from those commits that are still relevant were included here. I also went through all commits that have been applied to the be/src/kudu directory since the last rebase and ensured that all relevant changes from those are included here. Testing: - Passed an exhaustive DEBUG build and a core ASAN build. Change-Id: I1eb4caf927c729109426fb50a28b5e15d6ac46cb Reviewed-on: http://gerrit.cloudera.org:8080/15144 Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell --- M be/src/kudu/rpc/CMakeLists.txt M be/src/kudu/rpc/transfer.cc M be/src/kudu/security/CMakeLists.txt M be/src/kudu/security/test/mini_kdc.cc M be/src/kudu/util/CMakeLists.txt M be/src/kudu/util/flags.cc A be/src/kudu/util/kudu_export.h M be/src/kudu/util/logging.cc M be/src/kudu/util/logging.h M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/runtime/io/data-cache.cc M be/src/runtime/io/data-cache.h M be/src/util/webserver.cc M bin/rat_exclude_files.txt M cmake_modules/kudu_cmake_fns.txt M tests/custom_cluster/test_restart_services.py 17 files changed, 177 insertions(+), 86 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15144 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1eb4caf927c729109426fb50a28b5e15d6ac46cb Gerrit-Change-Number: 15144 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15149 )
Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu
table in Impala
..
Patch Set 5:
(6 comments)
Left some comments and suggestions. Rest looks great to me! Thanks!
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml
File docs/topics/impala_create_table.xml:
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml@259
PS5, Line 259: [TBLPROPERTIES
('key1'='value1',
'key2'='value2', ...)] |
[('external.table.purge'='true',
'key1'='value1',...)]
Since this section is specifically talking about external kudu tables, we can
just say [TBLPROPERTIES ('external.table.purge'='true','key1'='value1',...)]
I previously suggested having a both the TBLPROPERTIES with and without the
external.table.purge property in there since I thought we will have one syntax
for both managed and external tables. But this works for me too.
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml@264
PS5, Line 264: Integrate HMS-Kudu services.
I am not sure what this means? Does this mean HMS-Kudu integration feature of
Kudu? If yes, it is not really necessary to have that enabled for creation of
these external kudu tables.
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml@274
PS5, Line 274: alter table rename
: ...
can we just say renaming the table changes the name of the table in HMS and
Kudu?
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml@275
PS5, Line 275: Kudo
spell-check
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml@275
PS5, Line 275: hrough HMS-Kudo integration, Kudu
: synchronizes changes to the actual data and metadat
not sure if need this line. Its unclear to me why we refer to HMS-Kudu
integration here.
http://gerrit.cloudera.org:8080/#/c/15149/5/docs/topics/impala_create_table.xml@593
PS5, Line 593:
do we need the word "can" here?
--
To view, visit http://gerrit.cloudera.org:8080/15149
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e
Gerrit-Change-Number: 15149
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 04 Feb 2020 22:21:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Anonymous Coward (498) has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java File fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java: http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@125 PS7, Line 125: if (db_ == null) return path; if uri is not null, should you add it into path before return? -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 7 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Feb 2020 22:17:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15154 ) Change subject: IMPALA-8712: Make ExecQueryFInstances async .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5123/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 04 Feb 2020 21:07:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15149 ) Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala .. Patch Set 5: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/538/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/15149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e Gerrit-Change-Number: 15149 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 20:59:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15149 ) Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/537/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/15149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e Gerrit-Change-Number: 15149 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 20:58:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15149 ) Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala .. Patch Set 5: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/538/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/15149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e Gerrit-Change-Number: 15149 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 20:29:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15149 ) Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/537/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/15149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e Gerrit-Change-Number: 15149 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 20:29:25 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5122/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Feb 2020 20:29:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/15068 )
Change subject: IMPALA-9242: Filter privileges before returning them to Sentry
..
Patch Set 7:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/Principal.java
File fe/src/main/java/org/apache/impala/catalog/Principal.java:
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/Principal.java@104
PS7, Line 104:* Returns all privilege names for this principal, or an empty
set if no privileges are
The doc comment is the same as for getPrivilegeNames() and does not mention the
filter.
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java
File fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java:
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@34
PS7, Line 34: for
Isn't this 'for' superfluous?
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@37
PS7, Line 37: public class PrincipalPrivilegeTree {
The implementation seems correct to me, though I haven't tried it yet, only
read it.
I think it is a good idea to add unit tests.
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@134
PS7, Line 134: s
Nit: object.
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@150
PS7, Line 150: void add(T value, List path, int depth) {
I think 'depth' should be documented.
Also it seems to me that depth is always 0 when the method is called from
outside and any other depth value only appears during the recursion. I think it
would be clearer to make a public wrapper method that calls this with depth=0
and make this method private.
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@167
PS7, Line 167: void remove(T value, List path, int depth) {
See the comment about 'depth' for the 'add' method.
http://gerrit.cloudera.org:8080/#/c/15068/7/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@200
PS7, Line 200: void getAllMatchingValues(List results, List
path, int depth) {
See the comment about 'depth' for the 'add' method.
--
To view, visit http://gerrit.cloudera.org:8080/15068
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e
Gerrit-Change-Number: 15068
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 04 Feb 2020 18:36:06 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15154 ) Change subject: IMPALA-8712: Make ExecQueryFInstances async .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/15154/1/tests/custom_cluster/test_rpc_exception.py File tests/custom_cluster/test_rpc_exception.py: http://gerrit.cloudera.org:8080/#/c/15154/1/tests/custom_cluster/test_rpc_exception.py@27 PS1, Line 27: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/15154/1/tests/custom_cluster/test_rpc_exception.py@33 PS1, Line 33: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/15154/1/tests/custom_cluster/test_rpc_exception.py@147 PS1, Line 147: a > flake8: F841 local variable 'result' is assigned to but never used Done -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 04 Feb 2020 18:15:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8712: Make ExecQueryFInstances async
Hello Sahil Takiar, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15154 to look at the new patch set (#2). Change subject: IMPALA-8712: Make ExecQueryFInstances async .. IMPALA-8712: Make ExecQueryFInstances async This patch refactors the ExecQueryFInstances rpc to be asychronous. Previously, Impala would issue all the Exec()s, wait for all of them to complete, and then check if any of them resulted in an error. We now stop issuing Exec()s and cancel any that are still in flight as soon as an error occurs. It also performs some cleanup around the thread safety of Coordinator::BackendState, including adding comments and DCHECKS. === Exec RPC Thread Pool === This patch also removes the 'exec_rpc_thread_pool_' from ExecEnv. This thread pool was used to partially simulate async Exec() prior to the switch to KRPC, which provides built-in async rpc capabilities. Removing this thread pool has potential performance implications, as it means that the Exec() parameters are serialized in serialize rather than in parallel (with the level of parallelism determined by the size of the thread pool, which was configurable by an Advanced flag and defaulted to 12). To ensure we don't regress query startup times, I did some performance testing. All tests were done on a 10 node cluster. The baseline used for the tests did not include IMPALA-9181, a perf optimization for query startup done to facilitate this work. I ran TPCH 100 at concurrency levels of 1, 4, and 8 and extracted the query startup times from the profiles. For each concurrency level, the average regression in query startup time was < 2ms. Because query e2e running time was much longer than this, there was no noticable change in total query time. I also ran a 'worst case scenario' with a table with 10,000 pertitions to create a very large Exec() payload to serialize (~1.21MB vs. ~10KB-30KB for TPCH 100). Again, change in query startup time was neglible. TODO: once IMPALA-9335 (krpc rebase) goes in, the change in be/src/kudu/rpc/connection.cc can be removed from this patch. Testing: - Added a e2e test that verifies that a query where an Exec() fails doesn't wait for all Exec()s to complete before cancelling and returning the error to the client. Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 --- M be/src/common/global-flags.cc M be/src/kudu/rpc/connection.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/query-state.cc M be/src/util/counting-barrier.h M tests/custom_cluster/test_rpc_exception.py M tests/failure/test_failpoints.py 12 files changed, 387 insertions(+), 203 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15154/2 -- To view, visit http://gerrit.cloudera.org:8080/15154 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I33ec96e5885af094c294cd3a76c242995263ba32 Gerrit-Change-Number: 15154 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Hello Vihang Karajgaonkar, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15149 to look at the new patch set (#5). Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala .. IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala Summary of changes: - Changed title of "Kudu tables:" to "Managed Kudu tables:". - Added syntax "External Kudu tables" to show alternative create table syntax. - Described alternative syntax and differences between resulting tables. - In Kudu considerations, added example of creating synchronized external Kudu table. - Covered synchronized tables vs managed tables and HMS translation of external tables. Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e --- M docs/topics/impala_create_table.xml 1 file changed, 67 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/15149/5 -- To view, visit http://gerrit.cloudera.org:8080/15149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e Gerrit-Change-Number: 15149 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala
Hello Vihang Karajgaonkar, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15149 to look at the new patch set (#4). Change subject: IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala .. IMPALA-9337 [DOCS] Document new way to create external Kudu table in Impala Summary of changes: - Changed title of "Kudu tables:" to "Managed Kudu tables:". - Added syntax "External Kudu tables" to show alternative create table syntax. - Described alternative syntax and differences between resulting tables. - In Kudu considerations, added example of creating synchronized external Kudu table. - Covered synchronized tables vs managed tables and HMS translation of external tables. Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e --- M docs/topics/impala_create_table.xml 1 file changed, 67 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/15149/4 -- To view, visit http://gerrit.cloudera.org:8080/15149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic07380fd53898dd21fbb5dacb4d9f7a84f160d4e Gerrit-Change-Number: 15149 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9335 (part 1): Rebase copied Kudu source code
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15143 ) Change subject: IMPALA-9335 (part 1): Rebase copied Kudu source code .. Patch Set 1: Code-Review+2 Looks good. (Checked out kudu at that commit and compared the files in be/src/kudu to the corresponding files in kudu.) -- To view, visit http://gerrit.cloudera.org:8080/15143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0190f15b96563f5edc40065fa4690c467c50f83 Gerrit-Change-Number: 15143 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Tue, 04 Feb 2020 17:31:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4224: execute separate join builds fragments
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/14859 )
Change subject: IMPALA-4224: execute separate join builds fragments
..
Patch Set 35:
(6 comments)
I focused on the backend. It looks good to me, although I still need to go over
the resource management part to really understand all moving parts.
http://gerrit.cloudera.org:8080/#/c/14859/35//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14859/35//COMMIT_MSG@38
PS35, Line 38:accordingly.
Something lost between edits.
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/blocking-join-node.h
File be/src/exec/blocking-join-node.h:
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/blocking-join-node.h@63
PS35, Line 63: separate build fragment
nit: maybe "separate but co-located build fragment"? I feel it's worth to
emphasize that it runs in the same process but in a different thread.
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/blocking-join-node.cc
File be/src/exec/blocking-join-node.cc:
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/blocking-join-node.cc@196
PS35, Line 196: if (!open_called_) {
nit: maybe an early return would make it a bit more readable
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/join-builder.h
File be/src/exec/join-builder.h:
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/join-builder.h@54
PS35, Line 54: class JoinBuilder : public DataSink {
Maybe you could add a timeline, or sequence diagram-like thingy about the
interaction between the BlockingJoinNode and the JoinBuilder.
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/join-builder.cc
File be/src/exec/join-builder.cc:
http://gerrit.cloudera.org:8080/#/c/14859/35/be/src/exec/join-builder.cc@90
PS35, Line 90: HandoffToProbe
nit: HandoffToProbes (plural)?
http://gerrit.cloudera.org:8080/#/c/14859/32/common/thrift/PlanNodes.thrift
File common/thrift/PlanNodes.thrift:
http://gerrit.cloudera.org:8080/#/c/14859/32/common/thrift/PlanNodes.thrift@617
PS32, Line 617: 28
> I did this mainly to reduce the size of the diff. I think I'd prefer to kee
No, I also don't feel strongly about it.
--
To view, visit http://gerrit.cloudera.org:8080/14859
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4403c8e62d9c13854e7830602ee613f8efc80c58
Gerrit-Change-Number: 14859
Gerrit-PatchSet: 35
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 04 Feb 2020 17:27:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9335 (part 2): Fix rebased KRPC to compile
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15144 ) Change subject: IMPALA-9335 (part 2): Fix rebased KRPC to compile .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15144 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eb4caf927c729109426fb50a28b5e15d6ac46cb Gerrit-Change-Number: 15144 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 04 Feb 2020 17:17:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9335 (part 2): Fix rebased KRPC to compile
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15144 ) Change subject: IMPALA-9335 (part 2): Fix rebased KRPC to compile .. Patch Set 2: (1 comment) I think I'm ready to +2 unless someone else wants to review. http://gerrit.cloudera.org:8080/#/c/15144/2/be/src/runtime/io/data-cache.cc File be/src/runtime/io/data-cache.cc: http://gerrit.cloudera.org:8080/#/c/15144/2/be/src/runtime/io/data-cache.cc@558 PS2, Line 558: handle.get() > I know. I prefer doing it this way because it makes it more obvious to read UniqueHandle is a bit weird, because it is a unique_ptr without exposing that fact in the type name. I looked through the Kudu codebase, and they always wrap the UniqueHandle in something else. I don't feel too strongly, so let's leave it this way. -- To view, visit http://gerrit.cloudera.org:8080/15144 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eb4caf927c729109426fb50a28b5e15d6ac46cb Gerrit-Change-Number: 15144 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 04 Feb 2020 17:13:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5120/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Feb 2020 16:35:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5121/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 7 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Feb 2020 16:35:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5119/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Feb 2020 16:33:21 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15105 )
Change subject: WIP: Asynchronous code generation
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/codegen/llvm-codegen.cc
File be/src/codegen/llvm-codegen.cc:
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/codegen/llvm-codegen.cc@1155
PS3, Line 1155: for (const std::pair&
fn_pair : fns_to_jit_compile_) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/exec/union-node.h
File be/src/exec/union-node.h:
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/exec/union-node.h@113
PS3, Line 113: std::vector>
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/exec/union-node.cc@239
PS3, Line 239: UnionMaterializeBatchFn fn =
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/exprs/scalar-expr.h
File be/src/exprs/scalar-expr.h:
http://gerrit.cloudera.org:8080/#/c/15105/3/be/src/exprs/scalar-expr.h@402
PS3, Line 402: /// calling the functions.
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/15105
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b
Gerrit-Change-Number: 15105
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 04 Feb 2020 16:27:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/codegen/codegen-fn-ptr.h File be/src/codegen/codegen-fn-ptr.h: http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/codegen/codegen-fn-ptr.h@30 PS1, Line 30: rn ptr_.load(mem_order_); > Yeah I think we should either just hardcode the memory order as a static co Yes, probably the best solution is a constexpr memory order, I just wasn't sure in the beginning and thought we may want to test two different versions in the same build. What do you think would be the best memory order? Do you think relaxed should be enough? http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/codegen/codegen-fn-ptr.h@45 PS1, Line 45: inline CodegenFnPtrBase::~CodegenFnPtrBase() = default; > Would it make sense to make the function pointer type a template parameter I don't think we can do it type-safely because we store pointers to instances in containers where the instances have different underlying function pointers, for example in LlvmCodegen. But I have implemented a version where there is a general base class and the templated classes with concrete function types are subclasses of the base class. This way the function type is included in the type name of the atomic function pointers (good for code clarity) and the reinterpret casts can be collected to one place in the majority of the cases (however not all, for example 'codegend_fn_map_' in hdfs-scan-node.h still has to use void* as the function type because there are different kinds of functions in the collection). http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/exec/hdfs-sequence-scanner.cc File be/src/exec/hdfs-sequence-scanner.cc: http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/exec/hdfs-sequence-scanner.cc@310 PS1, Line 310: int tuples_returned = WriteAlignedTuplesCodegenOrInterpret(row_batch->tuple_data_pool(), > Maybe remove this comment? Not sure it is that informative. Done http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/exec/hdfs-text-scanner.cc File be/src/exec/hdfs-text-scanner.cc: http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/exec/hdfs-text-scanner.cc@869 PS1, Line 869: int tuples_returned = WriteAlignedTuplesCodegenOrInterpret(pool, row, fields, > Maybe remove this comment? Not sure it is that informative. Done http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/runtime/runtime-state.h File be/src/runtime/runtime-state.h: http://gerrit.cloudera.org:8080/#/c/15105/1/be/src/runtime/runtime-state.h@200 PS1, Line 200: return !CodegenDisabledByQueryOption() && !CodegenDisabledByHint(); > nit: I think the member should be is_interpretable_ and the accessor method Done -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Feb 2020 16:26:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. WIP: Asynchronous code generation This commit introduces optional asynchronous code generation. Asynchronous code generation means that instead of waiting for codegen to finish, the query starts in interpreted mode while codegen is done on another thread. All the function pointers that point to codegen'd functions are changed to be atomic, wrapped in a CodegenFnPtr. These are initialised to nullptr and as long as they are nullptr, the corresponding interpreted functions are used (as before). When code generation is ready, the funtion pointers are set by the codegen thread. No synchronisation is needed as the function pointers are atomic and it is not a problem if, at a given moment, only a subset of the codegen'd function pointers are set and the rest are interpreted. Asynchronous code generation can be turned on using the ASYNC_CODEGEN boolean query option. TODO: The default should be synchronous codegen for now. TODO: Testing. TODO: Benchmarks. Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks/hash-benchmark.cc A be/src/codegen/codegen-fn-ptr.h M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exprs/expr-codegen-test.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-expr.inline.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift 45 files changed, 459 insertions(+), 226 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15105/3 -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15068/6/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java File fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java: http://gerrit.cloudera.org:8080/#/c/15068/6/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@143 PS6, Line 143: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Feb 2020 15:50:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Hello Anonymous Coward (498), Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15068 to look at the new patch set (#7). Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. IMPALA-9242: Filter privileges before returning them to Sentry This change implements the new FilteredPrivilegeCache, which adds functions for filtering privileges based on the authorizable and for returning Privileges directly instead of their String form. The filtering is based on server + db + table (or just server in case of URI privileges) to filter out the bulk of unrelated privileges. Efficient filtering is done by a new class PrincipalPrivilegeTree. It was tempting to reuse Sentry's TreePrivilegeCache, which has a very similar role, but it lacks a "remove" function that is needed to keep this index in sync with the CatalogObjectCache in Principal. I am also a bit concerned about the possible side effect of Sentry's interning of names in privileges - we try to avoid using String.intern() on massive amount of names in Impala. Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e --- M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Principal.java A fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java 3 files changed, 342 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15068/7 -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 7 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Hello Anonymous Coward (498), Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15068 to look at the new patch set (#6). Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. IMPALA-9242: Filter privileges before returning them to Sentry This change implements the new FilteredPrivilegeCache, which adds functions for filtering privileges based on the authorizable and for returning Privileges directly instead of their String form. The filtering is based on server + db + table (or just server in case of URI privileges) to filter out the bulk of unrelated privileges. Efficient filtering is done by a new class PrincipalPrivilegeTree. It was tempting to reuse Sentry's TreePrivilegeCache, which has a very similar role, but it lacks a "remove" function that is needed to keep this index in sync with the CatalogObjectCache in Principal. I am also a bit concerned about the possible side effect of Sentry's interning of names in privileges - we try to avoid using String.intern() on massive amount of names in Impala. Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e --- M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Principal.java A fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java 3 files changed, 342 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15068/6 -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15068 ) Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/15068/5/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java File fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java: http://gerrit.cloudera.org:8080/#/c/15068/5/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@48 PS5, Line 48: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15068/5/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@143 PS5, Line 143: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15068/5/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@145 PS5, Line 145: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15068/5/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java@212 PS5, Line 212: } line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 04 Feb 2020 15:48:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9242: Filter privileges before returning them to Sentry
Hello Anonymous Coward (498), Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15068 to look at the new patch set (#5). Change subject: IMPALA-9242: Filter privileges before returning them to Sentry .. IMPALA-9242: Filter privileges before returning them to Sentry This change implements the new FilteredPrivilegeCache, which adds functions for filtering privileges based on the authorizable and for returning Privileges directly instead of their String form. The filtering is based on server + db + table (or just server in case of URI privileges) to filter out the bulk of unrelated privileges. Efficient filtering is done by a new class PrincipalPrivilegeTree. It was tempting to reuse Sentry's TreePrivilegeCache, which has a very similar role, but it lacks a "remove" function that is needed to keep this index in sync with the CatalogObjectCache in Principal. I am also a bit concerned about the possible side effect of Sentry's interning of names in privileges - we try to avoid using String.intern() on massive amount of names in Impala. Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e --- M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Principal.java A fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilegeTree.java 3 files changed, 342 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15068/5 -- To view, visit http://gerrit.cloudera.org:8080/15068 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iecd4281368d1c9fe88cfe850ea725cd68895712e Gerrit-Change-Number: 15068 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5118/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 14:52:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9330: Fix column masking in nested tables + enable column masking by default
Hello Anurag Mantripragada, Fang-Yu Rao, Vihang Karajgaonkar, Kurt Deschler, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15108 to look at the new patch set (#12). Change subject: IMPALA-9330: Fix column masking in nested tables + enable column masking by default .. IMPALA-9330: Fix column masking in nested tables + enable column masking by default Column masking policies on primitive columns of a table which contains nested types (though they won't be masked) will cause query failures. To be specifit, if tableA(id int, int_array array) has a masking policy on column "id", all queries on "tableA" will fail, e.g. select id from tableA; select t.id, a.item from tableA t, t.int_array a; Column masking is implemented by wrapping the underlying table/view with a table masking view. However, as we don't support nested types in SelectList, the table masking view can't expose nested columns of the masked table, which causes collection refs not being resolved correctly. This patch fixes the issue by 2 steps: 1) Expose nested columns of the underlying table in the output Type of the table masking view (see InlineViewRef#createTupleDescriptor()). So nested Paths in the original query block can be resolved. 2) For such kind of Paths, resolved them again inside the table masking view. So they can point to the underlying table as what they mean (see Analyzer#resolvePathWithMasking()). TupleDescriptor of such kind of table masking view won't be materialized since the view is simple enough that its query plan is just a ScanNode of the underlying table. The whole query plan can be stitched as if the table is not masked. Note that one day when we support nested columns in SelectList, we may don't need these 2 hacks. This patch also adds some TRACE level loggings to improve debuggability, and enables column masking by default. Test changes in TestRanger.test_column_masking: - Add column masking policy on a table containing nested types. - Add queries on the masked tables. Some queries are borrowed from existing tests for nested types. Tests: - Run CORE tests. Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M tests/authorization/test_ranger.py 16 files changed, 538 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15108/12 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9330: Resolve unmasked complex types in masked tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Resolve unmasked complex types in masked tables .. Patch Set 11: (2 comments) http://gerrit.cloudera.org:8080/#/c/15108/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15108/10//COMMIT_MSG@7 PS10, Line 7: es > nit: I would prefer to somehow make the first line shorter and add "+ enabl I mention enable column masking by default at line36. Do you mean also mentioning it in the title? http://gerrit.cloudera.org:8080/#/c/15108/10/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/15108/10/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@a75 PS10, Line 75: > The member became unused. Oops, missing it... -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 14:06:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9330: Resolve unmasked complex types in masked tables
Hello Anurag Mantripragada, Fang-Yu Rao, Vihang Karajgaonkar, Kurt Deschler, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15108 to look at the new patch set (#11). Change subject: IMPALA-9330: Resolve unmasked complex types in masked tables .. IMPALA-9330: Resolve unmasked complex types in masked tables Column masking policies on primitive columns of a table which contains nested types (though they won't be masked) will cause query failures. To be specifit, if tableA(id int, int_array array) has a masking policy on column "id", all queries on "tableA" will fail, e.g. select id from tableA; select t.id, a.item from tableA t, t.int_array a; Column masking is implemented by wrapping the underlying table/view with a table masking view. However, as we don't support nested types in SelectList, the table masking view can't expose nested columns of the masked table, which causes collection refs not being resolved correctly. This patch fixes the issue by 2 steps: 1) Expose nested columns of the underlying table in the output Type of the table masking view (see InlineViewRef#createTupleDescriptor()). So nested Paths in the original query block can be resolved. 2) For such kind of Paths, resolved them again inside the table masking view. So they can point to the underlying table as what they mean (see Analyzer#resolvePathWithMasking()). TupleDescriptor of such kind of table masking view won't be materialized since the view is simple enough that its query plan is just a ScanNode of the underlying table. The whole query plan can be stitched as if the table is not masked. Note that one day when we support nested columns in SelectList, we may don't need these 2 hacks. This patch also adds some TRACE level loggings to improve debuggability, and enables column masking by default. Test changes in TestRanger.test_column_masking: - Add column masking policy on a table containing nested types. - Add queries on the masked tables. Some queries are borrowed from existing tests for nested types. Tests: - Run CORE tests. Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M tests/authorization/test_ranger.py 16 files changed, 538 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15108/11 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9330: Support resolving unmasked nested columns in masked tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Support resolving unmasked nested columns in masked tables .. Patch Set 10: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/15108/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15108/10//COMMIT_MSG@7 PS10, Line 7: masked tables nit: I would prefer to somehow make the first line shorter and add "+ enable column masking by default" http://gerrit.cloudera.org:8080/#/c/15108/10/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/15108/10/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@a75 PS10, Line 75: The member became unused. -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 14:00:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9330: Support resolving unmasked nested columns in masked tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15108 ) Change subject: IMPALA-9330: Support resolving unmasked nested columns in masked tables .. Patch Set 10: Update the patch to also enable column masking by default. -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 04 Feb 2020 13:53:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9330: Support resolving unmasked nested columns in masked tables
Hello Anurag Mantripragada, Fang-Yu Rao, Vihang Karajgaonkar, Kurt Deschler, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15108 to look at the new patch set (#10). Change subject: IMPALA-9330: Support resolving unmasked nested columns in masked tables .. IMPALA-9330: Support resolving unmasked nested columns in masked tables Column masking policies on primitive columns of a table which contains nested types (though they won't be masked) will cause query failures. To be specifit, if tableA(id int, int_array array) has a masking policy on column "id", all queries on "tableA" will fail, e.g. select id from tableA; select t.id, a.item from tableA t, t.int_array a; Column masking is implemented by wrapping the underlying table/view with a table masking view. However, as we don't support nested types in SelectList, the table masking view can't expose nested columns of the masked table, which causes collection refs not being resolved correctly. This patch fixes the issue by 2 steps: 1) Expose nested columns of the underlying table in the output Type of the table masking view (see InlineViewRef#createTupleDescriptor()). So nested Paths in the original query block can be resolved. 2) For such kind of Paths, resolved them again inside the table masking view. So they can point to the underlying table as what they mean (see Analyzer#resolvePathWithMasking()). TupleDescriptor of such kind of table masking view won't be materialized since the view is simple enough that its query plan is just a ScanNode of the underlying table. The whole query plan can be stitched as if the table is not masked. Note that one day when we support nested columns in SelectList, we may don't need these 2 hacks. This patch also adds some TRACE level loggings to improve debuggability. Column masking is enabled by default after this patch. Test changes in TestRanger.test_column_masking: - Add column masking policy on a table containing nested types. - Add queries on the masked tables. Some queries are borrowed from existing tests for nested types. Tests: - Run CORE tests. Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M tests/authorization/test_ranger.py 16 files changed, 538 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15108/10 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9174: Emit WARNING when ORC lib leaks memory
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15153 ) Change subject: IMPALA-9174: Emit WARNING when ORC lib leaks memory .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I370fb9f68734e0e555bd7224ab0f5440c4947c66 Gerrit-Change-Number: 15153 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 04 Feb 2020 12:35:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8361: Propagate predicates of outer-joined InlineView
Xianqing He has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/15047 ) Change subject: IMPALA-8361: Propagate predicates of outer-joined InlineView .. IMPALA-8361: Propagate predicates of outer-joined InlineView This is an improvement that tries to propagate predicates of the nullable side of the outer join into inline view. For example: SELECT * FROM functional.alltypessmall a LEFT JOIN ( SELECT id, upper(string_col) AS upper_val, length(string_col) AS len FROM functional.alltypestiny ) b ON a.id = b.id WHERE b.upper_val is NULL and b.len = 0 Before this change, the predicate b.len=0 can't be migrated into inline view since that is on the nullable side of an outer join if the predicate evaluates in the inline view nulls will not be rejected. However, we can be more aggressive. In particular, some predicates that must be evaluted at a join node can also be safely evaluted by the outer-joined inline view. Such predicates are not marked as assigned. The predicates propagate into the inline view and also be evaluated at a join node. We can divide predicates into two types. One that satisfies the condition that same as Analyzer#canEvalPredicate can be migrated into inline view， and one that satisfies the below three conditions is safe to be propagated into the nullable side of an outer join. 1) The predicate needs to be bound by tupleIds. 2) The predicate is not on-clause. 3) The predicate evaluates to false when all its referenced tuples are NULL. Therefore, 'b.upper_val is NULL' cannot be propagated to inline view but ‘b.len = 0’ can be propagated to inline view. Tests: * Add plan tests in inline-view.test * One baseline plan in inline-view.test, one in nested-collections.test and two in predicate-propagation.test had to be updated * Ran the full set of verifications in Impala Public Jenkins Change-Id: I6c23a45aeb5dd1aa06a95c9aa8628ecbe37ef2c1 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test 7 files changed, 281 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/15047/10 -- To view, visit http://gerrit.cloudera.org:8080/15047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c23a45aeb5dd1aa06a95c9aa8628ecbe37ef2c1 Gerrit-Change-Number: 15047 Gerrit-PatchSet: 10 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He
[Impala-ASF-CR] IMPALA-9290: ORC scanner should support schema evolution between date and timestamp types
Norbert Luksa has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/15152 ) Change subject: IMPALA-9290: ORC scanner should support schema evolution between date and timestamp types .. IMPALA-9290: ORC scanner should support schema evolution between date and timestamp types This feature adds support for schema evolution between date and timestamp for the ORC scanner. This means that we can have two tables, one with a date column, another with a timestamp column, and they can both point to the same set of ORC files. The result will be that for the first table everything will be converted to date, and for the second, everything to timestamp. In order to do that, the OrcTimestampReader and OrcDateColumnReader are modified to be able to handle batches of the two types. Their name now represents the destination Impala type. Note that the life cycle of a OrcColumnReader is within the life cycle of the HdfsOrcScanner which only reads a split of an ORC file, and an ORC file can't have two types for one column. Tests: * Added type conversion tests. * Tested manually following the use case steps of the Jira. Change-Id: I7979ecc61b2ab900090d01bc81e7bb7b28c99c9e --- M be/src/exec/hdfs-orc-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/orc-metadata-utils.cc M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M tests/query_test/test_scanners.py 6 files changed, 132 insertions(+), 53 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/15152/3 -- To view, visit http://gerrit.cloudera.org:8080/15152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7979ecc61b2ab900090d01bc81e7bb7b28c99c9e Gerrit-Change-Number: 15152 Gerrit-PatchSet: 3 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9290: ORC scanner should support schema evolution between date and timestamp types
Norbert Luksa has posted comments on this change. (
http://gerrit.cloudera.org:8080/15152 )
Change subject: IMPALA-9290: ORC scanner should support schema evolution
between date and timestamp types
..
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/15152/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15152/2//COMMIT_MSG@7
PS2, Line 7: between date and timestamp types
> Looking at IMPALA-6373 it seems like Impala usually supports these conversi
Parquet does not support this kind of compatibility between date and timestamp.
I created date and timestamp tables and set them to point to the same location
(as described in the Jira), and inserted some values to both tables. When I try
to query any of them, I get an error that the schemas of the files are
incompatible.
I think here we do not wish to follow Parquet and other file formats, but Hive.
http://gerrit.cloudera.org:8080/#/c/15152/2//COMMIT_MSG@12
PS2, Line 12: to the same set of O
> nit: "to the same set of ORC files" probably expresses the intent better
Done
http://gerrit.cloudera.org:8080/#/c/15152/2/be/src/exec/orc-column-readers.h
File be/src/exec/orc-column-readers.h:
http://gerrit.cloudera.org:8080/#/c/15152/2/be/src/exec/orc-column-readers.h@121
PS2, Line 121:
> nit: Please add comment about what is an invalid value and how it is handle
Done
http://gerrit.cloudera.org:8080/#/c/15152/2/be/src/exec/orc-column-readers.h@214
PS2, Line 214: split of an ORC file, an
> Does Hive convert in both directions?
Yes, it does.
http://gerrit.cloudera.org:8080/#/c/15152/2/be/src/exec/orc-column-readers.h@214
PS2, Line 214: HdfsOrcScanner which only re
> What does it mean "ORC support schema evolution"?
Done
http://gerrit.cloudera.org:8080/#/c/15152/2/be/src/exec/orc-column-readers.h@253
PS2, Line 253:
: class OrcDateColumnReader : public OrcDateAndTimeStampReader {
: public:
: OrcDateColumnReader(const orc::Type* node, const
SlotDescriptor* slot_desc,
: HdfsOrcScanner* scanner)
: : OrcDateAndTimeStampReader(node, slot_desc, scanne
> nit: complicated and duplicated in OrcTimestampReader. Can you put it into
Done
--
To view, visit http://gerrit.cloudera.org:8080/15152
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7979ecc61b2ab900090d01bc81e7bb7b28c99c9e
Gerrit-Change-Number: 15152
Gerrit-PatchSet: 3
Gerrit-Owner: Norbert Luksa
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Norbert Luksa
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 04 Feb 2020 10:01:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8587: Show inherited privileges with Ranger show grant
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15111 ) Change subject: IMPALA-8587: Show inherited privileges with Ranger show grant .. Patch Set 5: Code-Review+2 Thanks for the changes! -- To view, visit http://gerrit.cloudera.org:8080/15111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4e679dc6fcf8d0b0e4e0fc2e9b335e2d8bc0899 Gerrit-Change-Number: 15111 Gerrit-PatchSet: 5 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 04 Feb 2020 09:59:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9174: Speedup allocations of ORC Scanner
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/14804 ) Change subject: IMPALA-9174: Speedup allocations of ORC Scanner .. Patch Set 7: Based on my newest measurements it doesn't cause any perf improvement unfortunately. It also makes the code a bit less safe, since it doesn't free memory leaked by the ORC lib. Therefore I'm gonna abandon this CR for now. -- To view, visit http://gerrit.cloudera.org:8080/14804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia09e746883176d6f955c1718267bf55e2abb239b Gerrit-Change-Number: 14804 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Feb 2020 08:20:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9174: Speedup allocations of ORC Scanner
Zoltan Borok-Nagy has abandoned this change. ( http://gerrit.cloudera.org:8080/14804 ) Change subject: IMPALA-9174: Speedup allocations of ORC Scanner .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/14804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ia09e746883176d6f955c1718267bf55e2abb239b Gerrit-Change-Number: 14804 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
