[native-toolchain-CR] Add missing patch for orc 1.5.5
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13082 Change subject: Add missing patch for orc 1.5.5 .. Add missing patch for orc 1.5.5 This change adds a patch that I had forgotten to add to this review: https://gerrit.cloudera.org/#/c/12998/ I tested this locally by building the toolchain. Change-Id: Iff84c883c50efdc6b438f5428e947ff0d0960ff8 --- A source/orc/orc-1.5.5-patches/0001-ORC-396-also-look-for-LZ4-libs-in-lib64-subdir.patch 1 file changed, 25 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/82/13082/1 -- To view, visit http://gerrit.cloudera.org:8080/13082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iff84c883c50efdc6b438f5428e947ff0d0960ff8 Gerrit-Change-Number: 13082 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2856/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Apr 2019 01:13:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 2: Code-Review+1 I'll let Bikram take it to +2. -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Apr 2019 00:48:20 + Gerrit-HasComments: No
[native-toolchain-CR] WIP: add zstd to toolchain
Tim Armstrong has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13079 ) Change subject: WIP: add zstd to toolchain .. WIP: add zstd to toolchain This adds the latest release of Testing: Built locally, confirmed that binary utilities work. Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 --- M buildall.sh A source/zstd/build.sh 2 files changed, 40 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/79/13079/3 -- To view, visit http://gerrit.cloudera.org:8080/13079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 Gerrit-Change-Number: 13079 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong
[native-toolchain-CR] WIP: add zstd to toolchain
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13079 Change subject: WIP: add zstd to toolchain .. WIP: add zstd to toolchain Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 --- M buildall.sh A source/zstd/build.sh 2 files changed, 39 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/79/13079/1 -- To view, visit http://gerrit.cloudera.org:8080/13079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 Gerrit-Change-Number: 13079 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/13078 )
Change subject: IMPALA-8446: Create a unit test for Admission Controller.
..
Patch Set 2:
(4 comments)
Thanks Tim. I made fixes in patch set 2.
http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc
File be/src/scheduling/admission-controller-test.cc:
http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@49
PS1, Line 49: static const string HOST_2 = "host2:25000";
> nit: static const or static constexpr
Done
http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@86
PS1, Line 86: onst string pool_name, TPoolStats stats) {
: // Build topic item.
> unused args?
Done
http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@109
PS1, Line 109: ckPoo
> values
Done
http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@115
PS1, Line 115:
> Maybe ASSERT_OK since the rest of assertions don't make sense if this fails
Done
--
To view, visit http://gerrit.cloudera.org:8080/13078
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c
Gerrit-Change-Number: 13078
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 23 Apr 2019 00:28:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Andrew Sherman has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. IMPALA-8446: Create a unit test for Admission Controller. This test allows construction of white box tests that exercise Admission Controller code. The initial test case is a skeleton which does the following: + creates a RequestPoolService which reads some Admission Controller configuration files + checks that the configurations were read correctly + creates an Admission Controller object + creates a QuerySchedule for a query + tests if the query can be admitted by the Admission Controller + simulates activity in the cluster which consumes memory + tests that the earlier query cannot now be admitted by the Admission Controller. TESTING: Ran end-to-end tests cleanly and checked that the new test was executed. Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c --- M be/src/scheduling/CMakeLists.txt A be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h 6 files changed, 251 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13078/2 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12797 )
Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
..
Patch Set 10:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.h
File be/src/exprs/scalar-expr.h:
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.h@87
PS10, Line 87: Get*Val() dispatches to either
: /// a codegen'd function pointer or to an interpreted
implementation Get*ValInterpreted()
: /// These interpreted functions must be overridden by subclasses
of ScalarExpr for every
: /// type that they may return.
: ///
: /// The two main usage patterns for ScalarExpr are:
: /// * The codegen'd expressions are called from other codegen'd
functions, e.g. from a
: /// codegen'd join implementation
: /// * Get*Val() is called on the root of each expression subtree
by interpreted code.
: /// We can optimize for the second usage pattern by filling in
the codegen'd function
: /// pointer (codegend_compute_fn_) in root of each ScalarExpr
tree. Individual callsites
: /// can disable this optimisation if it's not needed. Expr
subtrees can be evaluated
: /// (e.g. by ScalarExprEvaluator::GetConstValue()) but may fail
back to a slower
: /// interpreted implementation.
These sets of comments seem to fit better after line 107-114. Reading this
comment, it's a bit hard to understand what's being talked about without
knowing what codegen is. It will be clearer to first do an introduction of
codegen and its relationship with ScalarExpr and various functions before
pointing out that Get*Val() is a wrapper which dispatches to codegen'd function
by default but can also fall back to the interpreted version if it's not
codegen'd.
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.h@170
PS10, Line 170: bool is_codegen_entry_point,
I wonder how much measurable benefit we get from this. IMHO, this can be an
optimization as a follow-up patch. Doesn't seem 100% necessary functionally for
this patch.
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.h@288
PS10, Line 288: If 'is_entry_point' is true, this indicates that Get*Val()
: /// may be called directly from interpreted code and that we
should generate an entry
: /// point into the codegen'd code.
Please see comments in scalar-expr.cc. 'is_entry_point' may not need to be
exposed.
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.h@349
PS10, Line 349: /// The vast majority of exprs support interpretation, so
default to true.
May want to point out that expressions which aren't interpretable should
override this function etc.
Will also help to explain more elaborately why an expression is not
interpretable. IMHO, this concept is a bit more on the side of implementation
details and exposing this interface is a bit unfortunate as it inter-mingles
the already complicated concept of ScalarExpr with codegen.
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.h@375
PS10, Line 375: codegend_compute_fn_ = nullptr;
Please document that this is left as null if this scalar expression is not an
"entry point".
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.cc
File be/src/exprs/scalar-expr.cc:
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.cc@284
PS10, Line 284: bool is_entry_point
May be I didn't follow all the details correctly but it seems to me that all
these plumbing for passing this flag around is only used in
GetCodegendComputeFn().
In theory, only the top level expression (i.e. the root of the scalar
expression tree) may require exposing itself with a function pointer after
codegen (i.e. it's an entry point).
So, why cannot 'is_entry_point' be a property of a ScalarExpr itself ? In other
words, a simple implementation is to set 'is_entry_point_' to true for the root
of all ScalarExpr and leave it as false for any sub-expressions in the tree. In
this way, we don't need to plumb this flag around. More importantly, this seems
to be implementation specific details / optimization that we probably should
try not to expose it in the interface of Init() if possible.
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.cc@409
PS10, Line 409: llvm::Function* static_getval_fn =
GetStaticGetValWrapper(type(), codegen);
if (static_getval_fn == nullptr) {
return
}
We only get a DCHECK in debug build. Failing to codegen shouldn't be fatal.
http://gerrit.cloudera.org:8080/#/c/12797/10/be/src/exprs/scalar-expr.inline.h
File be/src/exprs/scalar-expr.inline.h:
[Impala-ASF-CR] [WIP] IMPALA-8281: Add support for show grant user/group with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13074 ) Change subject: [WIP] IMPALA-8281: Add support for show grant user/group with Ranger .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2853/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Apr 2019 22:36:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-8281: Add support for show grant user/group with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13074 ) Change subject: [WIP] IMPALA-8281: Add support for show grant user/group with Ranger .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2855/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Apr 2019 22:43:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2854/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Apr 2019 22:42:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 1: (5 comments) Looks good overall. I think Bikram should take a look too since he's spent more time hands on with this code. http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc File be/src/scheduling/admission-controller-test.cc: http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@49 PS1, Line 49: string QUEUE_A = "root.queueA"; nit: static const or static constexpr http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@86 PS1, Line 86: , bool is_coordinator = false, : bool is_executor = false unused args? http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@109 PS1, Line 109: vales values http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@115 PS1, Line 115: EXPECT_OK Maybe ASSERT_OK since the rest of assertions don't make sense if this fails. http://gerrit.cloudera.org:8080/#/c/13078/1/be/src/scheduling/admission-controller-test.cc@126 PS1, Line 126: /// Simple AdmissionController test. Maybe elaborate slightly on what it's testing? -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Apr 2019 22:21:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-8281: Add support for show grant user/group with Ranger
Austin Nobis has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13074 ) Change subject: [WIP] IMPALA-8281: Add support for show grant user/group with Ranger .. [WIP] IMPALA-8281: Add support for show grant user/group with Ranger Add support for SHOW GRANT statements for Apache Ranger. This patch also adds the RangerImpaladAuthorizationManager as the show grant statement is called from impalad. Testing: - Ran all FE unit tests - Ran authorization E2E tests - Updated test_ranger to use show grant statement for verification of granted privileges Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/authorization/test_ranger.py 10 files changed, 489 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13074/6 -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2852/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 10 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 22 Apr 2019 22:14:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13078 Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. IMPALA-8446: Create a unit test for Admission Controller. This test allows construction of white box tests that exercise Admission Controller code. The initial test case is a skeleton which does the following: + creates a RequestPoolService which reads some Admission Controller configuration files + checks that the configurations were read correctly + creates an Admission Controller object + creates a QuerySchedule for a query + tests if the query can be admitted by the Admission Controller + simulates activity in the cluster which consumes memory + tests that the earlier query cannot now be admitted by the Admission Controller. TESTING: Ran end-to-end tests cleanly and checked that the new test was executed. Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c --- M be/src/scheduling/CMakeLists.txt A be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h 6 files changed, 250 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13078/1 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/307/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 22 Apr 2019 21:53:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-8281: Add support for show grant user/group with Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13074 ) Change subject: [WIP] IMPALA-8281: Add support for show grant user/group with Ranger .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/13074/5/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13074/5/tests/authorization/test_ranger.py@133 PS5, Line 133: flake8: E203 whitespace before ',' http://gerrit.cloudera.org:8080/#/c/13074/5/tests/authorization/test_ranger.py@145 PS5, Line 145: flake8: E203 whitespace before ',' http://gerrit.cloudera.org:8080/#/c/13074/5/tests/authorization/test_ranger.py@171 PS5, Line 171: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Apr 2019 21:57:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-8281: Add support for show grant user/group with Ranger
Austin Nobis has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13074 Change subject: [WIP] IMPALA-8281: Add support for show grant user/group with Ranger .. [WIP] IMPALA-8281: Add support for show grant user/group with Ranger Add support for SHOW GRANT statements for Apache Ranger. This patch also adds the RangerImpaladAuthorizationManager as the show grant statement is called from impalad. Testing: - Ran all FE unit tests - Ran authorization E2E tests - Updated test_ranger to use show grant statement for verification of granted privileges Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/authorization/test_ranger.py 10 files changed, 489 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13074/5 -- To view, visit http://gerrit.cloudera.org:8080/13074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4 Gerrit-Change-Number: 13074 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis
[Impala-ASF-CR] IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12939 ) Change subject: IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12939/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java: http://gerrit.cloudera.org:8080/#/c/12939/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1190 PS4, Line 1190: Expr expr = e.trySubstitute(inlineViewRef.getBaseTblSmap(), analyzer, false); > OK. Maybe we can add an assertion to flush out buggy cases of "smap[e1] == Yeah if we could add such a precondition at an appropriate place it would be useful, hopefully we can confirm they they hold or find out a reason why they don't? -- To view, visit http://gerrit.cloudera.org:8080/12939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa Gerrit-Change-Number: 12939 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Apr 2019 21:42:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 10: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 10 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 22 Apr 2019 21:33:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13006/1/docs/topics/impala_explain_plan.xml File docs/topics/impala_explain_plan.xml: http://gerrit.cloudera.org:8080/#/c/13006/1/docs/topics/impala_explain_plan.xml@238 PS1, Line 238: and other processes : running on the same system. > This does not include processes on the same system (we exclude the loopback Done -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 22 Apr 2019 21:31:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/307/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 22 Apr 2019 21:31:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13006 to look at the new patch set (#2). Change subject: IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles .. IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 --- M docs/topics/impala_explain_plan.xml 1 file changed, 21 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13006/2 -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12901 to look at the new patch set (#10). Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger Wrote unit tests to capture above assertions Ran fe unit and e2e tests Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 15 files changed, 304 insertions(+), 60 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/10 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 10 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8344: Add support for running the minicluster with S3Guard
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13020 ) Change subject: IMPALA-8344: Add support for running the minicluster with S3Guard .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2851/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c748529a494bb6e70fec96dc031523ff79bf61d Gerrit-Change-Number: 13020 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Apr 2019 20:45:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 2: (4 comments) These changes seem fine, thanks for fixing. There's going to be the risk of regressions until we test this pre-commit, and I don't want to ask contributors to do extra work before we've automated it, but no harm in fixing them best-effort. I think we could avoid print-related regressions with "from __future__ import print_function", so I suggested a few places to add that. Once the few minor things I suggested are fixed, looks good to merge. http://gerrit.cloudera.org:8080/#/c/13070/2/infra/python/bootstrap_virtualenv.py File infra/python/bootstrap_virtualenv.py: PS2: Maybe add "from __future__ import print_function" to prevent this regressing in future: https://stackoverflow.com/questions/32032697/how-to-use-from-future-import-print-function http://gerrit.cloudera.org:8080/#/c/13070/2/infra/python/deps/find_py26.py File infra/python/deps/find_py26.py: PS2: Maybe add "from __future__ import print_function" to prevent this regressing in future: https://stackoverflow.com/questions/32032697/how-to-use-from-future-import-print-function http://gerrit.cloudera.org:8080/#/c/13070/2/infra/python/deps/pip_download.py File infra/python/deps/pip_download.py: PS2: Maybe add "from __future__ import print_function" to prevent this regressing in future: https://stackoverflow.com/questions/32032697/how-to-use-from-future-import-print-function http://gerrit.cloudera.org:8080/#/c/13070/2/infra/python/deps/pip_download.py@32 PS2, Line 32: try: Can you add a one line comment to explain this? -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 2 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 22 Apr 2019 20:49:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/12901 )
Change subject: IMPALA-8309: add user authorization_provider flag
..
Patch Set 9:
(19 comments)
http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java@141
PS9, Line 141: getAuthorizationFactoryClassOrNull
> nit: remove the `orNull` in the function name
because the annotation is already present or because you don't like the name?
i think there should be some indication that the return value can be null, if
we're not using Optional
http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/JniCatalog.java@114
PS9, Line 114: //<<< 67f77d41d40523074385b8dbccfa6ef6ef81dd57
> Why is this commented out?
ugh was sanity checking a merge conflict and forgot to delete
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py
File tests/authorization/test_provider.py:
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@21
PS5, Line 21: import os
> flake8: F401 'time' imported but unused
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@25
PS5, Line 25: from tests.common.custom_cluster_test_suite import
CustomClusterTestSuite
> flake8: F401 'tests.util.filesystem_utils.IS_LOCAL' imported but unused
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@28
PS5, Line 28: class TestProviderFails(CustomClusterTestSuite):
> flake8: F401 'tests.common.impala_test_suite.ImpalaTestSuite' imported but
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@38
PS5, Line 38: LOG_DIR = tempfile.mkdtemp(prefix="test_provider_",
dir=os.getenv("LOG_DIR"))
> line has trailing whitespace
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@39
PS5, Line 39:
> flake8: E302 expected 2 blank lines, found 1
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@53
PS5, Line 53: e
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@54
PS5, Line 54:
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@55
PS5, Line 55: a
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@58
PS5, Line 58:
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@59
PS5, Line 59:
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@60
PS5, Line 60:
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@66
PS5, Line 66:
> flake8: E502 the backslash is redundant between brackets
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@67
PS5, Line 67:
> flake8: E501 line too long (92 > 90 characters)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/5/tests/authorization/test_provider.py@68
PS5, Line 68:
> flake8: E124 closing bracket does not match visual indentation
Ack
http://gerrit.cloudera.org:8080/#/c/12901/6/tests/authorization/test_provider.py
File tests/authorization/test_provider.py:
http://gerrit.cloudera.org:8080/#/c/12901/6/tests/authorization/test_provider.py@60
PS6, Line 60:
not the happiest with this approach; open to suggestions
http://gerrit.cloudera.org:8080/#/c/12901/7/tests/authorization/test_provider.py
File tests/authorization/test_provider.py:
http://gerrit.cloudera.org:8080/#/c/12901/7/tests/authorization/test_provider.py@27
PS7, Line 27:
> flake8: E302 expected 2 blank lines, found 1
Ack
http://gerrit.cloudera.org:8080/#/c/12901/4/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/12901/4/tests/authorization/test_ranger.py@38
PS4, Line 38: """
> Would prefer if there was an e2e test with an invalid as well as valid `--a
Ack
--
To view, visit http://gerrit.cloudera.org:8080/12901
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java File fe/src/main/java/org/apache/impala/service/BackendConfig.java: http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java@141 PS9, Line 141: getAuthorizationFactoryClassOrNull nit: remove the `orNull` in the function name http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/12901/9/fe/src/main/java/org/apache/impala/service/JniCatalog.java@114 PS9, Line 114: //<<< 67f77d41d40523074385b8dbccfa6ef6ef81dd57 Why is this commented out? -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 9 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 22 Apr 2019 19:58:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8344: Add support for running the minicluster with S3Guard
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13020 ) Change subject: IMPALA-8344: Add support for running the minicluster with S3Guard .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13020/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/13020/2/bin/impala-config.sh@310 PS2, Line 310: export S3GUARD_METADATASTORE_IMPL="org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore" line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c748529a494bb6e70fec96dc031523ff79bf61d Gerrit-Change-Number: 13020 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Apr 2019 20:02:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8344: Add support for running the minicluster with S3Guard
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13020 to look at the new patch set (#2). Change subject: IMPALA-8344: Add support for running the minicluster with S3Guard .. IMPALA-8344: Add support for running the minicluster with S3Guard Some tests can fail on S3 due to some operations that are eventually consistent. S3Guard stores extra metadata in a DynamoDB to solve several consistency issues. This adds support for running the minicluster on S3 with S3Guard. S3Guard is configured by the following environment variables: S3GUARD_ENABLED: defaults to false, set to true to enable S3Guard S3GUARD_DYNAMODB_TABLE: name of the DynamoDB table to use. This must be exclusively owned by this minicluster. The dataload scripts initialize this table and will purge entries if the table already exists. The table should be in the same region as the S3_BUCKET for the minicluster. S3GUARD_DYNAMODB_REGION - AWS region for S3GUARD_DYNAMODB_TABLE These environment variables only impact S3 configurations. The support comes from three pieces: 1. Configuration changes in core-site.xml to add the appropriate parameters. 2. Updating dataload to initialize/purge the s3guard dynamodb table and import data appropriately. 3. Update tests to manipulate files through the HDFS command line rather than through s3 utilities. This takes the filesystem utility code for ABFS (which actually calls HDFS command line), makes it generic, and uses it for S3Guard. Testing: - Ran multiple rounds of s3 tests - Aborted tests in the middle and restarted the s3 tests (to test the s3guard reinitialization code) Change-Id: I3c748529a494bb6e70fec96dc031523ff79bf61d --- M bin/impala-config.sh A bin/jenkins/release_cloud_resources.sh M testdata/bin/load-test-warehouse-snapshot.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py D tests/util/abfs_util.py M tests/util/filesystem_utils.py M tests/util/hdfs_util.py 8 files changed, 185 insertions(+), 117 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/13020/2 -- To view, visit http://gerrit.cloudera.org:8080/13020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3c748529a494bb6e70fec96dc031523ff79bf61d Gerrit-Change-Number: 13020 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
