[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4536/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 05:26:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 11 Sep 2019 05:08:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14203 to look at the new patch set (#10). Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 2 files changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/10 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4930/ -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 04:20:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 11: (1 comment) The changes look good, will wait to hear what you find out from debugging. http://gerrit.cloudera.org:8080/#/c/14143/11/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/14143/11/be/src/service/client-request-state.cc@780 PS11, Line 780: if (wait_thread_.get() == nullptr) return; Do we need the wait_thread_ check though? Since after the wait thread is destroyed is_wait_done_ will still be true. And I think we shouldn't be calling this before the wait thread is created (or if we do, we should block here). -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 11 Sep 2019 04:07:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
Patch Set 6:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java
File fe/src/main/java/org/apache/impala/authorization/Authorizable.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@62
PS6, Line 62: return getName().hashCode();
If we're comparing owner, we should update the hashCode() as well.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@70
PS6, Line 70: temp.getOwnerUser() == this.getOwnerUser()
Shouldn't this use equals()?
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@31
PS6, Line 31: private final String ownerUser_;
For consistency, annotate with @Nullable.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@34
PS6, Line 34: String ownerUser
I think it's more readable to also annotate the parameter with @Nullable.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java@33
PS6, Line 33: String ownerUser
Same comment about @Nullable parameter.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java@45
PS6, Line 45: /**
:* Creates a new instance of column {@link Authorizable} for a
given database name and
:* gives access to all tables and columns.
:*/
: Authorizable newColumnAllTbls(String dbName, @Nullable String
dbOwnerUser);
:
: /**
:* Creates a new instance of column {@link Authorizable} for
given database and table
:* names and gives access to all columns.
:*/
: Authorizable newColumnInTable(
: String dbName, String tableName, @Nullable String
tblOwnerUser);
:
: /**
:* Creates a new instance of column {@link Authorizable} for
given database, table, and
:* column names.
:*/
: Authorizable newColumnInTable(
: String dbName, String tableName, String columnName,
@Nullable String tblOwnerUser);
nit: update javadoc about the new owner parameter.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java@34
PS6, Line 34: String ownerUser
Same comment about @Nullable parameter.
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@94
PS6, Line 94: resources.add(new RangerImpalaResourceBuilder()
: .database("*").table("*").column("*").build());
: resources.add(new RangerImpalaResourceBuilder()
: .database("*").function("*").build());
: resources.add(new
RangerImpalaResourceBuilder().uri("*").build());
Should we add .owner() here as well?
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java@544
PS6, Line 544: if (db == null) return null;
: return db.getOwnerName();
nit: can be inlined: return db == null ? null : db.getOwnerName()
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
[Impala-ASF-CR] IMPALA-8936: Improve queuing reason for unhealthy executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14210 ) Change subject: IMPALA-8936: Improve queuing reason for unhealthy executor groups .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4932/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idceab7fb56335bab9d787b0f351a41e6efd7dd59 Gerrit-Change-Number: 14210 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 03:55:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8936: Improve queuing reason for unhealthy executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14210 ) Change subject: IMPALA-8936: Improve queuing reason for unhealthy executor groups .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4535/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idceab7fb56335bab9d787b0f351a41e6efd7dd59 Gerrit-Change-Number: 14210 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 03:55:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8936: Improve queuing reason for unhealthy executor groups
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14210 ) Change subject: IMPALA-8936: Improve queuing reason for unhealthy executor groups .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idceab7fb56335bab9d787b0f351a41e6efd7dd59 Gerrit-Change-Number: 14210 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 03:36:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8936: Improve queuing reason for unhealthy executor groups
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14210 Change subject: IMPALA-8936: Improve queuing reason for unhealthy executor groups .. IMPALA-8936: Improve queuing reason for unhealthy executor groups In some situations, users might actually expect not having a healthy executor group around, e.g. when they're starting one and it takes a while to come online. This change makes the queuing reason more generic and drops the "unhealthy" concept from it to reduce confusion. Change-Id: Idceab7fb56335bab9d787b0f351a41e6efd7dd59 --- M be/src/scheduling/admission-controller.cc M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py 3 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/14210/1 -- To view, visit http://gerrit.cloudera.org:8080/14210 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idceab7fb56335bab9d787b0f351a41e6efd7dd59 Gerrit-Change-Number: 14210 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4534/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 11 Sep 2019 01:08:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4533/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:53:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4532/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:54:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4931/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 11 Sep 2019 00:51:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14143 )
Change subject: IMPALA-8572: Log query events before unregister.
..
Patch Set 11:
(5 comments)
Still not clear on the test failures. Addressing the comments and kicking off
another run while I look into it.
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@749
PS10, Line 749: if (wait_thread_.get() != nullptr) {
> Should we drop the lock while joining on the wait thread? I feel like there
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@754
PS10, Line 754: // flush log events to audit authorization errors, if any.
> We could probably also drop the lock when calling this.
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@780
PS10, Line 780: if (wait_thread_.get() == nullptr) return;
> I think we can actually just delete this line, the next line is sufficient.
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@1395
PS10, Line 1395: Status status;
> Accessing query_status without acquiring a lock might not actually be safe
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@1426
PS10, Line 1426: }
> Should be a const TExecRequest& to avoid copying.
nice find.
--
To view, visit http://gerrit.cloudera.org:8080/14143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e
Gerrit-Change-Number: 14143
Gerrit-PatchSet: 11
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Wed, 11 Sep 2019 00:51:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4531/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:50:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#11). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,016 insertions(+), 243 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/11 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4530/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:47:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14203 )
Change subject: IMPALA-8933: Enforce ranger deny policy
..
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14203/9/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14203/9/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@237
PS9, Line 237: request.getPrivilege() != Privilege.INSERT) {
So we don't need to worry about ALTER?
--
To view, visit http://gerrit.cloudera.org:8080/14203
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da
Gerrit-Change-Number: 14203
Gerrit-PatchSet: 9
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 11 Sep 2019 00:36:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:14:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:14:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4930/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:14:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Tim Armstrong has uploaded a new patch set (#8) to the change originally created by Kurt Deschler. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java 1 file changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/8 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14203 to look at the new patch set (#7). Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java 1 file changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/7 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8897 (part 3): Fix some additional javascript urls
Thomas Tauber-Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14206 ) Change subject: IMPALA-8897 (part 3): Fix some additional javascript urls .. IMPALA-8897 (part 3): Fix some additional javascript urls Part 2 of this patch missed rewriting two urls in javascript. This patch fixes it by adding 'make_url()' around those urls. Testing: - Deployed to a real cluster and verified that Knox integreation works correctly on the relevant pages now. - Extended test_knox_integration to check for this case. Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Reviewed-on: http://gerrit.cloudera.org:8080/14206 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M tests/webserver/test_web_pages.py M www/query_backends.tmpl M www/query_finstances.tmpl 3 files changed, 6 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Gerrit-Change-Number: 14206 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14203 )
Change subject: IMPALA-8933: Enforce ranger deny policy
..
Patch Set 6:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@234
PS6, Line 234: // In order to support deny policies on columns
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@235
PS6, Line 235: if (hasTableSelectPriv &&
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@236
PS6, Line 236: request.getPrivilege() != Privilege.SELECT &&
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@237
PS6, Line 237: request.getPrivilege() != Privilege.INSERT) {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@238
PS6, Line 238: continue;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@239
PS6, Line 239: }
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@240
PS6, Line 240: if (hasAccess(authzCtx, analyzer.getUser(), request)) {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@241
PS6, Line 241: hasColumnSelectPriv = true;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@242
PS6, Line 242: continue;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/6/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@243
PS6, Line 243: }
tab used for whitespace
--
To view, visit http://gerrit.cloudera.org:8080/14203
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da
Gerrit-Change-Number: 14203
Gerrit-PatchSet: 6
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 11 Sep 2019 00:10:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14203 to look at the new patch set (#6). Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java 1 file changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/6 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8897 (part 3): Fix some additional javascript urls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14206 ) Change subject: IMPALA-8897 (part 3): Fix some additional javascript urls .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Gerrit-Change-Number: 14206 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 11 Sep 2019 00:09:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14203 )
Change subject: IMPALA-8933: Enforce ranger deny policy
..
Patch Set 5:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@234
PS5, Line 234: // In order to support deny policies on columns
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@235
PS5, Line 235: if (hasTableSelectPriv &&
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@236
PS5, Line 236: request.getPrivilege() != Privilege.SELECT &&
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@237
PS5, Line 237: request.getPrivilege() != Privilege.INSERT) {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@238
PS5, Line 238: continue;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@239
PS5, Line 239: }
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@240
PS5, Line 240: if (hasAccess(authzCtx, analyzer.getUser(), request)) {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@241
PS5, Line 241: hasColumnSelectPriv = true;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@242
PS5, Line 242: continue;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/5/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@243
PS5, Line 243: }
tab used for whitespace
--
To view, visit http://gerrit.cloudera.org:8080/14203
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da
Gerrit-Change-Number: 14203
Gerrit-PatchSet: 5
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 11 Sep 2019 00:07:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14203 to look at the new patch set (#5). Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java 1 file changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/5 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14209 ) Change subject: IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/477/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14209 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia48a4a6711ffcf326f3f51200040420101451967 Gerrit-Change-Number: 14209 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Sep 2019 23:53:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/service/Frontend.java@783
PS6, Line 783: TODO
mind filing a JIRA and putting the number here?
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
File
fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3060
PS6, Line 3060: authorize("select count(*) from functional.alltypes")
nit: might make this test a little easier to read to do something like create
an Map:
ImmutableMap.builder()
.put("select count(*) from functional.alltypes",
selectError("functional.alltypes"))
.put("select id from functional.alltypes",
selectError("functional.alltypes"))
...
.build();
and then you can loop over the queries in the before/after case below to avoid
having to duplicate all the queries twice
http://gerrit.cloudera.org:8080/#/c/14106/6/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/14106/6/tests/authorization/test_ranger.py@665
PS6, Line 665: # Run show tables on the db. The resulting list should be
empty. This happens
: # because the created table's ownership information is
not aggresively cached
: # by the current Catalog implementations. Hence the
analysis pass does not
: # have access to the ownership information to verify if
the current session
: # user is actually the owner. We need to fix this by
caching the HMS metadata
: # more aggressively when the table loads.
this is true only in localcatalog, right? otherwise the 'create table' call
would have fully populated the cache rather than leaving an incomplete table?
http://gerrit.cloudera.org:8080/#/c/14106/6/tests/authorization/test_ranger.py@669
PS6, Line 669: We need to fix this by caching the HMS metadata
: # more aggressively when the table loads.
add a TODO(IMPALA-x) with jira
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 10 Sep 2019 23:49:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14209 ) Change subject: IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/477/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14209 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia48a4a6711ffcf326f3f51200040420101451967 Gerrit-Change-Number: 14209 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 23:43:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14209 Change subject: IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT .. IMPALA-8548: [DOCS] Impala supports ordinal substitutions in SELECT Change-Id: Ia48a4a6711ffcf326f3f51200040420101451967 --- M docs/shared/impala_common.xml M docs/topics/impala_select.xml 2 files changed, 11 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/14209/1 -- To view, visit http://gerrit.cloudera.org:8080/14209 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia48a4a6711ffcf326f3f51200040420101451967 Gerrit-Change-Number: 14209 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4529/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 4 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 23:27:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4528/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 12 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 10 Sep 2019 23:20:04 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14197 ) Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4928/ -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 23:17:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8580: [DOCS] Document the date time patterns supported
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14207 ) Change subject: IMPALA-8580: [DOCS] Document the date time patterns supported .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/476/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba9952f3c8017accb1a9b7e4cd693a7aeb6630d0 Gerrit-Change-Number: 14207 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 23:16:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8580: [DOCS] Document the date time patterns supported
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14207 ) Change subject: IMPALA-8580: [DOCS] Document the date time patterns supported .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/476/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba9952f3c8017accb1a9b7e4cd693a7aeb6630d0 Gerrit-Change-Number: 14207 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 23:06:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8580: [DOCS] Document the date time patterns supported
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14207 Change subject: IMPALA-8580: [DOCS] Document the date time patterns supported .. IMPALA-8580: [DOCS] Document the date time patterns supported - Documented the subset of Java SimpleDateFormat that Impala supports - Changed "formats" to "patterns" to be consistent within this doc Change-Id: Iba9952f3c8017accb1a9b7e4cd693a7aeb6630d0 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 175 insertions(+), 48 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14207/1 -- To view, visit http://gerrit.cloudera.org:8080/14207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iba9952f3c8017accb1a9b7e4cd693a7aeb6630d0 Gerrit-Change-Number: 14207 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 10: (5 comments) Left a few more comments while trying to understand the test failure. http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@749 PS10, Line 749: wait_thread_->Join(); Should we drop the lock while joining on the wait thread? I feel like there's some risk of introducing a deadlock if Wait() ever acquires the lock. I think this should be safe since no other thread should be concurrently modifyin wait_thread_. http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@754 PS10, Line 754: LogQueryEvents(); We could probably also drop the lock when calling this. http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@780 PS10, Line 780: if (wait_thread_.get() == nullptr || is_wait_done_) return; I think we can actually just delete this line, the next line is sufficient. http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@1395 PS10, Line 1395: Status status = query_status(); Accessing query_status without acquiring a lock might not actually be safe now - I think it previous used to work because this code ran *after* the query status could no longer change. http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@1426 PS10, Line 1426: const TExecRequest request = exec_request(); Should be a const TExecRequest& to avoid copying. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 23:01:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4526/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 22:59:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4527/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 22:58:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14203 )
Change subject: IMPALA-8933: Enforce ranger deny policy
..
Patch Set 4:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@234
PS4, Line 234: // In order to support deny policies on columns
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@235
PS4, Line 235: if (hasTableSelectPriv &&
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@236
PS4, Line 236: request.getPrivilege() != Privilege.SELECT &&
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@237
PS4, Line 237: request.getPrivilege() != Privilege.INSERT) {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@238
PS4, Line 238: continue;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@239
PS4, Line 239: }
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@240
PS4, Line 240: if (hasAccess(authzCtx, analyzer.getUser(), request)) {
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@241
PS4, Line 241: hasColumnSelectPriv = true;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@242
PS4, Line 242: continue;
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/14203/4/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@243
PS4, Line 243: }
tab used for whitespace
--
To view, visit http://gerrit.cloudera.org:8080/14203
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da
Gerrit-Change-Number: 14203
Gerrit-PatchSet: 4
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 10 Sep 2019 22:47:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14203 to look at the new patch set (#4). Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java 1 file changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/4 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 4 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. Patch Set 12: fixed clang-tidy error -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 12 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 10 Sep 2019 22:39:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#12). Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. IMPALA-8858: Add metrics tracking num queries running on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors in it. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 13 files changed, 345 insertions(+), 168 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/12 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 12 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13786 )
Change subject: IMPALA-7322: Add storage wait time to profile
..
Patch Set 3: Code-Review+1
(1 comment)
Just a comment on the commit message.
I'm not super familiar with this code, so would be good if @Bharath could take
a look as well
http://gerrit.cloudera.org:8080/#/c/13786/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13786/3//COMMIT_MSG@26
PS3, Line 26: Sample output:
: Profile for Catalog V1: (storage-load-time is the added property
and
: it is part of Metadata load in Query Compilation):
: After ran a hbase query (Metadata load finished is divided into
: several lines because of limitation of commit message):
: Query Compilation: 4s401ms
: - Metadata load started: 661.084us (661.084us)
: - Metadata load finished. loaded-tables=1/1
: load-requests=1 catalog-updates=3
: storage-load-time=233ms: 3s819ms (3s819ms)
: - Analysis finished: 3s820ms (763.979us)
: - Value transfer graph computed: 3s820ms (63.193us)
: Profile for Catalog V2: (StorageLoad.Time is the added property
and it
: is in CatalogFetch):
: Frontend:
:- CatalogFetch.ColumnStats.Misses: 1
:- CatalogFetch.ColumnStats.Requests: 1
:- CatalogFetch.ColumnStats.Time: 0
:- CatalogFetch.Config.Misses: 1
:- CatalogFetch.Config.Requests: 1
:- CatalogFetch.Config.Time: 3ms
:- CatalogFetch.DatabaseList.Hits: 1
:- CatalogFetch.DatabaseList.Requests: 1
:- CatalogFetch.DatabaseList.Time: 0
:- CatalogFetch.PartitionLists.Misses: 1
:- CatalogFetch.PartitionLists.Requests: 1
:- CatalogFetch.PartitionLists.Time: 4ms
:- CatalogFetch.Partitions.Hits: 2
:- CatalogFetch.Partitions.Misses: 1
:- CatalogFetch.Partitions.Requests: 3
:- CatalogFetch.Partitions.Time: 1ms
:- CatalogFetch.RPCs.Bytes: 1.01 KB (1036)
:- CatalogFetch.RPCs.Requests: 4
:- CatalogFetch.RPCs.Time: 93ms
:- CatalogFetch.StorageLoad.Time: 68ms
:- CatalogFetch.TableNames.Hits: 2
:- CatalogFetch.TableNames.Requests: 2
:- CatalogFetch.TableNames.Time: 0
:- CatalogFetch.Tables.Misses: 1
:- CatalogFetch.Tables.Requests: 1
:- CatalogFetch.Tables.Time: 91ms
: Catalog metrics(this sample is from a hdfs table):
: storage-metadata-load-duration:
:Count: 1
:Mean rate: 0.0085
:1 min. rate: 0.032
:5 min. rate: 0.1386
:15 min. rate: 0.177
:Min (msec): 111
:Max (msec): 111
:Mean (msec): 111.1802
:Median (msec): 111.1802
:75th-% (msec): 111.1802
:95th-% (msec): 111.1802
:99th-% (msec): 111.1802
I still find this hard to understand, would recommend re-formatting as follows:
Sample Output:
Profile for Catalog V1 ('storage-load-time' is the added
property and it is part of the 'Metadata load finished' section of the
profile under the 'Query Compilation' section. The example below was
taken after running a HBase query. 'Metadata load finished' is divided
into several lines because of limitation of commit message):
Query Compilation: 4s401ms
- Metadata load started: 661.084us (661.084us)
- Metadata load finished. loaded-tables=1/1
load-requests=1 catalog-updates=3
storage-load-time=233ms: 3s819ms (3s819ms)
- Analysis finished: 3s820ms (763.979us)
- Value transfer graph computed: 3s820ms (63.193us)
Profile for Catalog V2 ('StorageLoad.Time' is the added property and it
is part of the 'CatalogFetch' counters):
Frontend:
- CatalogFetch.ColumnStats.Misses: 1
- CatalogFetch.ColumnStats.Requests: 1
- CatalogFetch.ColumnStats.Time: 0
- CatalogFetch.Config.Misses: 1
- CatalogFetch.Config.Requests: 1
- CatalogFetch.Config.Time: 3ms
- CatalogFetch.DatabaseList.Hits: 1
- CatalogFetch.DatabaseList.Requests: 1
- CatalogFetch.DatabaseList.Time: 0
- CatalogFetch.PartitionLists.Misses: 1
- CatalogFetch.PartitionLists.Requests: 1
- CatalogFetch.PartitionLists.Time: 4ms
- CatalogFetch.Partitions.Hits: 2
-
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4525/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 10 Sep 2019 22:33:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14180/2/be/src/exec/plan-root-sink.h File be/src/exec/plan-root-sink.h: http://gerrit.cloudera.org:8080/#/c/14180/2/be/src/exec/plan-root-sink.h@111 PS2, Line 111: > line has trailing whitespace Done -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 22:18:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14180 to look at the new patch set (#3). Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. IMPALA-8825: Add additional counters to PlanRootSink Adds the counters RowsSent and RowsSentRate to the PLAN_ROOT_SINK section of the profile: PLAN_ROOT_SINK: - PeakMemoryUsage: 4.01 MB (4202496) - RowBatchGetWaitTime: 0.000ns - RowBatchSendWaitTime: 0.000ns - RowsSent: 10 (10) - RowsSentRate: 416.00 /sec RowsSent tracks the number of rows sent to the PlanRootSink via PlanRootSink::Send. RowsSentRate tracks the rate that rows are sent to the PlanRootSink. Adds the counters NumRowsFetched, RowMaterializationRate, and RowsMaterialized to the ImpalaServer section of the profile. ImpalaServer: - ClientFetchWaitTimer: 11.999ms - NumRowsFetched: 10 (10) - RowMaterializationRate: 9.00 /sec - RowMaterializationTimer: 1s007ms - RowsMaterialized: 10 (10) NumRowsFetched tracks the total number of rows fetched by the query, including rows fetched from the cache. RowMaterializationRate tracks the rate at which rows are materialized and RowsMaterialized tracks the number of rows materialized (RowMaterializationTimer already existed and tracks how much time is spent materializing rows). NumRowsFetched and RowsMaterialized track similar but distinct values. NumRowsFetched tracks the number of rows read, including any rows read from the query results cache, whereas RowsMaterialized does not track rows read from the query results cache. NumRowsFetched will be greater than RowsMaterialized if the read cursor is reset (e.g TCLIService.TFetchOrientation.FETCH_FIRST is used). Testing: * Added tests to test_fetch_first.py and query_test/test_fetch.py * Enabled some tests in test_fetch_first.py that were pending the completion of IMPALA-8819 * Ran core tests Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/custom_cluster/test_result_spooling.py M tests/hs2/test_fetch_first.py A tests/query_test/test_fetch.py 10 files changed, 200 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14180/3 -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14180/2/be/src/exec/plan-root-sink.h File be/src/exec/plan-root-sink.h: http://gerrit.cloudera.org:8080/#/c/14180/2/be/src/exec/plan-root-sink.h@111 PS2, Line 111: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 22:17:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14180 )
Change subject: IMPALA-8825: Add additional counters to PlanRootSink
..
Patch Set 2:
(2 comments)
Rebased and fixed the test in tests/custom_cluster/test_result_spooling.py
http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/exec/plan-root-sink.cc
File be/src/exec/plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/exec/plan-root-sink.cc@48
PS1, Line 48: rows_sent_rate_ = profile_->AddDerivedCounter("RowsSentRate",
TUnit::UNIT_PER_SECOND,
> nit: indent seems off
yeah i though that too, but according to ClangFormat that is how it should be.
http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h@421
PS1, Line 421: /// The number of rows materialized for this query, the
counter is not reset if the
: /// fetch is restarted. It does not include any rows read from
the results cache, since
: /// those rows are already materialized.
: RuntimeProfile::Counter* rows_materialized_counter_ = nullptr;
> Doesn't this track closely with num_rows_fetched_counter_ ? Would it be suf
yeah, this is pretty similar to num_rows_fetched_counter_ the difference is
that num_rows_fetched_counter_ counts the number of rows read from the query
results cache whereas rows_materialized_counter_ does not. I've updated the
commit message to make this a bit clearer.
--
To view, visit http://gerrit.cloudera.org:8080/14180
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036
Gerrit-Change-Number: 14180
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 10 Sep 2019 22:17:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14180 to look at the new patch set (#2). Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. IMPALA-8825: Add additional counters to PlanRootSink Adds the counters RowsSent and RowsSentRate to the PLAN_ROOT_SINK section of the profile: PLAN_ROOT_SINK: - PeakMemoryUsage: 4.01 MB (4202496) - RowBatchGetWaitTime: 0.000ns - RowBatchSendWaitTime: 0.000ns - RowsSent: 10 (10) - RowsSentRate: 416.00 /sec RowsSent tracks the number of rows sent to the PlanRootSink via PlanRootSink::Send. RowsSentRate tracks the rate that rows are sent to the PlanRootSink. Adds the counters NumRowsFetched, RowMaterializationRate, and RowsMaterialized to the ImpalaServer section of the profile. ImpalaServer: - ClientFetchWaitTimer: 11.999ms - NumRowsFetched: 10 (10) - RowMaterializationRate: 9.00 /sec - RowMaterializationTimer: 1s007ms - RowsMaterialized: 10 (10) NumRowsFetched tracks the total number of rows fetched by the query, including rows fetched from the cache. RowMaterializationRate tracks the rate at which rows are materialized and RowsMaterialized tracks the number of rows materialized (RowMaterializationTimer already existed and tracks how much time is spent materializing rows). NumRowsFetched and RowsMaterialized track similar but distinct values. NumRowsFetched tracks the number of rows read, including any rows read from the query results cache, whereas RowsMaterialized does not track rows read from the query results cache. NumRowsFetched will be greater than RowsMaterialized if the read cursor is reset (e.g TCLIService.TFetchOrientation.FETCH_FIRST is used). Testing: * Added tests to test_fetch_first.py and query_test/test_fetch.py * Enabled some tests in test_fetch_first.py that were pending the completion of IMPALA-8819 * Ran core tests Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/custom_cluster/test_result_spooling.py M tests/hs2/test_fetch_first.py A tests/query_test/test_fetch.py 10 files changed, 200 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14180/2 -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/14103/11/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/14103/11/be/src/scheduling/admission-controller.h@a834 PS11, Line 834: : : : : : : : : : : : : : : rebased and cleaned up some code around these methods -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 10 Sep 2019 21:53:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8897 (part 3): Fix some additional javascript urls
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14206 ) Change subject: IMPALA-8897 (part 3): Fix some additional javascript urls .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Gerrit-Change-Number: 14206 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 21:52:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#11). Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. IMPALA-8858: Add metrics tracking num queries running on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors in it. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 13 files changed, 345 insertions(+), 167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/11 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 11 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 10: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4923/ -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 21:37:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4524/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 10 Sep 2019 20:48:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8897 (part 3): Fix some additional javascript urls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14206 ) Change subject: IMPALA-8897 (part 3): Fix some additional javascript urls .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4523/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Gerrit-Change-Number: 14206 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 20:44:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#10). Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. IMPALA-8858: Add metrics tracking num queries running on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors in it. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 12 files changed, 317 insertions(+), 132 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/10 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8858: Add metrics tracking num queries running on executor groups
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add metrics tracking num queries running on executor groups .. Patch Set 10: (6 comments) http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/runtime/exec-env.cc@161 PS9, Line 161: t : sna > nit: hand off. How about SendClusterMembershipToFrontend()? Done http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/runtime/exec-env.cc@177 PS9, Line 177: > move the helper into an anonymous namespace above the impala namespace Done http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/scheduling/admission-controller.h@951 PS9, Line 951: const string& g > Pass by const ref, looks like the implementation doesn't need a copy Done http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/scheduling/admission-controller.cc@1871 PS9, Line 1871: // There might be lingering queries from when this group was active. > Maybe this code could go into a helper, e.g. MaxQueriesOnExecutorGroupHosts good point, however I'll keep it for now to avoid passing around ClusterMembershipMgr::* typedefs http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/scheduling/admission-controller.cc@1873 PS9, Line 1873: auto new_grp_it = snapshot->executor_groups.find(new_grp); > usually we name iterators 'it', but since this is the second one in this me Done http://gerrit.cloudera.org:8080/#/c/14103/9/be/src/scheduling/admission-controller.cc@1877 PS9, Line 1877: const string& host = TNetworkAddressToString(be_desc.address); > const string& Done -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 10 Sep 2019 20:07:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8897 (part 3): Fix some additional javascript urls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14206 ) Change subject: IMPALA-8897 (part 3): Fix some additional javascript urls .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4929/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Gerrit-Change-Number: 14206 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 20:03:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8897 (part 3): Fix some additional javascript urls
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14206 Change subject: IMPALA-8897 (part 3): Fix some additional javascript urls .. IMPALA-8897 (part 3): Fix some additional javascript urls Part 2 of this patch missed rewriting two urls in javascript. This patch fixes it by adding 'make_url()' around those urls. Testing: - Deployed to a real cluster and verified that Knox integreation works correctly on the relevant pages now. - Extended test_knox_integration to check for this case. Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 --- M tests/webserver/test_web_pages.py M www/query_backends.tmpl M www/query_finstances.tmpl 3 files changed, 6 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14206/1 -- To view, visit http://gerrit.cloudera.org:8080/14206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I07a099b3a23324ad56133c5584d6a9bc88541582 Gerrit-Change-Number: 14206 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8718: project out collection slots in analytic's sort tuple
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14135 ) Change subject: IMPALA-8718: project out collection slots in analytic's sort tuple .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14135 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7edf74ff0f603dfd33ff546e61545bc724990655 Gerrit-Change-Number: 14135 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 10 Sep 2019 19:56:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 3: Bharath covered my comments. Clang-format might help formatting issues (it works for java as well as C++). We generally default to accepting its decisions unless they are obviously bad just to minimise the scope of code formatting debates. We have the instructions towards the bottom here https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868536 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 3 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 19:55:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1071: Distributable python package for impala-shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14181 ) Change subject: IMPALA-1071: Distributable python package for impala-shell .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib8c745bf6a16f0c039430152745a2f00e044 Gerrit-Change-Number: 14181 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Dinesh Garg (430) Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 19:50:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14197 ) Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4521/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 19:35:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1071: Distributable python package for impala-shell
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/14181 ) Change subject: IMPALA-1071: Distributable python package for impala-shell .. Patch Set 4: (2 comments) > Patch Set 3: Code-Review+1 > > (2 comments) > > Thanks for making this! Gladly! Hopefully customers find it useful. In patch 4, I also added a restriction on the python version in setup.py. $ pip install dist/impala_shell-3.3.0.dev20190910115339.tar.gz Processing ./dist/impala_shell-3.3.0.dev20190910115339.tar.gz ERROR: Package 'impala-shell' requires a different Python: 3.5.2 not in '<3.0.0' http://gerrit.cloudera.org:8080/#/c/14181/3/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/14181/3/shell/impala_client.py@1120 PS3, Line 1120: th > nit: typo Done http://gerrit.cloudera.org:8080/#/c/14181/3/shell/packaging/make_python_package.sh File shell/packaging/make_python_package.sh: http://gerrit.cloudera.org:8080/#/c/14181/3/shell/packaging/make_python_package.sh@37 PS3, Line 37: c > nit: remove space Done -- To view, visit http://gerrit.cloudera.org:8080/14181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib8c745bf6a16f0c039430152745a2f00e044 Gerrit-Change-Number: 14181 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Dinesh Garg (430) Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 19:05:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1071: Distributable python package for impala-shell
Hello Bharath Vissapragada, Greg Rahn, Lars Volker, David Rorke, Tim Armstrong, Dinesh Garg, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14181 to look at the new patch set (#4). Change subject: IMPALA-1071: Distributable python package for impala-shell .. IMPALA-1071: Distributable python package for impala-shell The patch adds a set of scripts for converting the impala-shell into a true distributable python package. The package can be installed using familiar python commands, e.g.: $ python setup.py (install|develop) or $ pip install -e /path/to/dist/dir The entry point script, make_python_package.sh, will run as a part of the standard sequence of steps that results from calling buildall.sh, and will produce a gzipped tarball inside of Impala/shell/dist as an artifact. Thereafter, make_python_package.sh can be run manually any time. The expectation is that an official maintainer would need to manually upload official releases to the Python Package Index as appropriate. Change-Id: Ib8c745bf6a16f0c039430152745a2f00e044 --- M bin/rat_exclude_files.txt M shell/impala_client.py M shell/make_shell_tarball.sh A shell/packaging/MANIFEST.in A shell/packaging/README.md A shell/packaging/__init__.py A shell/packaging/make_python_package.sh A shell/packaging/requirements.txt A shell/packaging/setup.py 9 files changed, 398 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/14181/4 -- To view, visit http://gerrit.cloudera.org:8080/14181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib8c745bf6a16f0c039430152745a2f00e044 Gerrit-Change-Number: 14181 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Dinesh Garg (430) Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14197 ) Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14197/4/be/src/exec/kudu-util.cc File be/src/exec/kudu-util.cc: http://gerrit.cloudera.org:8080/#/c/14197/4/be/src/exec/kudu-util.cc@242 PS4, Line 242: case DataType::VARCHAR: return ColumnType::CreateVarcharType(type_attributes.length()); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 18:55:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14197 ) Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4928/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 18:55:08 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14197 to look at the new patch set (#4). Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. [WIP] Add POC Kudu CHAR/VARCHAR Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 --- M be/src/exec/kudu-util.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test 7 files changed, 74 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14197/4 -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/14203/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14203/2//COMMIT_MSG@9 PS2, Line 9: This patch fixes a case where access to a given column is allowed at the table nit: Limit commit message line widths to 72 chars. http://gerrit.cloudera.org:8080/#/c/14203/2//COMMIT_MSG@16 PS2, Line 16: - Manually tested with table level allow and column level deny policies in No luck with automating it in AuthorizationStmtTest? http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@234 PS2, Line 234: Trailing white space http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@235 PS2, Line 235: if (hasTableSelectPriv && How about ALTER privilege? Alter table drop/rename col etc? In-general, shouldn't we just skip 'hasTableSelectPriv' for col privileges and let Ranger/Sentry do its check via hasAccess()? http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@236 PS2, Line 236: request.getPrivilege() != Privilege.SELECT && 4 spaced indents.https://cwiki.apache.org/confluence/display/IMPALA/Impala+Style+Guide -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 18:50:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Hello Bharath Vissapragada, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14203 to look at the new patch set (#3). Change subject: IMPALA-8933: Enforce ranger deny policy .. IMPALA-8933: Enforce ranger deny policy This patch fixes a case where access to a given column is allowed at the table level by a ranger policy and denied at the column level by a second ranger policy. The code previously skipped evaluating column level policies when a table level policy allowed access but that optimization can only be applied when the column level policy does not deny access. Testing: - Manually tested with table level allow and column level deny policies in ranger - Ran ranger-specific authorization funcional and unit tests Steps to Repro: Connect impala-shell as admin: CREATE table(c1 int, c2 int); INSERT INTO T1 VALUES(1,1); In Ranger: Add policies: 1) Name t1allow, Database *, Table t1, Allow conditions user: , Permissions: select 2) Name t1deny, Database *, Table t1, Deny conditions user: , Permissions: select Connect impala-shell as : SELECT c1 from t1; -- Not allowed SELECT c2 from t1; -- Allowed Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da --- M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14203/3 -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 3 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 1: May also want to rebase and address the TODO (i.e. removing sleep(1)) in https://github.com/apache/impala/blob/master/tests/custom_cluster/test_result_spooling.py#L64-L66 -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 18:02:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8825: Add additional counters to PlanRootSink
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14180 ) Change subject: IMPALA-8825: Add additional counters to PlanRootSink .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/exec/plan-root-sink.cc File be/src/exec/plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/exec/plan-root-sink.cc@48 PS1, Line 48: profile_->total_time_counter())); nit: indent seems off http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14180/1/be/src/service/client-request-state.h@421 PS1, Line 421: /// The number of rows materialized for this query, the counter is not reset if the : /// fetch is restarted. It does not include any rows read from the results cache, since : /// those rows are already materialized. : RuntimeProfile::Counter* rows_materialized_counter_ = nullptr; Doesn't this track closely with num_rows_fetched_counter_ ? Would it be sufficient just to track the row materialization time and num rows fetched only ? -- To view, visit http://gerrit.cloudera.org:8080/14180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9e101e2f3e2bf8324e149c780d35825ceecc036 Gerrit-Change-Number: 14180 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 17:57:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4923/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 17:26:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 21: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 21 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 17:03:33 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14197 ) Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. Patch Set 3: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4921/ -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 3 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 16:28:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 21: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4922/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 21 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 12:56:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Add POC Kudu CHAR/VARCHAR
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14197 ) Change subject: [WIP] Add POC Kudu CHAR/VARCHAR .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4921/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d4959410fdd882bfa980cb55e8a7837c7823da8 Gerrit-Change-Number: 14197 Gerrit-PatchSet: 3 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 12:44:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4520/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 21 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 12:33:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 21: PS21 is a rebase with master -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 21 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 10 Sep 2019 11:54:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Hello Attila Jeges, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13722
to look at the new patch set (#21).
Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
..
IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
This enhancement introduces FORMAT clause for CAST() operator that is
applicable for casts between string types and timestamp types. Instead
of accepting SimpleDateFormat patterns the FORMAT clause supports
datetime patterns following the ISO:SQL:2016 standard.
Note, the CAST() operator without the FORMAT clause still uses
Impala's implementation of SimpleDateFormat handling. Similarly, the
existing conversion functions such as to_timestamp(), from_timestamp()
etc. remain unchanged and use SimpleDateFormat. Contrary to how these
functions work the FORMAT clause must specify a string literal and
cannot be used with any other kind of a string expression.
Milestone 1 contains all the format tokens covered by the SQL
standard. Further milestones will add more functionality on top of
this list to cover functionality provided by other RDBMS systems.
List of tokens implemented by this change:
- , YYY, YY, Y: Year tokens
- , RR: Round year tokens
- MM: Month (1-12)
- DD: Day (1-31)
- DDD: Day of year (1-366)
- HH, HH12: Hour of day (1-12)
- HH24: Hour of day (0-23)
- MI: Minute (0-59)
- SS: Second (0-59)
- S: Second of day (0-86399)
- FF, FF1, ..., FF9: Fractional second
- AM, PM, A.M., P.M.: Meridiem indicators
- TZH: Timezone hour (-99-+99)
- TZM: Timezone minute (0-99)
- Separators: - . / , ' ; : space
- ISO8601 date indicators (T, Z)
Some notes about the matching algorithm:
- The parsing algorithm uses these tokens in a case insensitive
manner.
- The separators are interchangeable with each other. For example a
'-' separator in the format will match with a '.' character in the
input.
- The length of the separator sequences is handled flexibly meaning
that a single separator character in the format for instance would
match with a multi-separator sequence in the input.
- In a string type to timestamp conversion the timezone offset tokens
are parsed, expected to match with the input but they don't adjust
the result as the input is already expected to be in UTC format.
Usage example:
SELECT CAST('01-02-2019' AS TIMESTAMP FORMAT 'MM-DD-');
SELECT CAST('2019.10.10 13:30:40.123456 +01:30' AS TIMESTAMP
FORMAT '-MM-DD HH24:MI:SS.FF9 TZH:TZM');
SELECT CAST(timestamp_column as STRING
FORMAT " MM HH12 YY") from some_table;
Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23
---
M be/src/benchmarks/convert-timestamp-benchmark.cc
M be/src/benchmarks/parse-timestamp-benchmark.cc
M be/src/common/init.cc
M be/src/exec/text-converter.inline.h
M be/src/exprs/CMakeLists.txt
A be/src/exprs/cast-format-expr.cc
A be/src/exprs/cast-format-expr.h
M be/src/exprs/cast-functions-ir.cc
M be/src/exprs/date-functions-ir.cc
M be/src/exprs/expr-test.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-expr.h
M be/src/exprs/timestamp-functions-ir.cc
M be/src/exprs/timestamp-functions.cc
M be/src/exprs/timestamp-functions.h
M be/src/runtime/CMakeLists.txt
M be/src/runtime/date-parse-util.cc
M be/src/runtime/date-parse-util.h
M be/src/runtime/date-test.cc
M be/src/runtime/date-value.cc
M be/src/runtime/date-value.h
A be/src/runtime/datetime-iso-sql-format-parser.cc
A be/src/runtime/datetime-iso-sql-format-parser.h
A be/src/runtime/datetime-iso-sql-format-tokenizer.cc
A be/src/runtime/datetime-iso-sql-format-tokenizer.h
D be/src/runtime/datetime-parse-util.h
A be/src/runtime/datetime-parser-common.cc
A be/src/runtime/datetime-parser-common.h
R be/src/runtime/datetime-simple-date-format-parser.cc
A be/src/runtime/datetime-simple-date-format-parser.h
M be/src/runtime/runtime-state.cc
M be/src/runtime/timestamp-parse-util.cc
M be/src/runtime/timestamp-parse-util.h
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.cc
M be/src/runtime/timestamp-value.h
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/testutil/random-vector-generators.h
M be/src/util/dict-test.cc
M be/src/util/min-max-filter-test.cc
M be/src/util/string-parser.h
M common/thrift/Exprs.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CastExpr.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
A
testdata/workloads/functional-query/queries/QueryTest/cast_format_from_table.test
M testdata/workloads/functional-query/queries/QueryTest/date.test
A tests/query_test/test_cast_with_format.py
54 files changed, 3,530 insertions(+), 964 deletions(-)
git
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4920/ -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 11:21:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8917: Remove hostname from webui links if Knox isn't being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14199 ) Change subject: IMPALA-8917: Remove hostname from webui links if Knox isn't being used .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14199 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf77058dc6ce1d72422a9e3ca7868cdffacff76 Gerrit-Change-Number: 14199 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 09:25:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8917: Remove hostname from webui links if Knox isn't being used
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14199 ) Change subject: IMPALA-8917: Remove hostname from webui links if Knox isn't being used .. IMPALA-8917: Remove hostname from webui links if Knox isn't being used IMPALA-8897 added the hostname to all links on the debug webui in order to facilitate proxying connections through Apache Knox. This makes the webui difficult to use in situations where the hostname is not DNS-resolvable. This patch fixes this by only including the hostname with links if Knox proxying is actually being used, which we determine by looking for the 'x-forwarded-context' header in the request, which Knox adds to all requests. It also removes the hidden form fields that were added to support Knox integration when not being accessed through Knox. It also adds a class comment on Webserver explaining the requirements for keeping the webui compatible with Knox. Testing: - Added a test that checks that links on the webui are made absolute when the 'x-forwarded-context' header is in the request. Change-Id: Ifcf77058dc6ce1d72422a9e3ca7868cdffacff76 Reviewed-on: http://gerrit.cloudera.org:8080/14199 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M be/src/util/webserver.cc M be/src/util/webserver.h M tests/webserver/test_web_pages.py M www/form-hidden-inputs.tmpl 4 files changed, 67 insertions(+), 26 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14199 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifcf77058dc6ce1d72422a9e3ca7868cdffacff76 Gerrit-Change-Number: 14199 Gerrit-PatchSet: 6 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13955 ) Change subject: IMPALA-8755: Frontend support for Z-ordering .. Patch Set 12: Code-Review+1 I'm fine with the changes but not competent enough to +2. Thanks for taking care of my comments! Just a general comment: If you upload a patch set please make sure that it contains either your code changes or a rebase but not both. This makes it easier for reviewers to check the diffs between patch sets. -- To view, visit http://gerrit.cloudera.org:8080/13955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d Gerrit-Change-Number: 13955 Gerrit-PatchSet: 12 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Sep 2019 09:04:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4519/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 07:50:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4920/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 07:09:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#10). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,009 insertions(+), 243 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/10 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8586: Support download URLs for CDP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13432 ) Change subject: IMPALA-8586: Support download URLs for CDP .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67824fd82b820e68e9f5c87939ec94ca6abadb8c Gerrit-Change-Number: 13432 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 10 Sep 2019 06:32:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8917: Remove hostname from webui links if Knox isn't being used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14199 ) Change subject: IMPALA-8917: Remove hostname from webui links if Knox isn't being used .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4518/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14199 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcf77058dc6ce1d72422a9e3ca7868cdffacff76 Gerrit-Change-Number: 14199 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 06:00:09 + Gerrit-HasComments: No
