Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11358
to look at the new patch set (#4).
Change subject: IMPALA-7510. Support principals/privileges with LocalCatalog
......................................................................
IMPALA-7510. Support principals/privileges with LocalCatalog
This enables support for Sentry authorization when LocalCatalog is
enabled. The design is detailed in a change to the comment on
CatalogdMetaProvider, but to recap it briefly here:
At a high level, this patch takes the approach of duplicating the "v1"
catalog flow for PRINCIPAL and PRIVILEGE catalog objects. Namely, the catalog
daemon publishes complete objects into the statestore topic, and the
impalad fully replicates them locally.
I took this approach rather than trying to do fine-grained caching and
invalidation for the following reasons:
- The PRINCIPAL and PRIVILEGE metadata is typically many orders of magnitude
smaller than table metadata. So, the benefit of fine-grained caching
and eviction is not as great.
- The PRINCIPAL and PRIVILEGE catalog objects are fairly tightly intertwined
with relationships between them and backwards mappings maintained from
groups back to principals. This logic is implemented by the
AuthorizationPolicy class. Implementing similar mapping in a
fine-grained caching approach would be a reasonable amount of work.
- This bit of code is under some current flux as others are working on
implementing more fine grained permissioning. Thus, trying to
duplicate the logic in a "fetch-on-demand" implementation might turn
out to be chasing somewhat of a moving target.
In order to take this approach, the patch is organized as follows:
- refactored some of the role/principal removal logic from ImpaladCatalog
into AuthorizationPolicy. This makes it easier to perform the similar
"subscribe" with less duplicate cdoe.
- changed catalogd to publish PRINCIPAL and PRIVILEGE objects to v2
catalogs in addition to v1.
- passed through LocalCatalog.getAuthPolicy to CatalogdMetaProvider, and
added an AuthorizationPolicy member there. This member is maintained
when we see PRINCIPAL and PRIVILEGE objects come via the catalog
updates.
- had to implement LocalCatalog.isReady() to ensure that we don't allow
user access until the first topic update has been consumed.
- additionally had to copy some other code from ImpaladCatalog to
protect against various races -- we need a CatalogDeltaLog as well as
careful sequencing of the order in which the objects apply.
With this patch and the following one to enable UDF support, I was able
to run the tests in tests/authorization successfully with LocalCatalog
enabled.
Change-Id: Iccce5aabdb6afe466fdaeae0fb3700c66e658558
---
M fe/src/main/java/org/apache/impala/catalog/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M tests/common/custom_cluster_test_suite.py
9 files changed, 266 insertions(+), 67 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/11358/4
--
To view, visit http://gerrit.cloudera.org:8080/11358
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iccce5aabdb6afe466fdaeae0fb3700c66e658558
Gerrit-Change-Number: 11358
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fwij...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>
