Hello Fang-Yu Rao, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20742
to look at the new patch set (#2).
Change subject: IMPALA-11501: Add flag to allow catalog-cache operations on
masked tables
......................................................................
IMPALA-11501: Add flag to allow catalog-cache operations on masked tables
REFRESH/INVALIDATE METADATA <table> are the table level catalog-cache
operations. In Hive-Ranger plugin, when a table is masked (either by
column-masking or row-filtering policy) for a user, the user is unable
to perform any modification (insert/delete/update) on the table, i.e.
it's considered a read-only user (RANGER-1087, RANGER-1100). However,
Hive doesn't have these catalog-cache operations. It's a grey area
whether they should be blocked.
Before this patch, these catalog-cache operations are considered as
modifications on the table so they are also blocked for masked users.
Table metadata is required to be loaded so we have the column names to
fetch Ranger column masking policies. This causes a performance
regression on INVALIDATE METADATA <table> commands since in older
versions (e.g. CDH), IM commands don't need to load the table metadata
and runs pretty fast.
This patch adds a flag, allow_catalog_cache_op_from_masked_users, for
coordinators to skip checking masking policies for such statements. When
this flag is on, coordinators don't need to load the table metadata thus
fix the performance regression as well.
Note that Ranger ownership based policies can't be applied correctly
when the table is unloaded (so the owner is unknown). REFRESH/INVALIDATE
METADATA <table> commands could be denied on owners even if there are
Ranger policies allowing the owner's operations. This is a known issue
since IMPALA-8228. To ensure a user can always perform these operations,
grant REFRESH privilege to them to get rid of the unloaded table issue.
This patch also fixes a bug in local catalog mode which only occurs
after adding the new flag. The bug is that LocalDb#getTableIfCached()
doesn't make good use of the cache. If the table meta is cahced but
LocalDb#getTable() hasn't been invoked on the table, getTableIfCached()
will always return a LocalIncompleteTable which is missing some table
info, e.g. ownership. This causes REFRESH/INVALIDATE statements not able
to pass the ownership context to RangerAccessResourceImpl so ownership
policies can't be correctly applied.
Ideally, LocalDb#getTableIfCached() should return a LocalTable instance
if the table is cached. However, in local catalog mode, we don't cache
everything that constructs a LocalTable instance. Constructing a
LocalTable instance might still trigger external RPCs which should be
avoided. As an alternative, this patch checks if the msTable is cached.
If it's cached, add it to the LocalIncompleteTable instance so most of
the table info can be retrieved, including the ownership string.
Tests:
- Add e2e tests on both the legacy and local catalog mode.
Change-Id: I45935654cbf05a55d740f1b04781022c271f7678
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/Privilege.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/authorization/test_ranger.py
15 files changed, 181 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/20742/2
--
To view, visit http://gerrit.cloudera.org:8080/20742
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I45935654cbf05a55d740f1b04781022c271f7678
Gerrit-Change-Number: 20742
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
