-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59566/
-----------------------------------------------------------
(Updated June 1, 2017, 10:51 p.m.)
Review request for sentry, Alexander Kolbasov, kalyan kumar kalvagadda, Sergio
Pena, and Vamsee Yarlagadda.
Bugs: SENTRY-1784
https://issues.apache.org/jira/browse/SENTRY-1784
Repository: sentry
Description
-------
HDFS starts requesting changeID = 0 instead of 1. DBUpdateForwarder returns
full update if the request changeID <= 0. After first full update, the request
changeID = 1, so only delta update is sent unless clean up removes changes that
are not sent to HDFS. This fixes both issues in this Jira.
1) What happens when NN just starts
NN sends requestedId = 0 to sentry server. Sentry server sends full update
back. The latest changeID for perm and path are returned to NN. NN saves the
latest changeID for perm and path.
NN initial changeID for permission and path is -1. The requestedId is NN
changeID + 1. So initial requestedId from NN to sentry server is 0.
2) What happens once NN gets a full snapshot
Once NN gets a full snapshot, it creats new UpdateableAuthzPermissions for full
perm update; it creates new UpdateableAuthzPaths for full path update;
It saves the latest changeID for perm and path (referred as latestChangeID_path
and latestChangeID_perm). Next interval, NN sends requestedId
(latestChangeID_path + 1, latestChangeID_perm + 1) to Sentry server. Since the
lowest latestChangeID is 0, this requestedId >= 1.
If Sentry server has changes equal or newer than requestedId, it sends back
delta changes to NN. Otherwise, it sends back empty list to NN. The exceptions
are: 1) the requestedId does not exist in SentryStore (maybe it is cleaned up)
or 2) the returned delta change list from SentryStore is empty (maybe the
change table is corrupted). If exception happens, full snapshot is sent back to
NN.
3) What happens when a delta is received from HMS.
NN updates the privilege and path based on the delta changes. It update save
the latest changeID for perm and path.
Diffs (updated)
-----
sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/ServiceConstants.java
0741ebc
sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/UpdateableAuthzPaths.java
ad7f8c9
sentry-hdfs/sentry-hdfs-namenode-plugin/src/main/java/org/apache/sentry/hdfs/SentryAuthorizationInfo.java
90ba721
sentry-hdfs/sentry-hdfs-namenode-plugin/src/main/java/org/apache/sentry/hdfs/SentryUpdater.java
34caa0e
sentry-hdfs/sentry-hdfs-namenode-plugin/src/main/java/org/apache/sentry/hdfs/UpdateableAuthzPermissions.java
431c7fe
sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/DBUpdateForwarder.java
b8542b3
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java
9beb07b
Diff: https://reviews.apache.org/r/59566/diff/4/
Changes: https://reviews.apache.org/r/59566/diff/3-4/
Testing
-------
TestHDFSIntegrationEnd2End
Thanks,
Na Li
