> On March 6, 2017, 10:38 p.m., Lei Xu wrote: > > sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/DeltaRetriever.java > > Lines 35 (patched) > > <https://reviews.apache.org/r/57232/diff/2/?file=1653451#file1653451line35> > > > > If we retrieve `Perm` and `Path` separately, would that be a case that > > the deltas from them are not consistent, because the `latestChangeID` might > > be different in them? > > > > For example, if we do the following sequence: > > > > * Retrieve path deltas. > > * Create table > > * Grant permissions on the table. > > * Retrieve perm deltas. > > > > In such order, would some perm deltas can not be applied to the path, > > as they do not exist on NN yet?
Perm and Path deltas do not share the same changeID, and cannot be correlated. They are triggered by two seperate streams of events. NN periodically pull these two kind of deltas every configurable of time(default is 0.5 seconds). During that time, as you described Namenode cache could be in inconsistent. But after that time it should be consistent. This problem will still exist even we put those two kinds of update in one. - Hao ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57232/#review168057 ----------------------------------------------------------- On March 2, 2017, 7:39 a.m., Hao Hao wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57232/ > ----------------------------------------------------------- > > (Updated March 2, 2017, 7:39 a.m.) > > > Review request for sentry. > > > Repository: sentry > > > Description > ------- > > Add the logic for propagating Sentry Permissions or Sentry representation of > HMS Paths, from a persistent storage instead of in memoery cache of Sentry > service, to a Sentry consumer such as HDFS NameNode. It includes: > 1. Delta update retrieval logic from persistent storage. > 2. Propagation logic for consumer to get the delta update or a complete > snapshot. > > > Diffs > ----- > > > sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/DeltaRetriever.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/ImageRetriever.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-common/src/main/java/org/apache/sentry/hdfs/ThriftSerializer.java > 69aa098bc71362b1aba0ad219483cfce7d389964 > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/DBUpdateForwarder.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/MetastorePlugin.java > 085971b34e3901b7a1d59bd8e7516b25f81ca872 > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PathDeltaRetriever.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PathImageRetriever.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PermDeltaRetriever.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PermImageRetriever.java > PRE-CREATION > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryHDFSServiceProcessor.java > e4f3f580ee0a6bf83d5a32001bff130c0bff13aa > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryHdfsMetricsUtil.java > e68c7080cec007e3cdf2c89efb9701eef9cca0a0 > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryPlugin.java > f68f690436301c79e93b0742996dda25a750c0c3 > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/UpdateForwarder.java > 6d5c607273bb08597780b655d7b59cd41f0844bb > > sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/UpdateablePermissions.java > fe2baa6a446874185e8344bb16d76d803826d1f3 > > sentry-hdfs/sentry-hdfs-service/src/test/java/org/apache/sentry/hdfs/TestUpdateForwarder.java > 0f0d0a743d5a5f9f1678ac7c5217b9a27537e85b > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java > c1186ba405a05be70336e9169a1454208df016ca > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStore.java > 91f15c0f3999a25a20dfb45f2f28abbda54366f4 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegration.java > f2d74bfe5e413ea88c3c77ef303f1c7f9ee16253 > > > Diff: https://reviews.apache.org/r/57232/diff/2/ > > > Testing > ------- > > > Thanks, > > Hao Hao > >
