> On Jan. 4, 2018, 10:40 p.m., Vadim Spector wrote: > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java > > Line 205 (original), 206 (patched) > > <https://reviews.apache.org/r/64955/diff/1/?file=1930844#file1930844line207> > > > > I thought we've agreed that at this point we shall pass to > > notificationFetcher.fetchNotification() notificationId-delta, where delta > > is configurable and probably set to some significant number, like 10. > > Otherwise we'll be loosing lots of older notifications with smaller IDs > > that got persisted since the previous fetch, due to the HMS issue. > > kalyan kumar kalvagadda wrote: > we can handle that as seperate issue. we need this change to avaoid > notificated getting missed. Scope of this patch is just to avoid full > snapshots un-necessarily. > > Vadim Spector wrote: > Hmm, I'd like other peoples' take on this. These two issues are not > independent imo. Avoiding full snapshots is good, but full snapshots, even if > done for a wrong reason, would offset for missing updates, at least to some > extent. Now we may start having quite a few of them. > > It's a few line fix, really, just define some "delta" config parameter, > set its default to some value, could be fairly small, like 2, and pass > "notificationId-delta" instead; I'd rather just take care of it now. Based on > what we know of HMS ID generation problem, I'd feel really uncomfortable > releasing one fix w/o another.
Sounds reasonable. Will update the patch with this behavior. - kalyan kumar ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64955/#review194788 ----------------------------------------------------------- On Jan. 5, 2018, 12:10 a.m., kalyan kumar kalvagadda wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64955/ > ----------------------------------------------------------- > > (Updated Jan. 5, 2018, 12:10 a.m.) > > > Review request for sentry, Alexander Kolbasov, Arjun Mishra, Na Li, Sergio > Pena, and Vadim Spector. > > > Bugs: SENTRY-2109 > https://issues.apache.org/jira/browse/SENTRY-2109 > > > Repository: sentry > > > Description > ------- > > This patch does couple of things > 1. Avoid triggering full snapshots when gaps are observed while fetching new > notifications. While fetching new notifications HMSFollower would would fetch > notifications with last event-id as well. When it gets the notifications and > if it doesn't get the notifications with event-id, full snpshot is triggered. > 2. Added looging to report duplicate events and out of order events for debug > purpose. > 3. Added new e2e tests to verfy this behavior. > > > Diffs > ----- > > > sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/exception/SentryOutOfSyncException.java > PRE-CREATION > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java > 6c4631fa74760d8721b5740dd3dffb2c1d8e72e6 > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java > aa1b6a31c28f35af86952c213d5e20a8c9bb3490 > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java > 097aa62912e92ece7f8da0ac0fccb124579a88f2 > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java > 43535a7b50fea51049f3142837736e6a99a3a80f > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java > 501898bca261db2daf937a8d803d12a59616192b > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStore.java > b4100278392986c161625a366212c6fef66ec0a9 > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHMSFollower.java > edde886a7646539499149f2d86758979436567bd > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreation.java > PRE-CREATION > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreationWithShorterHMSEventTtl.java > PRE-CREATION > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotWithLongerHMSFollowerLongerInterval.java > PRE-CREATION > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java > c8eef09ed61e11583838a3dfd3117a9478df23ba > > > Diff: https://reviews.apache.org/r/64955/diff/2/ > > > Testing > ------- > > Made sure that tests pass. There are three test failures which need code > change done for SENTRY-2113. > > > Thanks, > > kalyan kumar kalvagadda > >
