-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64955/
-----------------------------------------------------------

(Updated March 13, 2018, 11:26 p.m.)


Review request for sentry, Alexander Kolbasov, Arjun Mishra, Na Li, Sergio 
Pena, and Vadim Spector.


Changes
-------

rebased the patch


Bugs: SENTRY-2109
    https://issues.apache.org/jira/browse/SENTRY-2109


Repository: sentry


Description
-------

This patch does couple of things
1. Avoid triggering full snapshots when gaps are observed while fetching new 
notifications. While fetching new notifications HMSFollower would would fetch 
notifications with last event-id as well. When it gets the notifications and if 
it doesn't get the notifications with event-id, full snpshot is triggered.
2. Functinalty to address gaps and out-of-sequence notificaitons by re-fetching 
addtional notifications that were already fetched. This solution is not fool 
proof. It does a best effort to reduse the chance of loosing notifications by 
re-fetching the notifications.This approach will introduce an over head of 
fetching addtional notifications that were already fetched. Overhead of DB 
look-up is addressed by using a cache. This reduces additional DB lookups 
needed to check if the notification was already processed.
2. Added looging to report duplicate events and out of order events for debug 
purpose.
3. Added new e2e tests to verfy this behavior.


Diffs (updated)
-----

  
sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/exception/SentryOutOfSyncException.java
 PRE-CREATION 
  
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HMSFollower.java
 929e6be 
  
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java
 4521ad4 
  
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java
 93cc34f 
  
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java
 bf5d85b 
  
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java
 7e02874 
  
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollower.java
 61e3f06 
  
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java
 91c90f9 
  
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcher.java
 83a1bec 
  
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcherCache.java
 PRE-CREATION 
  
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreation.java
 PRE-CREATION 
  
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreationWithShorterHMSEventTtl.java
 PRE-CREATION 
  
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotWithLongerHMSFollowerLongerInterval.java
 PRE-CREATION 
  
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java
 9b0aeb2 


Diff: https://reviews.apache.org/r/64955/diff/9/

Changes: https://reviews.apache.org/r/64955/diff/8-9/


Testing
-------

Made sure that tests pass.


Thanks,

kalyan kumar kalvagadda

Reply via email to