-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64955/
-----------------------------------------------------------
(Updated Jan. 29, 2018, 3:47 p.m.)
Review request for sentry, Alexander Kolbasov, Arjun Mishra, Na Li, Sergio
Pena, and Vadim Spector.
Changes
-------
Added more tests to verify the cache and addressed review comments.
Bugs: SENTRY-2109
https://issues.apache.org/jira/browse/SENTRY-2109
Repository: sentry
Description
-------
This patch does couple of things
1. Avoid triggering full snapshots when gaps are observed while fetching new
notifications. While fetching new notifications HMSFollower would would fetch
notifications with last event-id as well. When it gets the notifications and if
it doesn't get the notifications with event-id, full snpshot is triggered.
2. Functinalty to address gaps and out-of-sequence notificaitons by re-fetching
addtional notifications that were already fetched. This solution is not fool
proof. It does a best effort to reduse the chance of loosing notifications by
re-fetching the notifications.This approach will introduce an over head of
fetching addtional notifications that were already fetched. Overhead of DB
look-up is addressed by using a cache. This reduces additional DB lookups
needed to check if the notification was already processed.
2. Added looging to report duplicate events and out of order events for debug
purpose.
3. Added new e2e tests to verfy this behavior.
Diffs (updated)
-----
sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/exception/SentryOutOfSyncException.java
PRE-CREATION
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HMSFollower.java
2f2b984
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java
edea5b6
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HiveNotificationFetcher.java
93cc34f
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java
96c6810
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java
7e02874
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollower.java
7903078
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java
91c90f9
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStore.java
b410027
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcher.java
83a1bec
sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestHiveNotificationFetcherCache.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreation.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotCreationWithShorterHMSEventTtl.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestSnapshotWithLongerHMSFollowerLongerInterval.java
PRE-CREATION
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java
4cd00e6
Diff: https://reviews.apache.org/r/64955/diff/6/
Changes: https://reviews.apache.org/r/64955/diff/5-6/
Testing
-------
Made sure that tests pass.
Thanks,
kalyan kumar kalvagadda
