[
https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301652#comment-16301652
]
kalyan kumar kalvagadda commented on SENTRY-2109:
-------------------------------------------------
With this below logic HMSFollower excepts that event-id for the first
notification that is fetched should be greater than the last notification
processed by 1. This may not be the case.
{noformat}
public boolean areNotificationsOutOfSync(Collection<NotificationEvent> events,
long latestProcessedId) {
if (events.isEmpty()) {
return false;
}
/*
* If the sequence of notifications has a gap, then an out-of-sync might
* have happened due to the following issue:
*
* - HDFS sync was disabled or Sentry was shutdown for a time period longer
than
* the HMS notification clean-up thread causing old notifications to be
deleted.
*
* HMS notifications may contain both gaps in the sequence and duplicates
* (the same ID repeated more then once for different events).
*
* To accept duplicates (see NotificationFetcher for more info), then a gap
is found
* if the 1st notification received is higher than the current ID processed
+ 1.
* i.e.
* 1st ID = 3, latest ID = 3 (duplicate found but no gap detected)
* 1st ID = 4, latest ID = 3 (consecutive ID found but no gap detected)
* 1st ID = 5, latest ID = 3 (a gap is detected)
*/
List<NotificationEvent> eventList = (List<NotificationEvent>) events;
long firstNotificationId = eventList.get(0).getEventId();
if (firstNotificationId > (latestProcessedId + 1)) {
LOGGER.info("Current HMS notifications are out-of-sync with latest Sentry
processed"
+ "notifications. Need to request a full HMS snapshot.");
return true;
}
return false;
}
{noformat}
I have attached the snapshot for notification_log table for reference.
> Fix the logic of identifying HMS out of Sync
> --------------------------------------------
>
> Key: SENTRY-2109
> URL: https://issues.apache.org/jira/browse/SENTRY-2109
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
> Affects Versions: 2.1.0
> Reporter: kalyan kumar kalvagadda
> Assignee: kalyan kumar kalvagadda
> Fix For: 2.1.0
>
> Attachments: Screenshot_HMS_NOTIFICATION_LOG.png
>
>
> Currently HMSFollower proactively checks if sentry is out of sync with HMS
> and initiates full snapshot, if needed.
> There will be false positives with the current logic if there are gaps in the
> event-id in the notification log sequence.
> This jira is aimed at making that logic robust.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
