[
https://issues.apache.org/jira/browse/SENTRY-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1669:
--------------------------------------------
Description:
Instead of using in-memory current processed notification ID, HMSFollower
should read the processed notification ID info from database every time it
runs. Otherwise, after failover, the new leader cannot get the correct
up-to-date processed notification ID.
It should also update the Notification ID to database once it processes a
notification event. The Notification ID in database should keep on increase (we
don't handle its overflow scenario).
Notification id is also available in SENTRY_PATH_CHANGE table but it is not
dependable to get the last notification processed for couple of reasons.
1. Under various conditions notifications will not make it to
SENTRY_PATH_CHANGE table so we need separate table to store the ID's making
sure that every notification ID is recorded.
2. SENTRY_PATH_CHANGE table is periodically purged and the purging logic should
depend on the changeID that was processed by the Namenode-plugin. Even with the
current purging logic we retain only one record. I'm not sure if we can assume
that this event-id is the ID for the last notification that sentry server
processed.
3. In order to deal scenario where multiple HMSFollowers processing HMS
notifications we need a different purging logic around notification ID to
handle duplicate notifications this may not be same as needed by
SENTRY_PATH_CHANGE.
was:
Instead of using in-memory current processed notification ID, HMSFollower
should read the processed notification ID info from database every time it
runs. Otherwise, after failover, the new leader cannot get the correct
up-to-date processed notification ID.
It should also update the Notification ID to database once it processes a
notification event. The Notification ID in database should keep on increase (we
don't handle its overflow scenario).
notification id is also available in SENTRY_PATH_CHANGE table but it is not
dependable to get the last notification processed. Sentry server can not depend
on this information to get the last notification that the server processed as
it not guaranteed that every notification will make it to SENTRY_PATH_CHANGE.
Under various conditions notifications will not make it to SENTRY_PATH_CHANGE
table so we need separate table to store the ID's.
> HMSFollower should read current processed notification ID from database every
> time it runs
> ------------------------------------------------------------------------------------------
>
> Key: SENTRY-1669
> URL: https://issues.apache.org/jira/browse/SENTRY-1669
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
> Affects Versions: sentry-ha-redesign
> Reporter: Hao Hao
> Assignee: kalyan kumar kalvagadda
> Priority: Blocker
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1669.001-sentry-ha-redesign.patch,
> SENTRY-1669.002-sentry-ha-redesign.patch,
> SENTRY-1669.003-sentry-ha-redesign.patch,
> SENTRY-1669.004-sentry-ha-redesign.patch,
> SENTRY-1669.005-sentry-ha-redesign.patch,
> SENTRY-1669.006-sentry-ha-redesign.patch,
> SENTRY-1669.007-sentry-ha-redesign.patch,
> SENTRY-1669.008-sentry-ha-redesign.patch
>
>
> Instead of using in-memory current processed notification ID, HMSFollower
> should read the processed notification ID info from database every time it
> runs. Otherwise, after failover, the new leader cannot get the correct
> up-to-date processed notification ID.
> It should also update the Notification ID to database once it processes a
> notification event. The Notification ID in database should keep on increase
> (we don't handle its overflow scenario).
> Notification id is also available in SENTRY_PATH_CHANGE table but it is not
> dependable to get the last notification processed for couple of reasons.
> 1. Under various conditions notifications will not make it to
> SENTRY_PATH_CHANGE table so we need separate table to store the ID's making
> sure that every notification ID is recorded.
> 2. SENTRY_PATH_CHANGE table is periodically purged and the purging logic
> should depend on the changeID that was processed by the Namenode-plugin. Even
> with the current purging logic we retain only one record. I'm not sure if we
> can assume that this event-id is the ID for the last notification that sentry
> server processed.
> 3. In order to deal scenario where multiple HMSFollowers processing HMS
> notifications we need a different purging logic around notification ID to
> handle duplicate notifications this may not be same as needed by
> SENTRY_PATH_CHANGE.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
