[
https://issues.apache.org/jira/browse/SENTRY-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1669:
--------------------------------------------
Attachment: (was: SENTRY-1669.011-sentry-ha-redesign.patch)
> HMSFollower should read current processed notification ID from database every
> time it runs
> ------------------------------------------------------------------------------------------
>
> Key: SENTRY-1669
> URL: https://issues.apache.org/jira/browse/SENTRY-1669
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
> Affects Versions: sentry-ha-redesign
> Reporter: Hao Hao
> Assignee: kalyan kumar kalvagadda
> Priority: Blocker
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1669.001-sentry-ha-redesign.patch,
> SENTRY-1669.002-sentry-ha-redesign.patch,
> SENTRY-1669.003-sentry-ha-redesign.patch,
> SENTRY-1669.004-sentry-ha-redesign.patch,
> SENTRY-1669.005-sentry-ha-redesign.patch,
> SENTRY-1669.006-sentry-ha-redesign.patch,
> SENTRY-1669.007-sentry-ha-redesign.patch,
> SENTRY-1669.008-sentry-ha-redesign.patch,
> SENTRY-1669.009-sentry-ha-redesign.patch,
> SENTRY-1669.010-sentry-ha-redesign.patch
>
>
> Instead of using in-memory current processed notification ID, HMSFollower
> should read the processed notification ID info from database every time it
> runs. Otherwise, after failover, the new leader cannot get the correct
> up-to-date processed notification ID.
> It should also update the Notification ID to database once it processes a
> notification event. The Notification ID in database should keep on increase
> (we don't handle its overflow scenario).
> HMS notification ID's are stored in separate table for couple of reasons
> # SENTRY_PATH_CHANGE is not updated for every notification that is received
> from HMS. There are cases where HMSFollower doesn't process notifications and
> skip's them. Depending on SENTRY_PATH_CHANGE information may not provide the
> last notification processed.
> # There could be cases where HMSFollower thread in multiple sentry servers be
> acting as a leader and process HMS notifications. we need to avoid processing
> the notifications multiple times. This can be made sure by always having some
> number of notification information always regardless of purging interval.
> 3. SENTRY_PATH_CHANGE information stored can typically be removed once
> namenode plug-in has processed the update.
> *As purpose and usage of notification ID information is different from PATH
> update info, it locally makes sense to store notification ID separately.*
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
