[
https://issues.apache.org/jira/browse/SENTRY-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1669:
--------------------------------------------
Description:
Instead of using in-memory current processed notification ID, HMSFollower
should read the processed notification ID info from database every time it
runs. Otherwise, after failover, the new leader cannot get the correct
up-to-date processed notification ID.
It should also update the Notification ID to database once it processes a
notification event. The Notification ID in database should keep on increase (we
don't handle its overflow scenario).
HMS notification ID's are stored in separate table for couple of reasons
# SENTRY_PATH_CHANGE is updated for every notifications that is received from
HMS. There are cases where HMSFollower doesn't process notifications and skip's
them
# There could be cases where HMSFollower threads in multiple sentry serves
could act as a leader and process HMS notifications. we need to avoid
processing the notifications multiple times. This can be be made sure by always
having the notification ID's some number of notification information always
regardless of purging interval.
3. SENTRY_PATH_CHANGE information stored can typically bed removed once
namenode plug-in
has processed the update.
*As purpose and usage of notification ID information is different from PATH
update info, it locally makes sense to store notification ID separately.*
was:
Instead of using in-memory current processed notification ID, HMSFollower
should read the processed notification ID info from database every time it
runs. Otherwise, after failover, the new leader cannot get the correct
up-to-date processed notification ID.
It should also update the Notification ID to database once it processes a
notification event. The Notification ID in database should keep on increase (we
don't handle its overflow scenario).
HMS notification ID's are stored in separate table for couple of reasons
# Under various conditions notifications that will not make it to
SENTRY_PATH_CHANGE table, so we need separate table to store the ID's making
sure that every notification ID is recorded.
# SENTRY_PATH_CHANGE is updated for every notifications that is received from
HMS. There are cases where HMSFollower doesn't process notifications and skip's
them
# There could be cases where HMSFollower threads in multiple sentry serves
could act as a leader and process HMS notifications. we need to avoid
processing the notifications multiple times. This can be be made sure by always
having the notification ID's some number of notification information always
regardless of purging interval.
3. SENTRY_PATH_CHANGE information stored can typically bed removed once
namenode plug-in
has processed the update.
*As purpose and usage of notification ID information is different from PATH
update info, it locally makes sense to store notification ID separately.*
> HMSFollower should read current processed notification ID from database every
> time it runs
> ------------------------------------------------------------------------------------------
>
> Key: SENTRY-1669
> URL: https://issues.apache.org/jira/browse/SENTRY-1669
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
> Affects Versions: sentry-ha-redesign
> Reporter: Hao Hao
> Assignee: kalyan kumar kalvagadda
> Priority: Blocker
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1669.001-sentry-ha-redesign.patch,
> SENTRY-1669.002-sentry-ha-redesign.patch,
> SENTRY-1669.003-sentry-ha-redesign.patch,
> SENTRY-1669.004-sentry-ha-redesign.patch,
> SENTRY-1669.005-sentry-ha-redesign.patch,
> SENTRY-1669.006-sentry-ha-redesign.patch,
> SENTRY-1669.007-sentry-ha-redesign.patch,
> SENTRY-1669.008-sentry-ha-redesign.patch,
> SENTRY-1669.009-sentry-ha-redesign.patch,
> SENTRY-1669.010-sentry-ha-redesign.patch
>
>
> Instead of using in-memory current processed notification ID, HMSFollower
> should read the processed notification ID info from database every time it
> runs. Otherwise, after failover, the new leader cannot get the correct
> up-to-date processed notification ID.
> It should also update the Notification ID to database once it processes a
> notification event. The Notification ID in database should keep on increase
> (we don't handle its overflow scenario).
> HMS notification ID's are stored in separate table for couple of reasons
> # SENTRY_PATH_CHANGE is updated for every notifications that is received from
> HMS. There are cases where HMSFollower doesn't process notifications and
> skip's them
> # There could be cases where HMSFollower threads in multiple sentry serves
> could act as a leader and process HMS notifications. we need to avoid
> processing the notifications multiple times. This can be be made sure by
> always having the notification ID's some number of notification information
> always regardless of purging interval.
> 3. SENTRY_PATH_CHANGE information stored can typically bed removed once
> namenode plug-in
> has processed the update.
> *As purpose and usage of notification ID information is different from PATH
> update info, it locally makes sense to store notification ID separately.*
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
