[
https://issues.apache.org/jira/browse/SENTRY-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-1669:
--------------------------------------------
Description:
Instead of using in-memory current processed notification ID, HMSFollower
should read the processed notification ID info from database every time it
runs. Otherwise, after failover, the new leader cannot get the correct
up-to-date processed notification ID.
It should also update the Notification ID to database once it processes a
notification event. The Notification ID in database should keep on increase (we
don't handle its overflow scenario).
HMS notification ID's are stored in separate table for couple of reasons
# SENTRY_PATH_CHANGE is not updated for every notification that is received
from HMS. There are cases where HMSFollower doesn't process notifications and
skip's them. Depending on SENTRY_PATH_CHANGE information may not provide the
last notification processed.
# There could be cases where HMSFollower thread in multiple sentry servers be
acting as a leader and process HMS notifications. we need to avoid processing
the notifications multiple times. This can be made sure by always having some
number of notification information always regardless of purging interval.
3. SENTRY_PATH_CHANGE information stored can typically be removed once namenode
plug-in has processed the update.
*As purpose and usage of notification ID information is different from PATH
update info, it locally makes sense to store notification ID separately.*
was:
Instead of using in-memory current processed notification ID, HMSFollower
should read the processed notification ID info from database every time it
runs. Otherwise, after failover, the new leader cannot get the correct
up-to-date processed notification ID.
It should also update the Notification ID to database once it processes a
notification event. The Notification ID in database should keep on increase (we
don't handle its overflow scenario).
HMS notification ID's are stored in separate table for couple of reasons
# SENTRY_PATH_CHANGE is updated for every notifications that is received from
HMS. There are cases where HMSFollower doesn't process notifications and skip's
them
# There could be cases where HMSFollower threads in multiple sentry serves
could act as a leader and process HMS notifications. we need to avoid
processing the notifications multiple times. This can be be made sure by always
having the notification ID's some number of notification information always
regardless of purging interval.
3. SENTRY_PATH_CHANGE information stored can typically bed removed once
namenode plug-in
has processed the update.
*As purpose and usage of notification ID information is different from PATH
update info, it locally makes sense to store notification ID separately.*
> HMSFollower should read current processed notification ID from database every
> time it runs
> ------------------------------------------------------------------------------------------
>
> Key: SENTRY-1669
> URL: https://issues.apache.org/jira/browse/SENTRY-1669
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
> Affects Versions: sentry-ha-redesign
> Reporter: Hao Hao
> Assignee: kalyan kumar kalvagadda
> Priority: Blocker
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1669.001-sentry-ha-redesign.patch,
> SENTRY-1669.002-sentry-ha-redesign.patch,
> SENTRY-1669.003-sentry-ha-redesign.patch,
> SENTRY-1669.004-sentry-ha-redesign.patch,
> SENTRY-1669.005-sentry-ha-redesign.patch,
> SENTRY-1669.006-sentry-ha-redesign.patch,
> SENTRY-1669.007-sentry-ha-redesign.patch,
> SENTRY-1669.008-sentry-ha-redesign.patch,
> SENTRY-1669.009-sentry-ha-redesign.patch,
> SENTRY-1669.010-sentry-ha-redesign.patch
>
>
> Instead of using in-memory current processed notification ID, HMSFollower
> should read the processed notification ID info from database every time it
> runs. Otherwise, after failover, the new leader cannot get the correct
> up-to-date processed notification ID.
> It should also update the Notification ID to database once it processes a
> notification event. The Notification ID in database should keep on increase
> (we don't handle its overflow scenario).
> HMS notification ID's are stored in separate table for couple of reasons
> # SENTRY_PATH_CHANGE is not updated for every notification that is received
> from HMS. There are cases where HMSFollower doesn't process notifications and
> skip's them. Depending on SENTRY_PATH_CHANGE information may not provide the
> last notification processed.
> # There could be cases where HMSFollower thread in multiple sentry servers be
> acting as a leader and process HMS notifications. we need to avoid processing
> the notifications multiple times. This can be made sure by always having some
> number of notification information always regardless of purging interval.
> 3. SENTRY_PATH_CHANGE information stored can typically be removed once
> namenode plug-in has processed the update.
> *As purpose and usage of notification ID information is different from PATH
> update info, it locally makes sense to store notification ID separately.*
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
