[
https://issues.apache.org/jira/browse/SENTRY-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317081#comment-15317081
]
Sravya Tirukkovalur commented on SENTRY-872:
--------------------------------------------
[~Yibing], Hive-7973 comes in multiple parts: 1. HMS schema changes to include
notification logs, 2. DbNotificationListener which is essentially a
postEventListener, 3. ReplicationTasks which use these notification logs for
replication.
For 2 above, In Sentry's plugin for HMS, we have a very similar post event
listener implementation (SentryMetastorePostEventListener). The only difference
right now being: Sentry's listener sends these notifications as RPC to Sentry
service, where as DbNotificationListener persists the notifications in DB.
We think persisting in DB approach is better than RPC as it brings with it
multiple advantages mainly 1. Delta changes are persisted allowing the clients
(even in a Sentry HA failover case) to pull 2. This gets a global sequence of
HMS updates when there are more than one writers (HMS HA). 3. Opportunity to
make this part of transaction, so that notification log is a true WAL with
consistency guarantees.
Given that, we do have two options here. 1. Either improve Sentry's plugin to
persist to Notification Log. 2. Improve Hive plugin
We are inclining more towards option 1, as there are also some Sentry specific
optimizations we might be able to do like capture location in the notification
log. Which also avoids a second round trip to HMS when processing the
notification log, which avoids consistency issues around this.
Also, Sentry's plugin is running on production use cases for a while now,
whereas 7973 is pretty new, so it would be a while before it matures. We can
always contribute back the Sentry plugin to Hive upstream if there is enough
interest. What do you think?
> Uber jira for HMS HA + Sentry HA with HDFS plugin improvements
> --------------------------------------------------------------
>
> Key: SENTRY-872
> URL: https://issues.apache.org/jira/browse/SENTRY-872
> Project: Sentry
> Issue Type: Improvement
> Components: Hdfs Plugin
> Affects Versions: 1.5.0
> Reporter: Sravya Tirukkovalur
> Assignee: Sravya Tirukkovalur
> Fix For: 1.8.0
>
> Attachments: SENTRY-872.0.patch, SENTRY-872.pdf, SENTRY-872_design.pdf
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
