[
https://issues.apache.org/jira/browse/SENTRY-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133425#comment-16133425
]
Sergio Peña commented on SENTRY-1888:
-------------------------------------
I've been thinking about the best approach to handle this situation.
First, the idea is to fetch notifications during a time window expecting that
HMS will eventually commit some of the duplicated events IDs. We need to decide
on the best configuration for the time window. Perhaps we will need to add a
new timeout for commits on the HMS side so that HMS transactions and
notifications are committed in that time window, then we could use that same
value for Sentry.
Second, we would need to apply those new events fetched, but there is where I
have 2 options:
1) Apply only those new notifications that are fetched in a time window. This
requires that we keep track of the full HMS notifications content in a DB table
so that we compare against the notification fetched, and then apply only those
newer notifications. This is easier to do, and we get all processed
notifications from a time window and filter them out when fetching the
notifications again.
2) Re-apply all notifications in the DB and ignore the errors of notifications
already applied. This still requires us to keep track of at least HMS
notifications timestamps to know what's the next ID to request based on the
time window. This is more complicated as there places where we add paths that
can be repeated, drop/add privileges, and other persisting methods that would
need to be modified to avoid repeated information.
I think option #1 is the best approach as the HMS guarantees that notifications
with duplicated IDs won't collide and isolated operations. For instance, HMS
will never get these events:
- ID = 10, CREATE_DATABASE db1
- ID = 10, DROP_DATABASE db1
The above won't happen because CREATE cannot happen if db1 already exists, and
DROP cannot happen if db1 does not exist. Once the ID = 10 is committed, then
we can DROP it or CREATE it, and this will contain an ID = 11
Also, this other scenario will NOT happen on the HMS:
- ID = 10, CREATE_DATABASE db1
- ID = 12, CREATE_TABLE table1
Then, could Sentry fetch this the next time?
- ID = 10, CREATE_DATABASE db1
- ID = 11, DROP_TABLE table1
- ID = 12, CREATE_TABLE table1
HMS guarantees the order of the IDs because it increments the ID on the HMS
application (no datanucleus). So holes won't appeared.
[~LinaAtAustin] [~vamsee] [~akolb] What do you think about this?
> Sentry might not fetch all HMS duplicated events IDs when requested
> -------------------------------------------------------------------
>
> Key: SENTRY-1888
> URL: https://issues.apache.org/jira/browse/SENTRY-1888
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
> Affects Versions: 2.0.0
> Reporter: Sergio Peña
>
> HMS does not guarantee that each notification has unique IDs. SENTRY-1803
> solved the issue with Sentry to handle those duplicated events IDs. However,
> HMS notifications with duplicated events IDs could appear late on the HMS
> side due to delay issues on the DB (especially on HMS HA mode). These events
> could not be fetched by Sentry if we already processed a duplicated event ID
> before.
> Example:
> 1. HMS 1 attempts to persist event ID = 1
> 2. HMS 2 attempts to persist event ID = 1
> 3. HMS 1 commits event ID = 1
> 4. Sentry fetches notifications >= 1 (bringing the event from HMS 1)
> 5. HMS 2 commits event ID = 1
> 6. Sentry fetches notifications >= 2 (no events are fetched)
> HMS 2 event ID = 1 is never fetched nor processed by Sentry.
> The above scenario could cause Sentry to be out-of-sync because of these
> events that were not committed on time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
