Na Li created SENTRY-1882:
-----------------------------

             Summary: Stream HMS full snapshot
                 Key: SENTRY-1882
                 URL: https://issues.apache.org/jira/browse/SENTRY-1882
             Project: Sentry
          Issue Type: Improvement
          Components: Sentry
    Affects Versions: 2.0.0
            Reporter: Na Li
            Priority: Minor


Right now, we hold the meta store full snapshot in memory until it finishes. 
Where there are a lot of tables, the size could be really big and may cause out 
of memory issue. 

It is better to stream the snapshot processing, saving into sentry data store 
while getting next table. In this way, the memory requirement for getting full 
snapshot can drop dramatically.

When we finish, we save the notificationID right before taking the snapshot. 
Then get notifications with ID larger than notificationIDBefore, and process 
them as normal notification events.

We don't need atomic write of the snapshot in a single transaction. We may be 
able to do it in simpler ways




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to