Na Li created SENTRY-1882:
Summary: Stream HMS full snapshot
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Na Li
Right now, we hold the meta store full snapshot in memory until it finishes.
Where there are a lot of tables, the size could be really big and may cause out
of memory issue.
It is better to stream the snapshot processing, saving into sentry data store
while getting next table. In this way, the memory requirement for getting full
snapshot can drop dramatically.
When we finish, we save the notificationID right before taking the snapshot.
Then get notifications with ID larger than notificationIDBefore, and process
them as normal notification events.
We don't need atomic write of the snapshot in a single transaction. We may be
able to do it in simpler ways
This message was sent by Atlassian JIRA