ajothomas commented on PR #1708: URL: https://github.com/apache/samza/pull/1708#issuecomment-2486952101
> Now I think about it, I have a question about whether this can work for Samza incremental checkpoint. So assume the first checkpoint failed during the state upload, we suppress the exception and let it continue. Then the next checkpoint will invoke rocksDb.checkpoint() which will do a delta of the previous checkpoint. If the second checkpoint succeeds, but the first checkpoint actually fails, seems the states needed will not be in ambry anymore. > > We need to have a in-depth discussion on this. Thanks. Hey @xinyuiscool This is a great point and something I missed in the original review. As far as I know, rocksdb checkpoints are not incremental snapshots but point-in-time complete snapshot so we should be okay even if the previous commit failed as the rocksdb checkpoint would have hard links to all ssts. https://rocksdb.org/blog/2015/11/10/use-checkpoints-for-efficient-snapshots.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
