swamirishi commented on PR #7991: URL: https://github.com/apache/ozone/pull/7991#issuecomment-2698964143
> > FYI This still doesn't solve the problem of inconsistent rocksdb instance for a snapshot. Yes we are disabling the snapshot deleting service thread and ensuring there is no double buffer flush going on or background service thread writing data to snapshot rocksdb. But we aren't completely closing rocksdb when we do a bootstrap we should not have an open rocksdb instance in the snapshot cache while we do a bootstrap, there could be potential background rocksdb process which could be doing a WAL flush/sst file flush while the bootstrap thread performs an ls operation. > > Thanks, @swamirishi for pointing out the WAL. It is a good call out and I missed it. We can use [SnapshotCache#invalidateAll()] (https://github.com/apache/ozone/blob/770d3e62144267d2970f19ad29916186bec2b085/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java#L113) once all the locks are acquired. If another thread opens the RocksDB later, it will be mostly in read mode after that. While doing this operation we should not keep the rocksdb open. Otherwise we should take a checkpoint of the rocksdb before we iterate through a snapshot directory and iterate through files of the checkpoint. Let us not iterate through an opern rocksdb. Either we ensure the rocksdb remains closed, this may be helpful for that https://github.com/apache/ozone/pull/7745 or we take a checkpoint of a checkpointed rocksdb which in my opinion is an overaction. We can brainstorm on this, otherwise these changes look good to me for the problem it is trying to address. > > > Do you think we can possibly write some test case for this? > > This is a tricky one. I will have to introduce some wait by overriding one or more functions :/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
