swamirishi commented on PR #5035: URL: https://github.com/apache/ozone/pull/5035#issuecomment-1632965951
> @swamirishi As I said above, I'm not sure the solution to the delete problem is complete. Let me ask a few questions: > > 1. Is the problem that this method returns before the sst file is deleted?: https://github.com/apache/ozone/blob/4aa49370b47397ceee3e7f29069195923f5b8a1e/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java#L993 The rocksdb delete api is asynchronous. The files to be deleted are added to a scheduler. > 2. Does the delete rename the sst file to the ldb file, and then later deletes the ldb file? No, rocksdb supports both sst files & ldb files(To support older level db migrated data). The logging in rocksdb just shows it to be an ldb file. But in actuality it is just an sst file. > 3. If so, then don't we also need to wait for the non-existance of the ldb file? ldb files don't exist in ozone rocksdb > 4. Does the delete do anything else? How do we know when it is complete? Delete just deletes data and updates the manifest file. The actual problem is the snapshot cache & sst filtering service acting up together. So when we actually try to install a om metadata snapshot from leader we actually copy individual files from checkpoint directory to ozone metadata location. So if we have an open rocksdb instance for that particular db while we are copying. The delete scheduler could kick in and delete the sst file, which wouldn't happen if the rocksdb is closed. This problem becomes more prominent in https://github.com/apache/ozone/pull/5008 when sst filtering service also uses snapshot cache & can be reproduced more easily. So if we have an open instance of rocksdb in the snapshot cache we cannot really determine when the delete is going to happen, so we need to stop the sst filtering service before installation (which is already being done) and also invalidate the snapshot cache. We also should have only one instance install running at any given time thus I have made changes to also synchronize the particular function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
