smengcl commented on code in PR #5070:
URL: https://github.com/apache/ozone/pull/5070#discussion_r1268915282
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -588,6 +590,8 @@ public void onCompactionCompleted(RocksDB db,
return;
}
+ waitForTarballCreation();
Review Comment:
Using the compaction listener should also work but not as ideal.
Because in this case tarball creation process (which includes DB
checkpointing) does not know if or when a compaction will happen, tarball
creation thread has to block compaction threads similar to what we already have
in the listener while we add existing compaction log to the tarball (no need to
copy it elsewhere, just stream it into the tarball?) and hard-link the backup
SST files somewhere. -- It just occurred to me that we might need to block DAG
cleanup thread as well to prevent backup SST files from being cleaned up while
tarball is being created? cc @hemantk-12
Another solution without blocking or adding explicit lock would be: if
tarball creation is in-progress, the compaction listener can append to a
**staging** compaction log and hard-link SSTs to a **staging** SST backup dir,
and some other threads will have to be responsible of merging the staging parts
back. Tarball creation process and OM startup also has to trigger a check to
merge them if there were any previous staging logs and backup SSTs. This
approach might be a bit more error-prone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
