smengcl commented on PR #4680: URL: https://github.com/apache/ozone/pull/4680#issuecomment-1553565271
> > we ask bootstrapping followers to ignore any newer compaction log entries than the checkpointed active DB's [sequence number](https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java#L594-L595), > > I think that is a great solution if it is not too difficult. Does each compaction log entry have a corresponding sequence number? How do we new which entries are newer than a certain number? Here is an example of compaction log that is used in a UT: https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java#L281-L302 So it looks like atm we only have sequence number for `S ` (snapshot taken) event entries. For a cleaner solution we should prepend sequence number to those `C ` (compaction) entries as well. (At the time I thought about adding sequence number for `C ` entries but didn't because we didn't have a use for it back then. Now is the time to add it.) So by that I mean this line: https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java#L285 should become something like: ``` "C 2004 000001,000002:000062\n" ``` This could be achieved by adding this line: ``` sb.append(db.getLatestSequenceNumber()).append(SPACE_DELIMITER); ``` right after this: https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java#L545-L546 And we will also need to tune the compaction log reading/parsing to correctly handle this during OM startup. You could leave a TODO in this PR for this, and file a new jira for the compaction entry sequence number addition. (up to you) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
