reddycharan commented on a change in pull request #2090: Update and flush
lastLogMark when replaying journal
URL: https://github.com/apache/bookkeeper/pull/2090#discussion_r283991954
##########
File path:
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
##########
@@ -883,12 +884,51 @@ public void process(int journalVersion, long offset,
ByteBuffer recBuff) throws
};
for (Journal journal : journals) {
- journal.replay(scanner);
+ replay(journal, scanner);
}
long elapsedTs = System.currentTimeMillis() - startTs;
LOG.info("Finished replaying journal in {} ms.", elapsedTs);
}
+ /**
+ * Replay journal files and updates journal's in-memory lastLogMark object.
+ *
+ * @param journal Journal object corresponding to a journalDir
+ * @param scanner Scanner to process replayed entries.
+ * @throws IOException
+ */
+ private void replay(Journal journal, JournalScanner scanner) throws
IOException {
+ final LogMark markedLog = journal.getLastLogMark().getCurMark();
+ List<Long> logs =
Journal.listJournalIds(journal.getJournalDirectory(), journalId -> {
+ if (journalId < markedLog.getLogFileId()) {
+ return false;
+ }
+ return true;
+ });
+ // last log mark may be missed due to no sync up before
+ // validate filtered log ids only when we have markedLogId
+ if (markedLog.getLogFileId() > 0) {
+ if (logs.size() == 0 || logs.get(0) != markedLog.getLogFileId()) {
+ throw new IOException("Recovery log " +
markedLog.getLogFileId() + " is missing");
+ }
+ }
+
+ // TODO: When reading in the journal logs that need to be synced, we
+ // should use BufferedChannels instead to minimize the amount of
+ // system calls done.
+ for (Long id : logs) {
+ long logPosition = 0L;
+ if (id == markedLog.getLogFileId()) {
+ logPosition = markedLog.getLogFileOffset();
+ }
+ LOG.info("Replaying journal {} from position {}", id, logPosition);
+ journal.scanJournal(id, logPosition, scanner);
+ // Update LastLogMark to Long.MAX_VALUE position after replaying
journal
+ // After LedgerStorage flush, SyncThread should persist this to
disk
+ journal.getLastLogMark().setCurLogMark(id, Long.MAX_VALUE);
Review comment:
there are multiple things which we have to make sure about this
1) under no circumstance, we would try to write to this journal file beyond
this curLogMark (Long.MAX_VALUE position).
2) What if a reader tries to read this journal file upto this Long.MAX_VALUE
position? say for eg - readJournal BookieShell command.
In summary we have to be sure that all the components in our code deals
correctly with this special position.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
