karanmehta93 commented on a change in pull request #2090: Update and flush
lastLogMark when replaying journal
URL: https://github.com/apache/bookkeeper/pull/2090#discussion_r283997469
##########
File path:
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
##########
@@ -883,12 +884,51 @@ public void process(int journalVersion, long offset,
ByteBuffer recBuff) throws
};
for (Journal journal : journals) {
- journal.replay(scanner);
+ replay(journal, scanner);
}
long elapsedTs = System.currentTimeMillis() - startTs;
LOG.info("Finished replaying journal in {} ms.", elapsedTs);
}
+ /**
+ * Replay journal files and updates journal's in-memory lastLogMark object.
+ *
+ * @param journal Journal object corresponding to a journalDir
+ * @param scanner Scanner to process replayed entries.
+ * @throws IOException
+ */
+ private void replay(Journal journal, JournalScanner scanner) throws
IOException {
+ final LogMark markedLog = journal.getLastLogMark().getCurMark();
+ List<Long> logs =
Journal.listJournalIds(journal.getJournalDirectory(), journalId -> {
+ if (journalId < markedLog.getLogFileId()) {
+ return false;
+ }
+ return true;
+ });
+ // last log mark may be missed due to no sync up before
+ // validate filtered log ids only when we have markedLogId
+ if (markedLog.getLogFileId() > 0) {
+ if (logs.size() == 0 || logs.get(0) != markedLog.getLogFileId()) {
+ throw new IOException("Recovery log " +
markedLog.getLogFileId() + " is missing");
+ }
+ }
+
+ // TODO: When reading in the journal logs that need to be synced, we
+ // should use BufferedChannels instead to minimize the amount of
+ // system calls done.
+ for (Long id : logs) {
+ long logPosition = 0L;
+ if (id == markedLog.getLogFileId()) {
+ logPosition = markedLog.getLogFileOffset();
+ }
+ LOG.info("Replaying journal {} from position {}", id, logPosition);
+ journal.scanJournal(id, logPosition, scanner);
+ // Update LastLogMark to Long.MAX_VALUE position after replaying
journal
+ // After LedgerStorage flush, SyncThread should persist this to
disk
+ journal.getLastLogMark().setCurLogMark(id, Long.MAX_VALUE);
Review comment:
> under no circumstance, we would try to write to this journal file beyond
this curLogMark (Long.MAX_VALUE position).
Bookie restarts always create a new journal file with default size of 16 MB.
This ensures that we never write to old journal file. This scenario is covered
by the test case.
> What if a reader tries to read this journal file upto this Long.MAX_VALUE
position? say for eg - readJournal BookieShell command
Some text from the link in my previous comment, `Setting the position to a
value that is greater than the file's current size is legal but does not change
the size of the file. A later attempt to read bytes at such a position will
immediately return an end-of-file indication`, should help resolve it.
I understand that this is a critical fix and we have to think through all
possibilities. IMHO, the test case (although a bit long), tries to cover all
such potential scenarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
