[
https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027159#comment-13027159
]
Todd Lipcon commented on HDFS-1799:
-----------------------------------
bq. JournalManager currently deals only with output
bq. Lastly, this patch doesn't hide the fact that files are in use here
These two are explicitly out of scope for this JIRA (note in the description
that I call out that this is supposed to be less ambitious than 1580). This is
meant to be a start but not an end to the refactoring of these interfaces.
bq. Regarding LogSegment, I'm still on the fence about this. I see some of the
merits in it, and I can see why JournalMananger is possibly needed for it. If
you retain the list of EditLogOutputStreams as it is now, you'll have to
maintain a separate list of some other objects for the creation of new log
segments. This could be problematic as these lists need to be kept in sync for
error handling etc
Exactly what I was thinking - thanks for expressing this better than I could.
The issue is that, if an EditLogOutputStream gets an IOException, I don't want
to keep using that object. JournalManager allows us to abort that stream and
get a new one next time we start a new segment.
How would you feel about the following API as a goal:
{noformat}
JournalManager:
void setBufferCapacity(int size);
EditLogOutputStream startLogSegment(long txId);
EditLogOutputStream:
void close(); // closes and "finalizes"
void abort(); // closes but marks as possibly truncated (eg after an IOE)
... current calls related to write/flush/sync ...
{noformat}
Then, in FSEditLog, whenever we start a new segment, we'd call startLogSegment
on all JournalManagers and insert the ELOS instances into a list. Journal
Managers never get marked as "failed" -- we only remove the failed ELOS after
aborting it. Then on the next segment we always give each JM a new change to
start a segment.
> Refactor log rolling and filename management out of FSEditLog
> -------------------------------------------------------------
>
> Key: HDFS-1799
> URL: https://issues.apache.org/jira/browse/HDFS-1799
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: 0001-Added-state-management-to-FSEditLog.patch,
> 0002-Standardised-error-pattern.patch,
> 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch,
> HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt
>
>
> This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA
> focuses on pluggability, this task is simply the minimum needed for HDFS-1073:
> - Refactor the filename-specific code for rolling, diverting, and reverting
> log streams out of FSEditLog into a new class
> - Clean up the related code in FSEditLog a bit
> Notably, this JIRA is going to temporarily break the BackupNode. I plan to
> circle back on the BackupNode later on this branch.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
