[
https://issues.apache.org/jira/browse/HDFS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012630#comment-13012630
]
Ivan Kelly commented on HDFS-1538:
----------------------------------
{quote}
The "VERSION" files are one thing that will need to be discussed/attacked later
in this branch. There are currently some things in there that are
image-specific (eg checkpoint time) that don't really make a ton of sense when
multiple images may be present in a directory, and the image/edit files
themselves encode their txids. I will be opening a JIRA to discuss this soon.
The particular reason for this call being distinct from the latestNameSD is
that we might roll edit logs several times without actually saving a
checkpoint. This may cause the VERSION file to get updated in one of the edits
directories rather than one of the image directories.
I'm still not 100% sure that this makes sense, though. Can I leave this as is,
but add a "TODO" note to come back to this before merge? We will sweep for
TODO/FIXME/etc before the project is deemed merge-able.
{quote}
Is your multiple edit directory scenario referring to what may happen post1073?
It can happen now, loadPlan does a sweep to find the latest directory so thats
covered, but getStorageDirectoryForProperties() always returns latestNameSD.
In any case, I've no problem with this becoming a TODO.
{quote}
Who, then, should set storage.lastCheckpointTxId? To me, this makes sense -
storage.lastCheckpointTxId is the txid
of the last checkpoint (ie fsimage_X) that was either loaded or saved. Hence,
the loading of edit logs should always start at lastCheckpointTxId + 1.
{quote}
Set it when you load the image or save the image, so that VERSIONS and image
are in sync. This is all fine. What I was suggesting was to, instead of having
loadFSEdits retrieve it from NNStorage, return it from loadFSImage (after
setting on nnstorage) and pass it into the subsequent call to loadFSEdits. I
think this would make it clearer that when you load a image up to point N, you
load edits starting from N+1.
> Refactor more startup and image loading code out of FSImage
> -----------------------------------------------------------
>
> Key: HDFS-1538
> URL: https://issues.apache.org/jira/browse/HDFS-1538
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1538-1.txt, hdfs-1538-2.txt, hdfs-1538-on-1521.txt,
> hdfs-1538-on-branch.2.txt, hdfs-1538-on-branch.txt, hdfs-1538.txt
>
>
> For HDFS-1073, we need to be able to continue to load images in the old
> "fsimage/edits/edits.new" layout for the purposes of upgrade. But that code
> will be only for backwards compatibility, and we want to be able to switch to
> new code for the new layout. This subtask is to separate out much of that
> code into an interface which we can implement for both the old and new
> layouts.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
