[
https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kelly updated HDFS-1725:
-----------------------------
Description:
HDFS-1580 proposes extending FSEditLog to allow it to use editlog streams which
are not backed by StorageDirectory. Currently, to set the the directories used
for edits, NNStorage#setStorageDirectory is called with a list of URIs as the
second argument. NNStorage takes this list or URIs, takes all file:/// URIs and
adds them to its StorageDirectory list. Then, when opened, FSEditLog will
request a list of StorageDirectories from NNStorage and create a list of
EditLogOutputStreams based on these.
This approach cannot work with HDFS-1580. NNStorage exists solely to deal with
filesystem based storage. As such, only StorageDirectories can be retrieved
from NNStorage by FSEditLog. So, FSEditLog should get the URI from some place
other than NNStorage. This presents a further problem, in that,
NNStorage#setStorageDirectories is the current way of setting the URIs for
images and edits. This call can happen at any time, so the directories in
NNStorage can change at any time. If FSEditLog is to get its URIs from
elsewhere, this opens up the risk of the filesystem directories in NNStorage
and filesystem URIs being out of sync.
A solution to this is to stipulate that the URIs for NNStorage are set only
once, on construction. All proper uses of NNStorage#setStorageDirectories are
being called just after construction of the image in any case. All other cases
are using NNStorage#setStorageDirectories not to set the storage directories,
but for the side effects of this call. This guide explains these other cases.
was:
FSImage construction is messy. Sometimes the storagedirectories in use are set
straight away, sometimes they are not. This makes it hard for anything under
FSImage (i.e. FSEditLog) to make assumptions about what it can use. Therefore,
this patch makes FSImage set the storage directories in use during
construction, and never allows them to change. If you want to change
storagedirectories you create a new image.
Also, all the construction code should be the same with the only difference
being the parameters passed. When not passed, these should get sensible
defaults.
Fix Version/s: (was: 0.23.0)
Edit log branch (HDFS-1073)
Summary: Set storage directories only at FSImage construction (was
Cleanup FSImage construction) (was: Cleanup FSImage construction)
> Set storage directories only at FSImage construction (was Cleanup FSImage
> construction)
> ---------------------------------------------------------------------------------------
>
> Key: HDFS-1725
> URL: https://issues.apache.org/jira/browse/HDFS-1725
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: HDFS-1725-review-guide.pdf, HDFS-1725.diff,
> HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff
>
>
> HDFS-1580 proposes extending FSEditLog to allow it to use editlog streams
> which are not backed by StorageDirectory. Currently, to set the the
> directories used for edits, NNStorage#setStorageDirectory is called with a
> list of URIs as the second argument. NNStorage takes this list or URIs, takes
> all file:/// URIs and adds them to its StorageDirectory list. Then, when
> opened, FSEditLog will request a list of StorageDirectories from NNStorage
> and create a list of EditLogOutputStreams based on these.
> This approach cannot work with HDFS-1580. NNStorage exists solely to deal
> with filesystem based storage. As such, only StorageDirectories can be
> retrieved from NNStorage by FSEditLog. So, FSEditLog should get the URI from
> some place other than NNStorage. This presents a further problem, in that,
> NNStorage#setStorageDirectories is the current way of setting the URIs for
> images and edits. This call can happen at any time, so the directories in
> NNStorage can change at any time. If FSEditLog is to get its URIs from
> elsewhere, this opens up the risk of the filesystem directories in NNStorage
> and filesystem URIs being out of sync.
> A solution to this is to stipulate that the URIs for NNStorage are set only
> once, on construction. All proper uses of NNStorage#setStorageDirectories are
> being called just after construction of the image in any case. All other
> cases are using NNStorage#setStorageDirectories not to set the storage
> directories, but for the side effects of this call. This guide explains these
> other cases.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
