[
https://issues.apache.org/jira/browse/HADOOP-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609975#comment-14609975
]
Steve Loughran commented on HADOOP-11981:
-----------------------------------------
It's not yet ready to go in, because it's more javadocs than specification.
These documents constitute an attempt to define what a filesystem is expected
to do in way from which we can derive tests and implementations. For that they
have to be unambiguous and not omit things which were felt to be "too obvious
to mention". Its actually quite hard to do; for the files in there I had to go
through all the implementations we had and see what they really did ... tagging
as "newbie" is actually quite optimistic.
The storage policy APIs are going to have be another aspect of the filesystem
model covered in {{model.md}}. that is currently, the FS is
{code}
(Directories:set[Path], Files:[Path:List[byte]], Symlinks:set[Path])
{code}
It's going to have to become something like
{code}
(Directories:set[Path], Files:[Path:List[byte]], Symlinks:set[Path],
storagePolicies: set[BlockStoragePolicySpi], storagePolicy[Path: String])
{code}
The Xatrrs can go in at the same time.
Operations on the storage policy then become actions which read/write these new
sets & maps, things we can write down in the python-base notation:
{code}
getStoragePolicies()
preconditions:
if FS.storagePolicies=={} raise UnsupportedOperationException
postconditions:
result = storagePolicies(FS)
{code}
A more interesting one is the setter:
{code}
setStoragePolicy(src, policyName)
preconditions:
if FS.storagePolicies=={} raise UnsupportedOperationException
if [p in FS.storagePolicies where p.getName==policyName] ==[] raise
HadoopIllegalArgumentException
postconditions:
FS' = FS where FS'.storagePolicy(src)==policyName
{code}
What this does is try to make things less ambiguous, and so make implementation
of filesystems and the tests easier. It also means that we can look at the HDFS
implementation and say whether or not this is what it should be doing.
For Example
# {{BlockStoragePolicySuite}} is actually using case insensitive checks without
specifying the locale ... that's exactly the kind of thing we need to be
defining, and so use it to identify issues like HDFS-8705.
# {{FSDirAttrOp}} will raise an {{IOE}} if the storage policy is disabled. This
is something which clearly needs specifying
# HDFS implies the caller needs write access to the path. Again, this need to
be part of the specification.
Accordingly: not yet, and it's going to be harder than you expect. I will help
review it for you, and help define that extended FS model that's needed.
> Add storage policy APIs to filesystem docs
> ------------------------------------------
>
> Key: HADOOP-11981
> URL: https://issues.apache.org/jira/browse/HADOOP-11981
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Labels: newbie
> Attachments: HADOOP-11981.incomplete.01.patch
>
>
> HDFS-8345 exposed the storage policy APIs via the FileSystem.
> The FileSystem docs should be updated accordingly.
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
