[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623684#comment-16623684
]
Steve Loughran commented on HADOOP-15691:
-----------------------------------------
Patch 003
* revise based on feedback
* catch up with trunk (pathhandles)
* add option for an FS connector to say "We know we are read only" (http, har)
* add checksum capability, wire up so those filesystems with it (DFS, S3A when
its enabled)
* symlink capability is based on {{symlinksEnabled}} flag in FileSystem class.
* expanded docs
* ~0 tests
I now realise that a bool "true/false" isn't quite right, as it doesn't cover
the "we think we'd support it but you'd have to try". Symlinks being a key
example. There's DFS: we know we have it, other stores "we know we don't", but
localFS is either "known disabled" or "try it and see"
An enum seems the outcome here, something like
{code}
known-available: its a known feature and known to work in this instance (ex:
symlinks on hdfs)
known-unavailable: we know of it, we know we don't support it (ex: append on
s3a)
known-maybe-available: we know of it, but to see if it works, you will have to
try the operation. (symlink on local)
unknown: we don't know of this feature.
{code}
the base FS would say "unknown feature" for ~everything; implementations would
list what they know about and their interpretation, so that everything with
rename would add "fs:rename"; posix-ish filesytems with atomic rename,
"fs-rename-atomic"; the object stores would mark the latter as
"known-unavailable", etc.
We'd also need to qualify everything with "depending on your permissions". An
FS may know of append, but it might be a read-only mount. I've already added to
filesystem.md the fact that the state may change depending on external events.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> ------------------------------------------------------------------
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
