[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593926#comment-16593926
]
Steve Loughran commented on HADOOP-15691:
-----------------------------------------
Patch 001: initial implementation. This defines a new interface,
PathCapabiliities, implemented by FileSystem, AbstractFileSystem and
FileContext, to probe for features. It has one method
{code}
boolean hasPathCapability(Path path, String capability) throws IOException;
{code}
* There's a core list of features added in StreamCapabilities (right place?);
for the main optional FS operations & features: permissions, xattrs, storage
policies, snapshots, append, truncate, concat, and wired up for the all built
in filesystems
* No tests; they can go into contract tests (e.g the append test can verify
that having the append capability -> append works, etc)
* Base implementation returns false for everything but the "fs:symlinks" probe,
which is forwarded to {{supportsSymlinks()}}
To do in separate JIRA;
* Object store operations/semantics, where I want to be able to declare when a
store does a PUT at the end of a write (e.g: no rename needed), and to declare
when renames are slow/non-atomic for dirs. We can use that in the mrv2
committers to know when to warn, perhaps.
* Multipart upload availablity
h3. Semantics of return value
w.r/t this patch, the spec says "return true iff you know the feature is
available". This means "false" can mean any of
* The capability is unknown.
* The capability is known, but not available on this instance.
* The capability is known but the connector does not know if it is supported by
this specific instance.
We could think about having an enum of all four states, so that you can
distinguish them, but I fear it gets complicated with forwarded filesystems.
h3. FSLinkResolver
new class {{org.apache.hadoop.fs.FSLinkResolver}} for implementing FileContext
operations as closures, here
{code}
FsLinkResolution.resolve(this,
fixRelativePart(path),
(fs, p) -> fs.hasPathCapability(p, capability));
{code}
It should be straightforward to move all FileContext operations to this & make
the file a lot leaner.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> ------------------------------------------------------------------
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
