[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006473#comment-16006473
]
Steve Loughran commented on HADOOP-9565:
----------------------------------------
stopping work on this briefly as the idea for a config-driven mech now has me
confused. Here's why
# The defaults for all filesystems need to be set *somewhere*
# The defaults for a specific schema need to be set. They may override this
explicit set, or just redeclare everything.
# At least for the blobstores, but potentially for file:// too, we need to
offer the ability to override the FS defaults. Presumably: core-site
# And for s3 we need per-bucket config, again. presumably, core-site.
customisation for schemas/bucket instances need to be on top of any schema
specific, s3a.xml, file.xml; but at the same time, anything in core-default.xml
has to come under the schema. And, as the s3a per-bucket config isn't yet
common across other instances, that's going to have to be set up by the FS
instance.
Overall then, we can't have a simple chain of {{new
Configuration(true).addResource("s3a.xml")}} as that doesnt let core-site
override s3a.xml values
And I can't load the s3a values and then core-default + core-site if there's
anything in core-default. plus there's per-bucket customisation.
I think the XML driven one is becoming too convoluted. Probably better to go to
switch statements, allowing FS implementations to implement their own policies
on top of this if they want. After all, if they were going to have any
path-specific state that was how things were going to end up anyway.
Also, copy & paste of the FS contract test pulls in too many low level
properties (the full set of rename semantics) which shouldn't really be
exposed. Better to have an initial set of options which are minimal, rely on
the new StreamCapabilities for streams.
* one for every optional method {{supports-append}}, {{supports-concat}};
default = false
* core requirements of a filesystem which object stores *may not support*:
{{has-consistent-create}}, {{has-consistent-list}}, {{has-consistent-update}},
{{has-atomic-rename}}...
> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
> Affects Versions: 2.6.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-010.patch,
> HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
