[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9565:
-----------------------------------
Attachment: HADOOP-9565-010.patch
Patch 010: resetting everything and starting again with nothing but an API to
query a filesystem for features. This is a WiP again, up for feedback.
* {{boolean FileSystem.hasFeature(Path, String)}} lets you see if the bit of an
FS under a path has a feature
* Feature names in {{FileSystemFeatures}} are pretty much a copy and paste of
{{org.apache.hadoop.fs.contract.ContractOptions}}.
* The base class mechanism for setting up feature resolution,
{{FeatureResolver}} is driven off XML configuration files; with a setup
mechanism which allows for subclasses to dynamically redefine the features,
such as when setting up a local FS.
All the work of defining features is already in the contract tests; I've just
pulled their declarations into the production hadoop-common, hadoop-hdfs-client
JARs, not the test JARs, and modified the two migrated FS's contracts to load
in the new XML file as well as the old one.
One big difference is in resolution of a feature: when a feature is looked for
it first looks for the schema-specific option
{{fs.hdfs.contract.supports-atomic-rename}} before falling back to look for
{{fs.contract.supports-atomic-rename}}, which is where a default can be
defined.
This allows for per-FS values to be set in core-default.xml,
{code}
# default: listings work
fs.contract.supports-consistent-listing = false
# but not S3A
fs.s3a.contract.supports-consistent-listing = false
{code}
And overridden in core-site.xml, as it'd be in a deployment against WDC's
servers
{code}
fs.s3a.contract.supports-consistent-listing = true
{code}
And it permits S3A per-bucket override for deployments where only some buckets
are consistent, such as "stevel-s3guard"
{code}
fs.s3a.bucket.stevel-s3guard.contract.supports-consistent-listing = true
{code}
Although the feature resolver is driven purely by the loaded hash of config
options, because the lookup is via the FS instance, it has the option of being
dynamic: per-path, on-demand evaluation rather than on construction probes for
OS type, ....
Also, the underlying feature resolver has a Java 8 optional API, so it can
return yes/no/don't know, the idea being "no" can be normative.
> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
> Affects Versions: 2.6.0
> Reporter: Steve Loughran
> Assignee: Thomas Demoor
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch, HADOOP-9565-008.patch, HADOOP-9565-010.patch,
> HADOOP-9565-branch-2-007.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
