[
https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616733#comment-15616733
]
Aaron Fabbri commented on HADOOP-13651:
---------------------------------------
Thanks for comments [~cnauroth]!
{quote}
would like to discuss the management of MetadataStore as a singleton.
{quote}
I'm in general agreement with you here. How do you feel about me creating a
followup jira to change that to 1-to-1?
My thoughts were: both cases, (A) singleton per JVM and (B) per-S3AFileSystem
are interesting.
I was concerned about wasting things like memory and connection pools. I'll go
study the FileSystem cache now (i.e. does it guarantee one instance
per-bucket, or get close to that?)
I started with (A) because it is an easy starting point that we can start
integration against. However, I think we want to be able to treat DynamoDB
tables as 1-to-1 with S3 buckets (makes auto-provisioning much saner, for
example). As an aside, (A) has actually helped me catch more bugs when getting
all the integration tests working (hint: they leave behind only some data, so
if delete is not perfect, all sorts of things break since my singleton lives
the whole time. Also, having different buckets hit the same MetadataStore
helped me harden treatment of Paths as a side effect).
Let's assume for sake of discussion that (B) is what most people want. As an
author of LocalMetadataStore, who wants to use this as an ephemeral directory
entry cache for certain cases, couldn't I implement my own singleton behind the
scenes? If for some reason, I wanted a single shared memory pool across
S3AFileSystem's.
In short, I see no reason not to replace the singleton allocation with a
per-S3AFileSystem.
> S3Guard: S3AFileSystem Integration with MetadataStore
> -----------------------------------------------------
>
> Key: HADOOP-13651
> URL: https://issues.apache.org/jira/browse/HADOOP-13651
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13651-HADOOP-13345.001.patch,
> HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch
>
>
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata
> consistency and caching.
> Implementation should have minimal overhead when no MetadataStore is
> configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
