[
https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622912#comment-15622912
]
Steve Loughran commented on HADOOP-13651:
-----------------------------------------
+1 with an instance per FS instance. S3A FS instances are becoming more
expensive; with a thread pool for uploads, soon one for copy and metadata
operations; adding a metadata store may make them more expensive.
But
* an instance with no metadata is no more expensive than now
* when all filesystems for a user are released, their resources get cleaned up.
This matters in things like hive, which call {{FileSystem.closeAllForUGI(ugi)}}
to release the resources after fielding a user's request.
If there is trouble, and it's important to be ready for, is that if two users
connect to the same bucket in separate RPC calls, they are going to end up with
separate FS instances, hence separate MD stores. When using dynamo backed
stores it's (probably) moot, but for local stores, it's going to complicate
things. If one caller modifies the state, the other will not pick it up. But if
you shared the store, then a user without write permission may be able to
manipulate the metadata seen by the other (at least if a delete() goes through
on the MD before the FS permissions are checked)
This raises another question: what does happen with security here?
> S3Guard: S3AFileSystem Integration with MetadataStore
> -----------------------------------------------------
>
> Key: HADOOP-13651
> URL: https://issues.apache.org/jira/browse/HADOOP-13651
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13651-HADOOP-13345.001.patch,
> HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch
>
>
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata
> consistency and caching.
> Implementation should have minimal overhead when no MetadataStore is
> configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
