[
https://issues.apache.org/jira/browse/HADOOP-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590167#comment-15590167
]
Aaron Fabbri commented on HADOOP-13736:
---------------------------------------
Chatted w/ [~eddyxu] a bit offline, paraphrasing here: He mentioned dealing
with FileStatus creates extra work for DynamoDB and maybe CLI as well. We
agreed that:
- DynamoDBMetadataStore can only support S3A. It can assert at initialize()
that the fs is S3AFileSystem,
- It can ignore FileStatus fields that are not used by S3A. We'll modify the
MetadataStore contract tests to take this into account.
- In general, it should only do minimal work to support S3a. No other fs
clients are supported.
I was thinking the only impact to keeping FileStatus is just a cast to
S3AFileStatus at the beginning of the public MetadataStore functions. (Let me
know if there are more difficulties). We agreed that casting is not that
great, but current S3A code should not have any non-S3A FileStatus in its code
paths. I originally tried to make the subtype of FileStatus a type parameter
but as [~cnauroth] mentioned, that doesn't play nicely with reflection-based
instantiation. (Open to ideas on this so we can get better type checking).
It sounds like there will be a little extra work in CLI as well, to keep this
FileStatus. I.e. optionally showing the extra fields that subclasses of
FileStatus might add (e.g. S3AFileStatus.isEmptyDirectory).
Overall I like the idea of keeping the MetadataStore interface FS-agnostic. I
was hoping we could easily move it to its own package and use it for other FS
clients in the future. [[email protected]] and [~vishwajeet.dusane] also
indicated they were interested in using LocalMetadataStore for a ephemeral
listing cache in HADOOP-12876. Maybe they can chime in if this is not still
interesting.
I want to be sensitive to any pain you guys are feeling on DynamoDB and/or CLI
implementation, so I'm hoping we can maintain the FS separation, but I want to
be flexible if needed. Thanks!
> Change PathMetadata to hold S3AFileStatus instead of FileStatus.
> ----------------------------------------------------------------
>
> Key: HADOOP-13736
> URL: https://issues.apache.org/jira/browse/HADOOP-13736
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
>
> {{S3AFileStatus}} is implemented differently with {{FileStatus}}, for
> instance {{S3AFileStatus#isEmptyDirectory()}} is not implemented in
> {{FileStatus()}}. And {{access_time}}, {{block_replication}}, {{owner}},
> {{group}} and a few other fields are not meaningful in {{S3AFileStatus}}.
> So in the scope of {{S3guard}}, it should use {{S3AFileStatus}} in instead
> of {{FileStatus}} in {{PathMetadaa}} to avoid casting the types back and
> forth in S3A.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
