[
https://issues.apache.org/jira/browse/HADOOP-16492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252843#comment-17252843
]
Steve Loughran commented on HADOOP-16492:
-----------------------------------------
bq. There are some optimizations like re-using the existing s3a
implementation, common utilities.. and doc updates which can be done separately
as this patch huge.
I'm a bit nervous there as I only worry about the hadoop-aws codebase and
having a different module depend on things there is doomed.
Anything to be shared should go into Hadoop-common and be split into a "pull up
from hadoop-aws with tests" and "use in the huawei cloud connector".
* Can this patch go in to github for a review?
Can I also point out"
https://github.com/steveloughran/engineering-proposals/blob/trunk/refactoring-s3a.md
That's my critique of what S3 has gradually evolved into. What was simple at
the beginning "we will have one class and just invoke other FS APIs", has
become "we have to pass an S3AFileSystem ref around everywhere and can't track
what changes will break other bits"
I'm trying to improve stuff one PR at a time. Now that S3 is consistent and
S3Guard no longer needed, this gets a lot, lot simpler: we probably can do a
layer model of
# Public FileSystem APIs (FileSystem, MultipartUploader. ...)
# object level view of world
# AWS SDK operation layer
+ interfaces to reference back to common code (executor pool, statistics)
Just because S3A does something doesn't mean that it's the right design
> Support HuaweiCloud Object Storage as a Hadoop Backend File System
> ------------------------------------------------------------------
>
> Key: HADOOP-16492
> URL: https://issues.apache.org/jira/browse/HADOOP-16492
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 3.4.0
> Reporter: zhongjun
> Assignee: zhongjun
> Priority: Major
> Attachments: Difference Between OBSA and S3A.pdf,
> HADOOP-16492.001.patch, HADOOP-16492.002.patch, HADOOP-16492.003.patch,
> HADOOP-16492.004.patch, HADOOP-16492.005.patch, HADOOP-16492.006.patch,
> HADOOP-16492.007.patch, HADOOP-16492.008.patch, HADOOP-16492.009.patch,
> HADOOP-16492.010.patch, HADOOP-16492.011.patch, HADOOP-16492.012.patch,
> HADOOP-16492.013.patch, HADOOP-16492.014.patch, HADOOP-16492.015.patch,
> HADOOP-16492.016.patch, HADOOP-16492.017.patch, OBSA HuaweiCloud OBS Adapter
> for Hadoop Support.pdf, image-2020-11-21-18-51-51-981.png
>
>
> Added support for HuaweiCloud OBS
> ([https://www.huaweicloud.com/en-us/product/obs.html]) to Hadoop file system,
> just like what we do before for S3, ADLS, OSS, etc. With simple
> configuration, Hadoop applications can read/write data from OBS without any
> code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
