[
https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376521#comment-15376521
]
uncleGen commented on HADOOP-12756:
-----------------------------------
Glad to see the progress, I have also do some works about supporting OSS in
Hadoop. There are some stability problems which we should pay attention to,
include but not limited to:
1. OSS will close long-time connection(> 3h) and idle connection(>1minute),
while it is pretty common.
2. The 'copy' operation is time-consuming, so we could use the existing
Job/Task executing logic, i.e. copy temp result from temp directory to final
directory.
I will open anonther JIRA to fix above-mentioned issue based on your work.
What is your suggestions? [~shimingfei], [~drankye]
> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>
> Key: HADOOP-12756
> URL: https://issues.apache.org/jira/browse/HADOOP-12756
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 2.8.0, HADOOP-12756
> Reporter: shimingfei
> Assignee: shimingfei
> Fix For: HADOOP-12756
>
> Attachments: HADOOP-12756-v02.patch, HADOOP-12756.003.patch,
> HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, HCFS
> User manual.md, OSS integration.pdf, OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not
> easy to access data laid on OSS storage from user’s Hadoop/Spark application,
> because of no original support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration,
> Spark/Hadoop applications can read/write data from OSS without any code
> change. Narrowing the gap between user’s APP and data storage, like what have
> been done for S3 in Hadoop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
