[ https://issues.apache.org/jira/browse/HADOOP-13377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
uncleGen updated HADOOP-13377: ------------------------------ Description: This work is based on [HADOOP-12756|https://issues.apache.org/jira/browse/HADOOP-12756]. There are some stability problems which we should pay attention to, include but not limited to: 1. OSS will close long-time connection(> 3h) and idle connection(>1minute), while it is pretty common. 2. The 'copy' operation is time-consuming, so we could use the existing Job/Task executing logic, i.e. copy temp result from temp directory to final directory. and some hack optimization: 1. use double buffering and multi-thread when read oss data 2. data is split in chunk and uploaded in ‘multipart’ way was: This work is based on [HADOOP-12756|https://issues.apache.org/jira/browse/HADOOP-12756]. There are some stability problems which we should pay attention to, include but not limited to: 1. OSS will close long-time connection(> 3h) and idle connection(>1minute), while it is pretty common. 2. The 'copy' operation is time-consuming, so we could use the existing Job/Task executing logic, i.e. copy temp result from temp directory to final directory. > Some improvement for incorporating Aliyun OSS file system implementation > ------------------------------------------------------------------------ > > Key: HADOOP-13377 > URL: https://issues.apache.org/jira/browse/HADOOP-13377 > Project: Hadoop Common > Issue Type: Improvement > Components: fs > Affects Versions: 2.8.0, HADOOP-12756 > Reporter: uncleGen > Fix For: HADOOP-12756 > > > This work is based on > [HADOOP-12756|https://issues.apache.org/jira/browse/HADOOP-12756]. > There are some stability problems which we should pay attention to, include > but not limited to: > 1. OSS will close long-time connection(> 3h) and idle connection(>1minute), > while it is pretty common. > 2. The 'copy' operation is time-consuming, so we could use the existing > Job/Task executing logic, i.e. copy temp result from temp directory to final > directory. > and some hack optimization: > 1. use double buffering and multi-thread when read oss data > 2. data is split in chunk and uploaded in ‘multipart’ way -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org