[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-14999:
-------------------------------
Attachment: HADOOP-14999.003.patch
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
> ------------------------------------------------------------------------
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
> Affects Versions: 3.0.0-beta1
> Reporter: Genmao Yu
> Assignee: Genmao Yu
> Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch,
> HADOOP-14999.003.patch, asynchronous_file_uploading.pdf
>
>
> This mechanism is designed for uploading file in parallel and asynchronously:
> - improve the performance of uploading file to OSS server. Firstly, this
> mechanism splits result to multiple small blocks and upload them in parallel.
> Then, getting result and uploading blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme
> example, there is a task which will output 100GB or even larger, we may need
> to output this 100GB to local disk and then upload it. Sometimes, it is
> inefficient and limited to disk space.
> This patch reuse {{SemaphoredDelegatingExecutor}} as executor service and
> depends on HADOOP-15039.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
