SammiChen commented on HADOOP-14999:

Hi [~uncleGen], some comments, 

Preconditions.checkArgument(v >= min,
    String.format("Value of %s: %d is below the minimum value %d",
        key, v, min));

Ignore the comment. String.format with %d is OK. 

bq.  Asynchronous multi-part based uploading mechanism to support huge file* 
which is larger than 5GB.

Please give a detail explain about where does 5GB threshold comes from?

 if (partSize < MULTIPART_MIN_SIZE) {
      LOG.warn("{} must be at least 5 MB; configured value is {}",
              property, partSize);
      partSize = MULTIPART_MIN_SIZE;

MULTIPART_MIN_SIZE is 100K. The threshold in warning message is 5 MB.

bq.  long partSize = AliyunOSSUtils.getMultipartSizeProperty(getConf(), 

 can we use uploadPartSize instead here?

   bq.  I also add the resource clean logic in try-finally
     try {
      blockStream.write(b, off, len);
      blockWritten += len;
      if (blockWritten >= blockSize) {
        blockWritten = 0L;
    } finally {
      for (File tFile: blockFiles) {
        if (tFile.exists() && !tFile.delete()) {
          LOG.warn("Failed to delete temporary file {}", tFile);
  I see you add the temp file delete in finally no matter the above operation 
succeeds or not .  When store.uploadPart() returns, is the upload finished 
already?  If it's 
  an async operation, delete the temp file in normal case may have trouble. 

6. the performance data looks good. 

> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
> ------------------------------------------------------------------------
>                 Key: HADOOP-14999
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14999
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/oss
>    Affects Versions: 3.0.0-beta1
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>            Priority: Major
>         Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch, 
> HADOOP-14999.003.patch, HADOOP-14999.004.patch, HADOOP-14999.005.patch, 
> HADOOP-14999.006.patch, HADOOP-14999.007.patch, HADOOP-14999.008.patch, 
> HADOOP-14999.009.patch, asynchronous_file_uploading.pdf, 
> diff-between-patch7-and-patch8.txt
> This mechanism is designed for uploading file in parallel and asynchronously:
>  - improve the performance of uploading file to OSS server. Firstly, this 
> mechanism splits result to multiple small blocks and upload them in parallel. 
> Then, getting result and uploading blocks are asynchronous.
>  - avoid buffering too large result into local disk. To cite an extreme 
> example, there is a task which will output 100GB or even larger, we may need 
> to output this 100GB to local disk and then upload it. Sometimes, it is 
> inefficient and limited to disk space.
> This patch reuse {{SemaphoredDelegatingExecutor}} as executor service and 
> depends on HADOOP-15039.
> Attached {{asynchronous_file_uploading.pdf}} illustrated the difference 
> between previous {{AliyunOSSOutputStream}} and 
> {{AliyunOSSBlockOutputStream}}, i.e. this asynchronous multi-part based 
> uploading mechanism.
> 1. {{AliyunOSSOutputStream}}: we need to output the whole result to local 
> disk before we can upload it to OSS. This will poses two problems´╝Ü
>  - if the output file is too large, it will run out of the local disk.
>  - if the output file is too large, task will wait long time to upload result 
> to OSS before finish, wasting much compute resource.
> 2. {{AliyunOSSBlockOutputStream}}: we cut the task output into small blocks, 
> i.e. some small local file, and each block will be packaged into a uploading 
> task. These tasks will be submitted into {{SemaphoredDelegatingExecutor}}. 
> {{SemaphoredDelegatingExecutor}} will upload this blocks in parallel, this 
> will improve performance greatly.
> 3. Each task will retry 3 times to upload block to Aliyun OSS. If one of 
> those tasks failed, the whole file uploading will failed, and we will abort 
> current uploading.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to