Rajesh Balamohan edited comment on HIVE-14776 at 9/19/16 12:51 AM:

Have you tried with "--hiveconf fs.trash.interval=0" setting if you would like 
to avoid sending to trash folder (In HDFS it is much cheaper operation, but in 
S3 this can be expensive based on the amount of data being moved to trash.)?

S3A writes to localsystem and does a upload only at the end of close() call. So 
internally it incurs 2 copies. There is fast S3A outputstream which tries to 
write data to buffer and stream it, but that has got its own set of issues and 
often OOMs due to memory management issues. HADOOP-13560 tries to address large 
file uploads. 

was (Author: rajesh.balamohan):
Have you tried with "--hiveconf fs.trash.interval=0" setting?

> Skip 'distcp' call when copying data from HDSF to S3
> ----------------------------------------------------
>                 Key: HIVE-14776
>                 URL: https://issues.apache.org/jira/browse/HIVE-14776
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>         Attachments: HIVE-14776.1.patch, HIVE-14776.2.patch
> Hive uses 'distcp' to copy files in parallel between HDFS encryption zones 
> when the {{hive.exec.copyfile.maxsize}} threshold is lower than the file to 
> copy. This 'distcp' is also executed when copying to S3, but it is causing 
> slower copies.
> We should not invoke distcp when copying to blobstore systems.

This message was sent by Atlassian JIRA

Reply via email to