[ https://issues.apache.org/jira/browse/HADOOP-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014095#comment-14014095 ]
Ken Krugler commented on HADOOP-8065: ------------------------------------- I think this is reasonable functionality to add to distcp. For reference (based on user input) see what Amazon has added to their version of distcp (S3distcp): http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html They support a --outputCodec <codec> parameter, to specify what compression to use. > discp should have an option to compress data while copying. > ----------------------------------------------------------- > > Key: HADOOP-8065 > URL: https://issues.apache.org/jira/browse/HADOOP-8065 > Project: Hadoop Common > Issue Type: Improvement > Components: fs > Affects Versions: 0.20.2 > Reporter: Suresh Antony > Priority: Minor > Labels: distcp > Fix For: 0.20.2 > > Attachments: patch.distcp.2012-02-10 > > > We would like compress the data while transferring from our source system to > target system. One way to do this is to write a map/reduce job to compress > that after/before being transferred. This looks inefficient. > Since distcp already reading writing data it would be better if it can > accomplish while doing this. > Flip side of this is that distcp -update option can not check file size > before copying data. It can only check for the existence of file. > So I propose if -compress option is given then file size is not checked. > Also when we copy file appropriate extension needs to be added to file > depending on compression type. -- This message was sent by Atlassian JIRA (v6.2#6252)