[ https://issues.apache.org/jira/browse/HADOOP-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675308#comment-15675308 ]
Ravi Prakash commented on HADOOP-8065: -------------------------------------- Hi Yongjun! Thanks for rebasing the patch and your polishing touches. I think HADOOP-13114 might be the more appropriate JIRA for these changes (which Suraj kindly filed at my request earlier.) since this patch does not compress *during* transfer; only after transfer and before writing to HDFS. - {{getCompressionCodcec}} has the same typo I pointed out to Suraj. He did post updated patches on HADOOP-13114. I apologize for neglecting to review those patches despite Suraj's requests. - {{getCompressionCodcec}} also uses ReflectionUtils. I don't know if it'd be better to use [this pattern|https://github.com/apache/hadoop/blob/b4f1971ff1dd578353036d7a123fe83c27c1e803/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java#L159] instead? - We're still not using a CodecPool like I suggested earlier. The patch in HADOOP-13114 actually is. Let me rebase and upload that. Could you please take a look at that? > distcp should have an option to compress data while copying. > ------------------------------------------------------------ > > Key: HADOOP-8065 > URL: https://issues.apache.org/jira/browse/HADOOP-8065 > Project: Hadoop Common > Issue Type: Improvement > Components: fs > Affects Versions: 0.20.2 > Reporter: Suresh Antony > Assignee: Suraj Nayak > Priority: Minor > Labels: distcp > Fix For: 0.20.2 > > Attachments: HADOOP-8065-trunk_2015-11-03.patch, > HADOOP-8065-trunk_2015-11-04.patch, HADOOP-8065-trunk_2016-04-29-4.patch, > HADOOP-8065.005.patch, HADOOP-8065.006.patch, patch.distcp.2012-02-10 > > > We would like compress the data while transferring from our source system to > target system. One way to do this is to write a map/reduce job to compress > that after/before being transferred. This looks inefficient. > Since distcp already reading writing data it would be better if it can > accomplish while doing this. > Flip side of this is that distcp -update option can not check file size > before copying data. It can only check for the existence of file. > So I propose if -compress option is given then file size is not checked. > Also when we copy file appropriate extension needs to be added to file > depending on compression type. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org