[ https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562859#comment-15562859 ]
Andrew Wang commented on HDFS-10971: ------------------------------------ I wonder what the right distcp behavior is in this case of "-pr". It indicates that the user wants the dest file to be replicated if the src file is replicated, even if a dst parent directory has an EC policy set. I don't think we have a create API that supports that right now. Combinations: * no "-pr" specified: dst file is written with whatever is the default for that destination path, which could be EC or not * "-pr" on a replicated file: dst is written replicated, with the same replication factor * "-pr" on a striped file: Not sure, but I lean toward the same behavior as no "-pr" flag: dst file is written with whatever is the default for that destination path, which could be EC or not. We could then add a new "-p" flag to additionally preserve EC policy. > Distcp should not copy replication factor if source file is erasure coded > ------------------------------------------------------------------------- > > Key: HDFS-10971 > URL: https://issues.apache.org/jira/browse/HDFS-10971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: distcp > Affects Versions: 3.0.0-alpha1 > Reporter: Wei-Chiu Chuang > Assignee: Wei-Chiu Chuang > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-10971.testcase.patch > > > The current erasure coding implementation uses replication factor field to > store erasure coding policy. > Distcp copies the source file's replication factor to the destination if > {{-pr}} is specified. However, if the source file is EC, the replication > factor (which is EC policy) should not be replicated to the destination file. > When a HdfsFileStatus is converted to FileStatus, the replication factor is > set to 0 if it's an EC file. > In fact, I will attach a test case that shows trying to replicate the > replication factor of an EC file results in an IOException: "Requested > replication factor of 0 is less than the required minimum of 1 for > /tmp/dst/dest2" -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org