[
https://issues.apache.org/jira/browse/HDFS-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200597#comment-17200597
]
Uma Maheswara Rao G edited comment on HDFS-15592 at 9/23/20, 6:38 AM:
----------------------------------------------------------------------
{code:java}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{code}
was (Author: umamaheswararao):
{noformat}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt -->
hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at
org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by:
java.io.IOException: Couldn't run retriable-command: Copying
hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more Caused by: java.lang.UnsupportedOperationException: This
API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{noformat}
> DistCP fails with ViewHDFS if the actual target path is non HDFS
> ----------------------------------------------------------------
>
> Key: HDFS-15592
> URL: https://issues.apache.org/jira/browse/HDFS-15592
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, ViewHDFS
> Affects Versions: 3.4.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
>
> When we configure target path mount point with Ozone (or any other fs),
> distcp will fail.
> The reason is, if the src path having ec policy enabled, it will try to
> retain that properties.SO, in this case it is using DFS specific createFile
> API.
> But here we have to ensure, tareget path can from non hdfs in ViewHDFS case.
> In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece
> of code.
>
> {code:java}
> if (preserveEC && sourceStatus.isErasureCoded()
> && sourceStatus instanceof HdfsFileStatus
> && targetFS instanceof DistributedFileSystem) {
> ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
> }{code}
>
> Here it's just checking targetFs instanceof DistributedFileSystem, but in
> ViewHDFS case, fs will be DFS only but actual target can point to mounted fs.
> So, to handle this case, we should use resolvePath API and check the resolved
> target path scheme is dfs or or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
