[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---------------------------
Description:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
was:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
> DistCp: distcp fail when copying to ftp filesystem
> --------------------------------------------------
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
> Affects Versions: 3.3.0
> Reporter: jerry
> Assignee: jerry
> Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_000000_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> 'distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
