[
https://issues.apache.org/jira/browse/HADOOP-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138461#comment-18138461
]
ASF GitHub Bot commented on HADOOP-20064:
-----------------------------------------
LuciferYang opened a new pull request, #8928:
URL: https://github.com/apache/hadoop/pull/8928
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-20064
The recursive `FileUtil#copy` overloads (FileSystem to FileSystem, File to
FileSystem, FileSystem to File) ignore the result of copying each directory
entry. An entry returns false when `mkdirs` for its destination returns false,
for example on a local directory without write permission. The parent then
returns true and, with `deleteSource=true`, deletes the whole source tree. For
example, `moveToLocalFile` of an HDFS directory that only contains
subdirectories into an existing, read-only local directory copies nothing and
permanently deletes the HDFS directory. `LocalFileSystem#moveFromLocalFile` and
the copy fallback of `RawLocalFileSystem#rename` use the same code.
This patch keeps copying the other entries, remembers the failure, and
returns false before deleting the source directory, like `copy(Path[])` in the
same class. Entries that were copied are still moved; the failed one and the
directory are kept. Callers with `deleteSource=false` still get every other
entry copied, and now see false instead of true. The same applies when deleting
a copied entry's source returns false: the parent directory is kept and false
is returned.
### How was this patch tested?
Added one test per overload in `TestFileUtil`, each with a source directory
where one child fails:
- `testCopyChildMkdirsFailure` (FileSystem to FileSystem) and
`testCopyFromLocalChildMkdirsFailure` (File to FileSystem): the mocked
destination fails `mkdirs` for whichever child comes first. The copy returns
false, the other child is moved, and the failed one stays in the source.
- `testCopyToLocalChildMkdirsFailure` (FileSystem to File): the mocked
source lists a directory name twice, so the second local `mkdirs` returns
false. The copy returns false, the later sibling is copied, and the source
directory is not deleted.
All three fail without the fix. They do not depend on directory listing
order.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
- [x] The PR includes the phrase "Contains content generated by Claude Opus".
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Opus.
> FileUtil#copy ignores a failed child copy and deletes the source directory
> when deleteSource is true
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20064
> URL: https://issues.apache.org/jira/browse/HADOOP-20064
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Yang Jie
> Priority: Major
>
> The recursive FileUtil#copy overloads (FileSystem to FileSystem, File to
> FileSystem, FileSystem to File) copy each entry of a directory and ignore the
> boolean that the copy of the entry returns. An entry returns false when
> creating its destination directory fails without an exception, for example
> when mkdirs returns false for permission denied on the local file system, or
> when an FTP server refuses it. The parent goes on, returns true and, with
> deleteSource=true, deletes the whole source directory.
> Example: {{hdfs.moveToLocalFile(/warehouse/t, /data/export)}} where
> /data/export/t already exists and is not writable, and /warehouse/t only
> contains partition directories. Every child mkdirs fails, nothing is copied,
> and /warehouse/t is deleted permanently (delete does not use the trash). The
> same code is used by LocalFileSystem#moveFromLocalFile and by the copy
> fallback of RawLocalFileSystem#rename.
> The fix copies the remaining entries, remembers the failure, and returns
> false before deleting the source directory, like copy(Path[]) in the same
> class does for a failed source. Entries that were copied are still moved; the
> failed entry and the directory itself are kept.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
