[
https://issues.apache.org/jira/browse/ARROW-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bb updated ARROW-7805:
----------------------
Description:
The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option (which
is _not_ the Hadoop default behavior). This turned out to be a pretty major
issue for a project. The HadoopFileSystem `delete` method currently has a
default behavior of `recursive=False` I would believe a similar
`skipTrash=False` default would be appropriate here.
{code:java}
# test using hadoop fs shell commands
# setup test & confirm that file exists
$ testfile="/user/myusername/testfile1"
$ hadoop fs -touchz $testfile && hadoop fs -ls $testfile
-rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/myusername/testfile1
# remove the file and confirm that it is moved to the Trash
$ hadoop fs -rm $testfile
20/02/08 13:26:04 INFO fs.TrashPolicyDefault: Moved:
'hdfs://nameservice1/user/myusername/testfile1' to trash at:
hdfs://nameservice1/user/.Trash/myusername/Current/user/myusername/testfile1
# verify that it is in the Trash
$ hadoop fs -ls /user/.Trash/myusername/Current/user/myusername/testfile1
-rw-r----- 3 myusername mygroup 0 2020-02-08 13:25
/user/.Trash/myusername/Current/user/myusername/testfile1
{code}
{code:java}
{code}
was:
The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option (which
is _not_ the Hadoop default behavior). This turned out to be a pretty major
issue for a project. The HadoopFileSystem `delete` method currently has a
default behavior of `recursive=False` I would believe a similar
`skipTrash=False` default would be appropriate here.
{code:java}
# test using hadoop fs shell commands
# setup test & confirm that file exists
$ testfile="/user/myusername/testfile1"
$ hadoop fs -touchz $testfile && hadoop fs -ls $testfile
-rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/myusername/testfile1
# remove the file and confirm that it is moved to the Trash
$ hadoop fs -rm $testfile
20/02/08 13:26:04 INFO fs.TrashPolicyDefault: Moved:
'hdfs://nameservice1/user/myusername/testfile1' to trash at:
hdfs://nameservice1/user/.Trash/myusername/Current/user/myusername/testfile1
# verify that it is in the Trash
$ hadoop fs -ls /user/.Trash/myusername/Current/user/myusername/testfile1
-rw-r----- 3 myusername mygroup 0 2020-02-08 13:25
/user/.Trash/myusername/Current/user/myusername/testfile1
{code}
> Apache Arrow HDFS Remove (rm) operation defaults to SkipTrash
> -------------------------------------------------------------
>
> Key: ARROW-7805
> URL: https://issues.apache.org/jira/browse/ARROW-7805
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 0.13.0
> Reporter: bb
> Priority: Major
>
> The Pyarrow lib (using LIBHDFS) appears to default to a SkipTrash option
> (which is _not_ the Hadoop default behavior). This turned out to be a pretty
> major issue for a project. The HadoopFileSystem `delete` method currently
> has a default behavior of `recursive=False` I would believe a similar
> `skipTrash=False` default would be appropriate here.
>
> {code:java}
> # test using hadoop fs shell commands
> # setup test & confirm that file exists
> $ testfile="/user/myusername/testfile1"
> $ hadoop fs -touchz $testfile && hadoop fs -ls $testfile
> -rw-r----- 3 myusername mygroup 0 2020-02-08 13:25 /user/myusername/testfile1
> # remove the file and confirm that it is moved to the Trash
> $ hadoop fs -rm $testfile
> 20/02/08 13:26:04 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://nameservice1/user/myusername/testfile1' to trash at:
> hdfs://nameservice1/user/.Trash/myusername/Current/user/myusername/testfile1
> # verify that it is in the Trash
> $ hadoop fs -ls /user/.Trash/myusername/Current/user/myusername/testfile1
> -rw-r----- 3 myusername mygroup 0 2020-02-08 13:25
> /user/.Trash/myusername/Current/user/myusername/testfile1
> {code}
> {code:java}
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
