[
https://issues.apache.org/jira/browse/ARROW-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503868#comment-17503868
]
Weston Pace commented on ARROW-15892:
-------------------------------------
I can reproduce this and I agree that the permission in question is probably
{{ListBucket}}. In this case it isn't the logic around checking bucket
existence exactly that is causing the problem. In the datasets module we
always call `CreateDir` since the function succeeds if the directory already
exists.
In s3fs, if the directory is a bucket (e.g. if it is `foo` and not
`foo/my_dataset`), we call `CreateBucket` without doing any existence checking
at all (then if we get a "bucket already exists" error we just return ok).
That explains why the error you are getting is {{"OSError: When creating
bucket"}} and not {{"OSError: When testing for existence of bucket"}}
It should be straightforward to add an option to completely disable `CreateDir`
calls in datasets. This would be a little bizarre from a local filesystem
perspective but makes perfect sense in an object store where directories don't
need to be created. You would also need to make sure you specify
{{existing_data_behavior=overwrite_or_ignore}}
> [C++] Dataset APIs require s3:ListBucket Permissions
> ----------------------------------------------------
>
> Key: ARROW-15892
> URL: https://issues.apache.org/jira/browse/ARROW-15892
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Jonny Fuller
> Priority: Minor
>
> Hi team, first time posting an issue so I apologize if the format is lacking.
> My original comment is on ARROW-13685 Github Issue
> [here|https://github.com/apache/arrow/pull/11136#issuecomment-1062406820].
> Long story short, our environment is super locked down, and while my
> application has permission to write data against an s3 prefix, I do not have
> the {{ListBucket}} permission nor can I add it. This does not prevent me from
> using the "individual" file APIs like {{pq.write_table}} but the bucket
> validation logic in the "dataset" APIs breaks when trying to test for the
> bucket's existence.
> {code:java}
> pq.write_to_dataset(pa.Table.from_batches([data]), location,
> filesystem=s3fs){code}
> {code:java}
> OSError: When creating bucket '<my bucket>': AWS Error [code 15]: Access
> Denied{code}
> The same is true for the generic {{pyarrow.dataset}} APIs. My understanding
> is the bucket validation logic is part of the C++ code, not the Python API.
> As a Pythonista who knows nothing of C++ I am not sure how to resolve this
> problem.
>
> Would it be possible to disable the bucket existence check with an optional
> key word argument? Thank you for your time!
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
