Hussain Towaileb created ASTERIXDB-3378:
-------------------------------------------
Summary: Allow user to opt out of validating write permission
before COPY TO S3
Key: ASTERIXDB-3378
URL: https://issues.apache.org/jira/browse/ASTERIXDB-3378
Project: Apache AsterixDB
Issue Type: Improvement
Components: EXT - External data
Affects Versions: 0.9.9
Reporter: Hussain Towaileb
Assignee: Hussain Towaileb
Fix For: 0.9.9
Currently, for COPY TO S3, one of our early checks validates that:
* We have the write permission, by writing a small test file to the
destination, then deleting it, which also tests that
* Multipart upload is supported, as this is the form of operation we use for
writing.
This test helps identify failure scenarios early before processing (for
example) large amount of data, only to fail eventually.
The limitation of this validation is that it is not always possible to know the
final destination we will be writing to, depending on the statement itself. And
so, we always perform this test at the root of the bucket.
According to the S3 docs, it seems it might be possible to grant the
permissions only on specific prefixes in a bucket, but not all. This can be
problematic if we have permission on our destination prefix, but not the root
of the bucket, because the validation will keep trying to validate against the
root, which will lead to a failure of the operation.
This ticket suggests the addition of a flag, e.g.
{*}"validate-write-permission" (default to true){*}, to skip this operation and
let the user carry the responsibility of having the right permissions at the
right time. Note this will only skip the write permission validation, all other
validations will still take place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
