[
https://issues.apache.org/jira/browse/ASTERIXDB-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838870#comment-17838870
]
ASF subversion and git services commented on ASTERIXDB-3378:
------------------------------------------------------------
Commit ab0abc111eaa663f9115756ba4e7cbf1da1c5f3b in asterixdb's branch
refs/heads/master from Hussain Towaileb
[ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=ab0abc111e ]
[ASTERIXDB-3378][EXT]: Allow user to opt out of validating write permission
before COPY TO S3
Change-Id: Ie099710714e032674727d0cb426cc450d8dda167
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234
Integration-Tests: Jenkins <[email protected]>
Tested-by: Jenkins <[email protected]>
Reviewed-by: Hussain Towaileb <[email protected]>
> Allow user to opt out of validating write permission before COPY TO S3
> ----------------------------------------------------------------------
>
> Key: ASTERIXDB-3378
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3378
> Project: Apache AsterixDB
> Issue Type: Task
> Components: EXT - External data
> Affects Versions: 0.9.9
> Reporter: Hussain Towaileb
> Assignee: Hussain Towaileb
> Priority: Major
> Fix For: 0.9.9
>
>
> Currently, for COPY TO S3, one of our early checks validates that:
> * We have the write permission, by writing a small test file to the
> destination, then deleting it, which also tests that
> * Multipart upload is supported, as this is the form of operation we use for
> writing.
>
> This test helps identify failure scenarios early before processing (for
> example) large amount of data, only to fail eventually.
> The limitation of this validation is that it is not always possible to know
> the final destination we will be writing to, depending on the statement
> itself. And so, we always perform this test at the root of the bucket.
> According to the S3 docs, it seems it might be possible to grant the
> permissions only on specific prefixes in a bucket, but not all. This can be
> problematic if we have permission on our destination prefix, but not the root
> of the bucket, because the validation will keep trying to validate against
> the root, which will lead to a failure of the operation.
> This ticket suggests the addition of a flag, e.g.
> {*}"validate-write-permission" (default to true){*}, to skip this operation
> and let the user carry the responsibility of having the right permissions at
> the right time. Note this will only skip the write permission validation, all
> other validations will still take place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
