[ https://issues.apache.org/jira/browse/ASTERIXDB-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838870#comment-17838870 ]
ASF subversion and git services commented on ASTERIXDB-3378: ------------------------------------------------------------ Commit ab0abc111eaa663f9115756ba4e7cbf1da1c5f3b in asterixdb's branch refs/heads/master from Hussain Towaileb [ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=ab0abc111e ] [ASTERIXDB-3378][EXT]: Allow user to opt out of validating write permission before COPY TO S3 Change-Id: Ie099710714e032674727d0cb426cc450d8dda167 Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234 Integration-Tests: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Reviewed-by: Hussain Towaileb <hussai...@gmail.com> > Allow user to opt out of validating write permission before COPY TO S3 > ---------------------------------------------------------------------- > > Key: ASTERIXDB-3378 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3378 > Project: Apache AsterixDB > Issue Type: Task > Components: EXT - External data > Affects Versions: 0.9.9 > Reporter: Hussain Towaileb > Assignee: Hussain Towaileb > Priority: Major > Fix For: 0.9.9 > > > Currently, for COPY TO S3, one of our early checks validates that: > * We have the write permission, by writing a small test file to the > destination, then deleting it, which also tests that > * Multipart upload is supported, as this is the form of operation we use for > writing. > > This test helps identify failure scenarios early before processing (for > example) large amount of data, only to fail eventually. > The limitation of this validation is that it is not always possible to know > the final destination we will be writing to, depending on the statement > itself. And so, we always perform this test at the root of the bucket. > According to the S3 docs, it seems it might be possible to grant the > permissions only on specific prefixes in a bucket, but not all. This can be > problematic if we have permission on our destination prefix, but not the root > of the bucket, because the validation will keep trying to validate against > the root, which will lead to a failure of the operation. > This ticket suggests the addition of a flag, e.g. > {*}"validate-write-permission" (default to true){*}, to skip this operation > and let the user carry the responsibility of having the right permissions at > the right time. Note this will only skip the write permission validation, all > other validations will still take place. -- This message was sent by Atlassian Jira (v8.20.10#820010)