Hi Yushu, This would certainly be a useful contribution. Unfortunately, I don’t see any possible workaround …
I’m happy to support you on a PR there if needed. In any case, adding a nullable pipeline option to S3Options and using it in S3FileSystem / S3WritableByteChannel should be straight forward. Regards, Moritz From: Alexey Romanenko <[email protected]> Date: Thursday, 10. March 2022 at 11:09 To: [email protected] <[email protected]> Cc: Moritz Mack <[email protected]> Subject: Re: Write S3 File with CannedACL The contributions are very welcome! So, if you decide to go forward with this, please, take a look on these guides [1][2]. In short, I may assign this jira [3] to you and once you have a fix then just submit a PR and ask for review by pinging ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Exercise caution when opening attachments or clicking any links. ZjQcmQRYFpfptBannerEnd The contributions are very welcome! So, if you decide to go forward with this, please, take a look on these guides [1][2]. In short, I may assign this jira [3] to you and once you have a fix then just submit a PR and ask for review by pinging me or Moritz (cc). Please, don’t hesitate to ask any questions if you have. — Alexey [1] https://beam.apache.org/contribute/<https://urldefense.com/v3/__https:/beam.apache.org/contribute/__;!!CiXD_PY!H5Op6mpGxwZY2Q7nd3mwMYrahGtq5w4Hv46dPWN5ozt5zAIDEKyQ5v9ArvqB$> [2] https://cwiki.apache.org/confluence/display/BEAM/Developer+Guides<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/BEAM/Developer*Guides__;Kw!!CiXD_PY!H5Op6mpGxwZY2Q7nd3mwMYrahGtq5w4Hv46dPWN5ozt5zAIDEKyQ5nacs1kw$> [3] https://issues.apache.org/jira/browse/BEAM-10850<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/BEAM-10850__;!!CiXD_PY!H5Op6mpGxwZY2Q7nd3mwMYrahGtq5w4Hv46dPWN5ozt5zAIDEKyQ5nEh1Uuw$> On 9 Mar 2022, at 19:33, Yushu Yao <[email protected]<mailto:[email protected]>> wrote: As long as we can use AvroIO to save files to "s3://xx" we are fine. Looks like the JIRA has been around for a while. What is the procedure to contribute from our end? Thanks On Wed, Mar 9, 2022 at 9:59 AM Alexey Romanenko <[email protected]<mailto:[email protected]>> wrote: Hi Yushu, I’m not sure that we have a workaround for that since the related jira issue [1] is still open. Side question: are you interested only in multipart version or both? — Alexey [1] https://issues.apache.org/jira/browse/BEAM-10850<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/BEAM-10850__;!!CiXD_PY!H5Op6mpGxwZY2Q7nd3mwMYrahGtq5w4Hv46dPWN5ozt5zAIDEKyQ5nEh1Uuw$> On 9 Mar 2022, at 00:19, Yushu Yao <[email protected]<mailto:[email protected]>> wrote: Hi Team, We have a use case that needs to add `--acl bucket-owner-full-control` whenever we want to upload a file to S3. So if we want to use aws cli, it will be: aws s3 cp myfile s3://bucket/path/file --acl bucket-owner-full-control So to do it in java code, we use (assuming aws s3 sdk v1): InitiateMultipartUploadRequest.withCannedACL(CannedAccessControlList.BucketOwnerFullControl) In the beam code below it seems cannedACL is not supported: https://github.com/apache/beam/blob/95a5c26c4c6d0a1c4155ad209b61e623781c47df/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java#L485<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/95a5c26c4c6d0a1c4155ad209b61e623781c47df/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java*L485__;Iw!!CiXD_PY!H5Op6mpGxwZY2Q7nd3mwMYrahGtq5w4Hv46dPWN5ozt5zAIDEKyQ5jUl_Rhh$> It's similar in the s3 sdk v2 beam code as well. Wondering how do I work around this? Thanks a lot! -Yushu
