MisterT314 opened a new issue, #38421: URL: https://github.com/apache/arrow/issues/38421
### Describe the bug, including details regarding any error messages, version, and platform. Hi, I'm initializing S3FileSystem with a role_arn, to get refreshable temporary credentials, as described in https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html The init call is the following: ``` self._fs = S3FileSystem( region=_AWS_REGION, role_arn=self._role_arn, anonymous=False, ) ``` A bit later I'm initializing a parquetwriter with the file system ``` fs = _get_service().get_file_system() output_url = output_url.replace("s3://", "") schema = Schema.from_pandas(df=data) writer = ParquetWriter(where=output_url, schema=schema, filesystem=fs) ``` I then get the error: OSError: When initiating multiple part upload for key 'resources/01HBTZN7DBR1BE62S5TJRQDZXP/data.parquet' in bucket 'my-bucket-name': AWS Error ACCESS_DENIED during CreateMultipartUpload operation: Anonymous users cannot initiate multipart uploads. Please authenticate. I expect that S3FileSystem would use STS to get temporary credentials like described in the documentation. The role is configured to allow the service (glue) to assume the role. Pyarrow version 12.0.0 S3FS 2023.6.0 Using Pyarrow in a aws glue v4 environment ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
