Shawn Li created ARROW-8435: ------------------------------- Summary: A TypeError is raised while token expires while writing to S3 Key: ARROW-8435 URL: https://issues.apache.org/jira/browse/ARROW-8435 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.15.1 Reporter: Shawn Li
This issue only occurs when a STS token expires *in the middle of* writing to S3. An OSError: Write failed: TypeError("'NoneType' object is not subscriptable",) is raised instead of a PermissionError. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1450, in write_to_dataset write_table(subtable, f, **kwargs) File "/usr/local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1344, in write_table writer.write_table(table, row_group_size=row_group_size) File "/usr/local/lib/python3.6/site-packages/pyarrow/parquet.py", line 474, in write_table self.writer.write_table(table, row_group_size=row_group_size) File "pyarrow/_parquet.pyx", line 1375, in pyarrow._parquet.ParquetWriter.write_table File "pyarrow/error.pxi", line 80, in pyarrow.lib.check_statuspyarrow.lib.ArrowIOError: Arrow error: IOError: The provided token has expired.. Detail: Python exception: PermissionError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/s3fs/core.py", line 1096, in _upload_chunk PartNumber=part, UploadId=self.mpu['UploadId'],TypeError: 'NoneType' object is not subscriptable environment is: s3fs==0.4.0 boto3==1.10.27 botocore==1.13.27 pyarrow==0.15.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)