legout opened a new issue, #34363:
URL: https://github.com/apache/arrow/issues/34363
### Describe the bug, including details regarding any error messages,
version, and platform.
When I try to write a pyarrow.table to cloudflare r2 object store, I got an
error when files are larger than x MB (I do not know the exact threshold) and
pyarrow internally switches to multipart uploading.
I´ve used `s3fs.S3FileSystem` (fsspec) and also tried
`pyarrow.fs.S3FileSystem`. Here is some example code:
```python
import pyarrow.fs as pafs
import s3fs
import pyarrow.parquet as pq
import pyarrow as pa
table_small = pq.read_table("small_data.parquet")
table_large = pq.read_table("large_data.parquet")
fs1 = s3fs.S3FileSystem(
key="some_key",
secret="some_secret",
client_kwargs=dict(endpoint_url="https://123456.r2.cloudflarestorage.com"),
s3_additional_kwargs=dict(ACL="private") # <- this is neccesary for
writting.
)
fs2 = pafs.S3FileSystem(
access_key="some_key",
secret_key="some_secret",
endpoint_override="https://123456.r2.cloudflarestorage.com"
)
pq.write_table(table_small, "test/test.parquet", filesystem=fs1) # <- works
pq.write_table(table_small, "test/test.parquet", filesystem=fs2) # <- works
# failed with OSError: [Errno 22] There was a problem with the multipart
upload.
pq.write_table(table_large, "test/test.parquet", filesystem=fs1)
# failed with OSError: When initiating multiple part upload for key
'test.parquet' in bucket 'test': AWS Error NETWORK_CONNECTION during
CreateMultipartUpload operation: curlCode: 28, Timeout was reached
pq.write_table(table_large, "test/test.parquet", filesystem=fs2)
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]