legout opened a new issue, #34363:
URL: https://github.com/apache/arrow/issues/34363

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   When I try to write a pyarrow.table to cloudflare r2 object store, I got an 
error when files are larger than x MB (I do not know the exact threshold) and 
pyarrow internally switches  to multipart uploading.
   
   I´ve used `s3fs.S3FileSystem` (fsspec) and also tried 
`pyarrow.fs.S3FileSystem`. Here is some example code:
   
   
   ```python
   import pyarrow.fs as pafs
   import s3fs
   import pyarrow.parquet as pq
   import pyarrow as pa
   
   table_small = pq.read_table("small_data.parquet")
   table_large = pq.read_table("large_data.parquet")
   
   fs1 = s3fs.S3FileSystem(
     key="some_key", 
     secret="some_secret", 
     
client_kwargs=dict(endpoint_url="https://123456.r2.cloudflarestorage.com";), 
     s3_additional_kwargs=dict(ACL="private") # <- this is neccesary for 
writting. 
   ) 
   
   fs2 = pafs.S3FileSystem(
     access_key="some_key", 
     secret_key="some_secret", 
     endpoint_override="https://123456.r2.cloudflarestorage.com";
   )
   
   
   pq.write_table(table_small, "test/test.parquet", filesystem=fs1) # <- works 
   pq.write_table(table_small, "test/test.parquet", filesystem=fs2) # <- works 
   
   #  failed with OSError: [Errno 22] There was a problem with the multipart 
upload. 
   pq.write_table(table_large, "test/test.parquet", filesystem=fs1) 
   
   # failed with OSError: When initiating multiple part upload for key 
'test.parquet' in bucket 'test': AWS Error NETWORK_CONNECTION during 
CreateMultipartUpload operation: curlCode: 28, Timeout was reached
   pq.write_table(table_large, "test/test.parquet", filesystem=fs2) 
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to