Vadym Dytyniak created ARROW-18228:
--------------------------------------
Summary: AWS Error SLOW_DOWN during PutObject operation
Key: ARROW-18228
URL: https://issues.apache.org/jira/browse/ARROW-18228
Project: Apache Arrow
Issue Type: Bug
Affects Versions: 10.0.0
Reporter: Vadym Dytyniak
We use Dask to parallelise read/write operations and pyarrow to write dataset
from worker nodes.
After pyarrow released version 10.0.0, our data flows automatically switched to
the latest version and some of them started to fail with the following error:
{code:java}
File "/usr/local/lib/python3.10/dist-packages/org/store/storage.py", line 768,
in _write_partition
ds.write_dataset(
File "/usr/local/lib/python3.10/dist-packages/pyarrow/dataset.py", line 988,
in write_dataset
_filesystemdataset_write(
File "pyarrow/_dataset.pyx", line 2859, in
pyarrow._dataset._filesystemdataset_write
check_status(CFileSystemDataset.Write(c_options, c_scanner))
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
raise IOError(message)
OSError: When creating key 'equities.us.level2.by_security/' in bucket
'org-prod': AWS Error SLOW_DOWN during PutObject operation: Please reduce your
request rate. {code}
Do you have any idea what was changed for dataset write between 9.0.0 and
10.0.0 to help us to fix the issue?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
