krfricke commented on issue #32372:
URL: https://github.com/apache/arrow/issues/32372#issuecomment-1421097792
We experience the same issuein Ray, and it's easily reproducible. The issue
comes up when requesting a recursive upload of more or equal files than CPUs
are available.
For instance, on my MacBook with 8 cores, I can upload a folder with 7
files, but not with 8 files:
```
mkdir -p /tmp/pa-s3
cd /tmp/pa-s3
for i in {1..7}; do touch $i.txt; done
# This works
python -c "import pyarrow.fs; pyarrow.fs.copy_files('/tmp/pa-s3',
's3://bucket/folder')"
for i in {1..8}; do touch $i.txt; done
# This hangs forever
python -c "import pyarrow.fs; pyarrow.fs.copy_files('/tmp/pa-s3',
's3://bucket/folder')"
```
The problem comes up with pyarrow 6-11 and can be avoided with
`use_threads=False`, but this obviously harms performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
