Juan Galvez created ARROW-12760:
-----------------------------------
Summary: [Python] S3FileSystem: IO thread parallelism limited to 8
threads
Key: ARROW-12760
URL: https://issues.apache.org/jira/browse/ARROW-12760
Project: Apache Arrow
Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: Juan Galvez
On Arrow 4, I'm not able to get more than 8 thread parallelism (for example
when N threads call fs.ls() with different paths) for S3FileSystem created with
pyarrow, because the default IO context is created with 8 thread limit
(hardcoded), and I don't see a way to change this or pass a custom IO context
through pyarrow APIs. I believe this affects other Arrow filesystem
implementations as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
