JonnyWaffles opened a new issue, #34414: URL: https://github.com/apache/arrow/issues/34414
### Describe the bug, including details regarding any error messages, version, and platform. Hi team, I hope you are well. Apologies if this is documented somewhere, I searched around but could not find anything related. I am using Pyarrow 10.0.1 inside a Ubuntu 20.04 container. For reasons I need to use the third party [fsspec.S3FileSystem](https://github.com/fsspec/s3fs) in lieu of the native `pyarrow.fs.S3FileSystem`. When trying to read a single 220MB Parquet file from S3 using the fsspec s3fs, the PyArrow Dataset API takes north of 3 minutes! Using s3fs to read the bytes without leveraging Pyarrow, or using the Pyarrow native S3FileSystem, works within 16 seconds as expected. My understanding is that leveraging `fsspec.S3FileSystem` should not decrease performance in Pyarrow. Test example ```python pa_s3fs = fs.S3FileSystem() time_start = time.perf_counter() ds.dataset(path, format="parquet", filesystem=pa_s3fs).to_table() print("pa S3Filesystem!: ", time.perf_counter() - time_start) time_start = time.perf_counter() s3fs.read_bytes(key) print("fsspec raw_bytes:!", time.perf_counter() - time_start) time_start = time.perf_counter() ds.dataset(path, format="parquet", filesystem=s3fs).to_table() print("PA w/ fsspec S3Filesystem!: ", time.perf_counter() - time_start) ``` Output ``` airflow-worker | Wrote key='mybucket/mypath/part-0.parquet' size=211661451 airflow-worker | Landed total 201 MB to s3! airflow-worker | pa S3Filesystem!: 14.232619059999706 airflow-worker | fsspec raw_bytes:! 15.852554664001218 airflow-worker | PA w/ fsspec S3Filesystem!: 194.75488279599995 ``` I found this phrase in the Pyarrow docs > pre_buffer[bool](https://docs.python.org/3/library/stdtypes.html#bltin-boolean-values), default [True](https://docs.python.org/3/library/constants.html#True) Coalesce and issue file reads in parallel to improve performance on high-latency filesystems (e.g. S3). If True, Arrow will use a background I/O thread pool. This option is only supported for use_legacy_dataset=False. If using a filesystem layer that itself performs readahead (e.g. fsspecās S3FS), disable readahead for best results. So I tried setting `pre_buffer=False` but the performance issue remains. Any idea what could be causing the performance issue when leveraging fsspec inside of Pyarrow? ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
