JonnyWaffles opened a new issue, #34414:
URL: https://github.com/apache/arrow/issues/34414

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Hi team, I hope you are well. Apologies if this is documented somewhere, I 
searched around but could not find anything related.
   
   I am using Pyarrow 10.0.1 inside a Ubuntu 20.04 container. For reasons I 
need to use the third party 
[fsspec.S3FileSystem](https://github.com/fsspec/s3fs) in lieu of the native 
`pyarrow.fs.S3FileSystem`. When trying to read a single 220MB Parquet file from 
S3 using the fsspec s3fs, the PyArrow Dataset API takes north of 3 minutes!
   
   Using s3fs to read the bytes without leveraging Pyarrow, or using the 
Pyarrow native S3FileSystem, works within 16 seconds as expected. My 
understanding is that leveraging `fsspec.S3FileSystem` should not decrease 
performance in Pyarrow.
   
   Test example
   
   ```python
       pa_s3fs = fs.S3FileSystem()
       time_start = time.perf_counter()
       ds.dataset(path, format="parquet", filesystem=pa_s3fs).to_table()
       print("pa S3Filesystem!: ", time.perf_counter() - time_start)
   
       time_start = time.perf_counter()
       s3fs.read_bytes(key)
       print("fsspec raw_bytes:!", time.perf_counter() - time_start)
   
       time_start = time.perf_counter()
       ds.dataset(path, format="parquet", filesystem=s3fs).to_table()
       print("PA w/ fsspec S3Filesystem!: ", time.perf_counter() - time_start)
   ```
   Output
   ```
   airflow-worker  | Wrote key='mybucket/mypath/part-0.parquet' size=211661451
   airflow-worker  | Landed total 201 MB to s3!
   airflow-worker  | pa S3Filesystem!:  14.232619059999706
   airflow-worker  | fsspec raw_bytes:! 15.852554664001218
   airflow-worker  | PA w/ fsspec S3Filesystem!:  194.75488279599995
   ```
   
   I found this phrase in the Pyarrow docs
   
   > 
pre_buffer[bool](https://docs.python.org/3/library/stdtypes.html#bltin-boolean-values),
 default [True](https://docs.python.org/3/library/constants.html#True)
   Coalesce and issue file reads in parallel to improve performance on 
high-latency filesystems (e.g. S3). If True, Arrow will use a background I/O 
thread pool. This option is only supported for use_legacy_dataset=False. If 
using a filesystem layer that itself performs readahead (e.g. fsspec’s S3FS), 
disable readahead for best results.
   
   So I tried setting `pre_buffer=False` but the performance issue remains.
   
   Any idea what could be causing the performance issue when leveraging fsspec 
inside of Pyarrow?
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to