jorisvandenbossche commented on issue #34414: URL: https://github.com/apache/arrow/issues/34414#issuecomment-1464097361
@xu2011 what you see is essentially behaviour controlled by pandas (passing an fsspec filesystem to pyarrow). It suffers from the same issue that apparently using an fsspec wrapped filesystem in pyarrow can be a lot slower, but avoiding doing that is something that needs to be done in pandas (pyarrow only sees the filesystem being passed to it). Now, there is actually an open pandas PR to avoid using fsspec when using pyarrow in the pandas read_parquet (https://github.com/pandas-dev/pandas/pull/51609) > Oh I just wanted to add one more note, when using the dataset API with the above `fsspec` I noticed it's much slower to use the dataset API versus `pq.read_table` @JonnyWaffles hmm, that's strange, as that should essentially do the same these days. Is the `s3 _key` pointing to a single file or to a directory? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
