LucaMaurelliC commented on issue #39005: URL: https://github.com/apache/arrow/issues/39005#issuecomment-1835577595
What do you mean by collapsed IO? Also, does ffspec and its implemented fs [Azure Blob Storage](https://github.com/fsspec/adlfs) exploits the PreBuffering already? From the arrow doc: "Coalesce and issue file reads in parallel to improve performance on high-latency filesystems (e.g. S3). If True, Arrow will use a background I/O thread pool. This option is only supported for use_legacy_dataset=False. If using a filesystem layer that itself performs readahead (e.g. fsspec’s S3FS), disable readahead for best results." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
