lhoestq commented on issue #44900: URL: https://github.com/apache/arrow/issues/44900#issuecomment-3000993813
It is, because the HF filesystem in fsspec is already quite advanced and performant. It's a thin wrapper over `huggingface_hub.HfApi` which optimizes I/O using Xet (a git variant that enables deduplicated uploads and downloads) which provides nice performance and is quite useful to users. As far as I know this isn't easily transferable to arrow c++ since our Xet implementation is tailored for `huggingface_hub`. Though I understand that in the general case having a c++ implementation potentially removes unnecessary overhead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
