kszucs commented on issue #44900:
URL: https://github.com/apache/arrow/issues/44900#issuecomment-2984046199
I would still like to have support for popular filesystems schemes for
better usability.
I understand the possible backward compatibility problems, though:
- it is rather unlikely to have a native `hf://` implementation directly in
arrow
- it would be most certainly wrap the same underlying huggingface library as
the fsspec implementation does
- we would have control over the compatibility
Also `pandas` do support fsspec URIs, at least the following does work out
of the box:
```py
pd.read_parquet("hf://datasets/HuggingFaceTB/smoltalk/data/everyday-conversations/test-00000-of-00001.parquet")
```
See the relevant [pandas code
here](https://github.com/pandas-dev/pandas/blob/8a1d5a06f9fb3c232249e3ed301932053efb06d8/pandas/io/parquet.py#L83-L143).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
