siomporas opened a new issue, #6831: URL: https://github.com/apache/opendal/issues/6831
### Feature Description Currently there is no support for arbitrary git service over http with LFS support. ### Problem and Solution ### Problem: We have a project that relies heavily on OpenDAL for remote data access, dealing primarily with AI data like weights and datasets. The HuggingFace service is awesome for this (as is s3 and others)! But we want to be able to support any git repository that may house the same sort of data, including internal repositories running in our self-hosted Gitlab instance. We currently launch git in a subprocess to fetch and download ref history, then checkout the right commit, then clone LFS files - this feels clunky in a Rust application, and requires us to complete downloading the model before we can stream the contents to clients. ### Solution: Using [gix](https://github.com/GitoxideLabs/gitoxide/) along with the OpenDAL http service, I was able to make a functioning prototype that can fetch the remote state of any remote repo at any ref or oid, pull the repository files, then go through the LFS pointers and start streaming them down with OpenDAL http service. This fits our particular use case, but before I say good enough and call it a day, I wanted to know if this functionality might be of interest to the project maintainers here - and if so what is a good way to get this formally added as a feature request, and eventually contribute a crate feature? I wouldn't want to go through the effort of porting what I did thus far to OpenDAL's service APIs unless I had a path forward. Thanks! ### Additional Context _No response_ ### Are you willing to contribute to the development of this feature? - [ ] Yes, I am willing to contribute to the development of this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
