GitHub user Xuanwo edited a discussion: Some ideas I have for OpenDAL
This isn't a proposal, request, or plan—just some ideas I've been considering for a while. I'm still unsure if they are valid, feasible, or worth implementing. However, I think it's beneficial to start a discussion and share these thoughts with the entire community instead of keeping them to myself. Feel free to leave your comment if you like this idea or suggesting more. ## List With Glob/Pattern Initially reported at https://github.com/apache/opendal/issues/1251, and subsequently at https://github.com/apache/opendal/issues/3500. This makes sense as it's very common for users to list items using a pattern. For example, list all `pdf` files or all files that match `2024/**/result.json`. Additionally, some services such as Google Cloud Storage offer native features like [`matchGlob`](https://cloud.google.com/storage/docs/json_api/v1/objects/list) for enhanced functionality. This can reduce a lot of API requests if the prefix is large. I'm struggling with how to design the API. - Just support glob? - Like `op.list(path).glob("2024/**/result.json").await` - How to implement fallback if services doesn't support glob? - Services might have different behavior for the same glob pattern, how to support them? - Do we need to support `regex`? - Like `op.list(path).regex(r"(?m)^([^:]+):([0-9]+):(.+)$").await` ## Expose HttpClient API Initially reported at https://github.com/apache/opendal/issues/4471. This is an interesting idea that allows users to choose the HTTP client they prefer. Users can do more things like metrics, logging inside the client by set the client for service. I'm struggling with the API naming: `HttpClient` doesn't look like a good name for trait. We need better one. I want we can have a trait `X`, and a type called `HttpClient = Arc<dyn X>`. Additionally, we should consider the internal handling of requests and responses. It's better to keep these processes hidden and not require users to implement them. ## `opendal_tokio_fs` and `opendal_std_fs` Initially reported at https://github.com/apache/opendal/issues/4194 We can have two new crates that allowing users to use similiar API like `tokio::fs`, `std::fs` but backed by OpenDAL. I'm struggling with how to design the API: `tokio::fs` and `std::fs` are provide globally, how users init opendal operator first? My current idea is to have a `set_fs(op: Operator)` function or macro. GitHub link: https://github.com/apache/opendal/discussions/4882 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
