GitHub user TennyZhuang created a discussion: rg-opendal, a ripgrep-style search CLI over OpenDAL objects
## Background [ripgrep](https://github.com/BurntSushi/ripgrep) ships its core matching/searching/printing logic as public crates on crates.io: `grep-searcher`, `grep-regex`, `grep-matcher`, `grep-printer`, `ignore`. Those crates expose stable APIs for embedding ripgrep's regex engine and output behavior in other tools without forking ripgrep itself. [rg-opendal](https://github.com/TennyZhuang/rg-opendal) uses those crates, with [OpenDAL](https://github.com/apache/opendal) as the I/O layer, to bring ripgrep-compatible search to S3-compatible object stores. The CLI takes the form `rg-opendal <pattern> s3://bucket/prefix/` and aims to mirror native `rg`'s flag set and output format on object-store inputs. ## Current status Early-stage feasibility study. The crate is a single repository with `[lib] + [[bin]]` structure plus a separate `benchmarks/seam-a/` microbenchmark crate. Verification is shell-harness-based: each feature family has a script that diffs the binary's output against native `rg` on a fixture hosted in a local MinIO instance. Two reader strategies are wired: - A full-buffer reader (default), which is the fastest path on local MinIO in our measurements. - A streaming reader (`--streaming`) that bridges OpenDAL's `BufferStream` to sync `std::io::Read` via `Handle::block_on`. Local benchmark on 2.7–525 MB files measured the per-chunk bridge cost at ≤1% of total search time; the dominant cost is S3 I/O. ## Implemented scope The following ripgrep flag families have feature-parity-attested implementations against native `rg`'s output on a MinIO fixture (the per-family status table is in [MATRIX.md](https://github.com/TennyZhuang/rg-opendal/blob/main/MATRIX.md)): - Literal and regex search; case-insensitive (`-i`). - Glob and type filtering (`-g`, `-t`, `-T`); `-t` coverage is currently unit-test-level rather than full output-parity. - JSON output, context lines, stats (`--json`, `-A`/`-B`/`-C`, `--stats`). - Color output (`--color=auto|always|never`). - Count and files-with-matches (`-c`, `-l`). - Word and line regexp (`-w`, `-x`). - Max-count (`-m`). - Invert match (`-v`). - Printer flags: line-number, column, heading (`-N`, `--column`, `--heading`). - Path-prefix control with auto-detection (`-I`, `-H`). - NUL path terminator (`-0`) and NUL-separated records (`-z`). - Force-text mode for binary inputs (`-a`) — flag-acceptance smoke only at this point. - Fixed-strings (`-F`) — spot-checked, dedicated harness pending. - Streaming reader (`--streaming`), with output-parity-attested against the default reader. One small convention from working with OpenDAL: each harness probes `op.info().full_capability().<flag>` and skips a test rather than failing it when the underlying service doesn't support a capability. That keeps the service-vs-feature axes clean during development. ## Known limitations - All current measurements are against local MinIO, not real S3. Real-S3 latency hasn't been characterized. - CI is not yet configured; verification is run locally. - Walker is serial-only (no parallel/streaming walker yet). - The `-a/--text` family is flag-accepted but not yet output-parity attested against native `rg`. - The `-F/--fixed-strings` family has been spot-checked but does not yet have a dedicated harness. - The repository is not yet published to crates.io. ## Questions for the community - Is the integration shape (independent crate consuming `grep-*` from crates.io, with OpenDAL as the I/O layer) reasonable from the OpenDAL maintainers' perspective? - Has anyone in the OpenDAL ecosystem combined `grep-*` (or another sync `Read`-consuming searcher) with `BufferStream` before? If so, are there pitfalls or conventions on the streaming-bridge pattern we should know about? - Are there OpenDAL-side conventions for ecosystem tools (badge text, matrix-of-services compatibility, capability-gating language) that we should adopt to keep the project consistent with the rest of the ecosystem? Repo: [https://github.com/TennyZhuang/rg-opendal](https://github.com/TennyZhuang/rg-opendal) Thanks for OpenDAL — happy to take any feedback or pointers. GitHub link: https://github.com/apache/opendal/discussions/7780 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
