erickguan opened a new issue, #7854:
URL: https://github.com/apache/opendal/issues/7854
### Describe the bug
# services/kv: stat() materializes full values to compute content_length
## What happens
Several KV-like services implement `stat()` by fetching the full stored value
and then calling `len()` to populate `Metadata::content_length()`.
Examples include:
- `services/redis`
- `services/postgresql`
- `services/mysql`
- `services/d1`
- `services/mongodb`
- `services/surrealdb`
- `services/memcached`
- `services/foundationdb`
- `services/tikv`
- `services/rocksdb`
- `services/redb`
- `services/persy`
- `services/foyer`
- `services/dashmap`
This is byte-correct in the current full-fetch implementation, but it makes
`stat()` O(object size) and may transfer/materialize large values only to
learn
their length.
## Technical problem
OpenDAL `content_length` represents the full object size in bytes, aligned
with
HTTP `Content-Length`.
For metadata operations, services should avoid reading full object content
when
the backend can return byte length directly. Otherwise workflows that call
`stat()` before `read()` pay an extra full-object read.
A closely related case exists in range reads: many KV-like services fetch the
full value and slice locally. That broader direction is related to RFC-7660 /
#7661, but `stat()` can be improved independently.
## Possible fixes
Add service-specific length helpers and use them from `stat()`:
- Redis: use existing `STRLEN` helper instead of `GET`.
- PostgreSQL: use `OCTET_LENGTH(value)` for byte length.
- MySQL: use `OCTET_LENGTH(value)` for byte length.
- SQLite/D1-like SQL backends: use byte-oriented expressions such as
`LENGTH(CAST(value AS BLOB))`.
- Metadata-capable/cache services: return stored metadata or stored content
length without cloning/fetching full content.
Important: avoid plain SQL `LENGTH(value)` unless byte semantics are
guaranteed.
For text values, it may return character count instead of byte count.
## Validation
For each fixed service:
1. Write a large value, for example 100 MiB.
2. Compare old `stat()` behavior against the new length-only path.
3. Assert `stat().content_length()` equals the byte length of the stored
data.
4. Add a multibyte UTF-8 case where applicable, for example `"你好"` should
report `6` bytes, not `2` characters.
5. Verify `stat()` no longer fetches/materializes the full value.
6. Keep large benchmarks opt-in or off-repo; keep regression tests small.
### Steps to Reproduce
Not applicable.
### Expected Behavior
stat uses length
### Additional Context
## Related context
- #7852 found this issue in `services/sqlite`.
- #7660 / #7661 are related to moving range execution into reusable readers
and
can guide future range-read optimizations, but they do not directly solve
`stat()` materializing full values.
### Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
