flxo opened a new issue, #574:
URL: https://github.com/apache/avro-rs/issues/574
# Problem
`apache_avro::types::Value` exposes two public methods for schema resolution:
1. `Value::resolve(self, schema: &Schema)`
2. `Value::resolve_schemata(self, schema: &Schema, schemata: Vec<&Schema>)`
Both rebuild a `ResolvedSchema` from the target schema on **every call**.
`resolve`
delegates to `resolve_schemata`, which in turn constructs the
`ResolvedSchema`
before doing any work:
```rust
pub fn resolve(self, schema: &Schema) -> AvroResult<Self> {
self.resolve_schemata(schema, Vec::with_capacity(0))
}
pub fn resolve_schemata(self, schema: &Schema, schemata: Vec<&Schema>) ->
AvroResult<Self> {
let enclosing_namespace = schema.namespace();
let rs = if schemata.is_empty() {
ResolvedSchema::try_from(schema)? // rebuilt every call
} else {
ResolvedSchema::try_from(schemata)? // rebuilt every call
};
self.resolve_internal(schema, rs.get_names(), enclosing_namespace, &None)
}
```
Building the `ResolvedSchema` walks the entire schema tree to collect every
named
type (records, enums, fixed, and other named schemas) and populates a fresh
`NamesRef` (`HashMap<Name, &Schema>`). This cost is paid once per value.
When the reader schema is fixed and many values are resolved against it — for
example decoding a stream of messages that all share one schema — the schema
tree
is re-walked and the name map is re-allocated for each value, even though the
result is identical every time. The per-value resolution itself is
unavoidable
work; the repeated `ResolvedSchema` construction is not.
# Proposed API
Add a method that accepts a caller-provided name map, so the map can be
built once
and reused. Taking the `NamesRef` directly (rather than a concrete
`ResolvedSchema`) keeps the method usable with any source of names:
```rust
impl Value {
/// Perform schema resolution using a pre-built name map.
///
/// Equivalent to [`Value::resolve`] / [`Value::resolve_schemata`] but
skips
/// rebuilding the name map on each call. The `names` must have been
built from
/// `schema` (or a set of schemata that includes it), e.g. via
/// `ResolvedSchema::get_names()`.
pub fn resolve_with_names(
self,
schema: &Schema,
names: &NamesRef,
) -> AvroResult<Self> {
let enclosing_namespace = schema.namespace();
self.resolve_internal(schema, names, enclosing_namespace, &None)
}
}
```
`NamesRef`, `ResolvedSchema`, and `get_names()` are already public, and
`resolve_internal` accepts `&HashMap<Name, S: Borrow<Schema>>`, so a
`&NamesRef`
(`&HashMap<Name, &Schema>`) lines up without further changes. Callers obtain
the
map from `ResolvedSchema::get_names()`:
```rust
let resolved = ResolvedSchema::try_from(&reader_schema)?;
value.resolve_with_names(&reader_schema, resolved.get_names())?;
```
# Caveat: lifetimes when caching
`ResolvedSchema<'s>` borrows from the schema it was built from, so the
pattern
above only works within a single scope: the `ResolvedSchema` cannot be
stored next
to an owned `Schema` in the same struct — that would be a self-referential
borrow.
The crate already has `ResolvedOwnedSchema` (an ouroboros-based owning
variant) for
exactly this case, but it is currently `pub(crate)`. To make a pre-built
name map
usable across the lifetime of a long-lived consumer, `ResolvedOwnedSchema`
should
be exported alongside this new method. It also exposes `get_names() ->
&NamesRef`,
so it feeds `resolve_with_names` directly:
```rust
let resolved = ResolvedOwnedSchema::try_from(reader_schema)?; // owns the
schema
value.resolve_with_names(resolved.get_root_schema(), resolved.get_names())?;
```
# Effect
- Removes the repeated schema-tree walk and `NamesRef` allocation on each
resolution when the schema is reused.
- Does **not** make resolution allocation-free: `resolve` still builds a new
`Value` (records, arrays, maps, strings), so allocation remains
proportional to
the value size. The saving is limited to the name-map construction.
- Behaviour is otherwise identical to `resolve` / `resolve_schemata`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
