chaokunyang opened a new issue, #3290:
URL: https://github.com/apache/fory/issues/3290
### Feature Request
Implement **schema evolution mode** (TypeDef meta share + compatible struct)
in the **Dart** runtime to match the xlang Type Meta specification. Current
Dart implementation does not support the schema evolution pathway described in
the spec, which breaks compatibility with other languages when
`compatible`/meta-share is enabled.
### Is your feature request related to a problem? Please describe
When other languages serialize with schema evolution enabled (TypeDef/meta
share), Dart fails to decode because it does not implement the TypeDef/TypeMeta
read/write flow. This blocks cross-language compatibility for evolving schemas.
### Describe the solution you'd like
Implement the **Type Meta + TypeDef** encoding/decoding flow in Dart
according to the spec, including:
#### 1) Type ID & Named Types handling
- Read/write type IDs as **varuint32 (small7)**.
- Support `ENUM`, `STRUCT`, `COMPATIBLE_STRUCT`, `EXT`, `TYPED_UNION`
(user-registered) plus `NAMED_*` types when unregistered.
- Ensure named types embed namespace + type name (or shared TypeDef when
meta share is enabled).
#### 2) Meta Share (streaming TypeDef)
- Implement the shared TypeDef cache with the marker encoding:
- `marker = (index << 1) | flag`
- `flag = 0` => new TypeDef follows
- `flag = 1` => reference cached TypeDef
- Write algorithm: lookup class in meta context map, emit reference or new
TypeDef bytes.
- Read algorithm: parse marker, load cached TypeDef or read + cache new
TypeDef.
#### 3) TypeDef decoding/encoding
TypeDef format:
```
| 8-byte global header | [optional size varuint] | TypeDef body |
```
- Global header (uint64 LE):
- low 8 bits: meta size (0xFF + extra size varuint if needed)
- bit 8: HAS_FIELDS_META
- bit 9: COMPRESS_META (decompress before parsing)
- bits 10-13 reserved
- high 50 bits: hash of TypeDef body
- TypeDef body:
- meta header byte: `num_fields`, `REGISTER_BY_NAME`
- type spec: namespace+type_name or type_id
- field list with field header, field type info, and field name bytes
#### 4) Field metadata parsing
- Field header bits:
- bits 6-7: name encoding (UTF8 / ALL_TO_LOWER_SPECIAL /
LOWER_UPPER_DIGIT_SPECIAL / TAG_ID)
- bits 2-5: size (name length-1 or tag id); if 0b1111, read extra varuint
size
- bit 1: nullable flag
- bit 0: reference tracking flag
- Field type info:
- top-level type as varuint32
- nested LIST/SET element type encoded with `(nested_type_id << 2) |
(nullable << 1) | tracking_ref`
- MAP key/value types encoded similarly
- one-dimensional primitive arrays use `*_ARRAY` type IDs; others use LIST
- Field name encoding:
- TAG_ID: no name bytes
- otherwise write meta string name bytes
- xlang uses `snake_case` conversion before encoding
#### 5) Compatibility behavior
- Decoders must match fields by **name/tag ID**, not position.
- Keep deterministic TypeDef generation (stable field order) for
cross-language match.
#### 6) Tests
- Round-trip tests for compatible struct with meta share enabled.
- Decode TypeDef bytes generated by other languages (cross-language golden
tests).
- Named vs registered type cases.
- Meta share cache correctness (reference reuse).
### Describe alternatives you've considered
- Only supporting strict type-id registration without TypeDef meta.
- Rejected: breaks schema evolution and cross-language compatibility.
### Additional context
Spec reference: **Type Meta / TypeDef** section (xlang serialization). Dart
runtime should match other languages’ behavior so compatible mode works
end-to-end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
