This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory-site.git
The following commit(s) were added to refs/heads/main by this push:
new 4c6d53e1f2 🔄 synced local 'docs/specification/' with remote
'docs/specification/'
4c6d53e1f2 is described below
commit 4c6d53e1f241b76883f2c5d5311a07612e0c2768
Author: chaokunyang <[email protected]>
AuthorDate: Wed Feb 4 08:07:25 2026 +0000
🔄 synced local 'docs/specification/' with remote 'docs/specification/'
---
docs/specification/xlang_implementation_guide.md | 257 +++++++++++++++++++++++
docs/specification/xlang_serialization_spec.md | 241 +--------------------
2 files changed, 261 insertions(+), 237 deletions(-)
diff --git a/docs/specification/xlang_implementation_guide.md
b/docs/specification/xlang_implementation_guide.md
new file mode 100644
index 0000000000..77b808cdd4
--- /dev/null
+++ b/docs/specification/xlang_implementation_guide.md
@@ -0,0 +1,257 @@
+---
+title: Xlang Implementation Guide
+sidebar_position: 10
+id: xlang_implementation_guide
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+## Implementation guidelines
+
+### How to reduce memory read/write code
+
+- Try to merge multiple bytes into an int/long write before writing to reduce
memory IO and bound check cost.
+- Read multiple bytes as an int/long, then split into multiple bytes to reduce
memory IO and bound check cost.
+- Try to use one varint/long to write flags and length together to save one
byte cost and reduce memory io.
+- Condition branches are less expensive compared to memory IO cost unless
there are too many branches.
+
+### Fast deserialization for static languages without runtime codegen support
+
+For type evolution, the serializer will encode the type meta into the
serialized data. The deserializer will compare
+this meta with class meta in the current process, and use the diff to
determine how to deserialize the data.
+
+For java/javascript/python, we can use the diff to generate serializer code at
runtime and load it as class/function for
+deserialization. In this way, the type evolution will be as fast as type
consist mode.
+
+For C++/Rust, we can't generate the serializer code at runtime. So we need to
generate the code at compile-time using
+meta programming. But at that time, we don't know the type schema in other
processes, so we can't generate the
+serializer code for such inconsistent types. We may need to generate the code
which has a loop and compare field name
+one by one to decide whether to deserialize and assign the field or skip the
field value.
+
+One fast way is that we can optimize the string comparison into `jump`
instructions:
+
+- Assume the current type has `n` fields, and the peer type has `n1` fields.
+- Generate an auto growing `field id` from `0` for every sorted field in the
current type at the compile time.
+- Compare the received type meta with current type, generate same id if the
field name is same, otherwise generate an
+ auto growing id starting from `n`, cache this meta at runtime.
+- Iterate the fields of received type meta, use a `switch` to compare the
`field id` to deserialize data
+ and `assign/skip` field value. **Continuous** field id will be optimized
into `jump` in `switch` block, so it will
+ very fast.
+
+Here is an example, suppose process A has a class `Foo` with version 1 defined
as `Foo1`, process B has a class `Foo`
+with version 2 defined as `Foo2`:
+
+```c++
+// class Foo with version 1
+class Foo1 {
+ int32_t v1; // id 0
+ std::string v2; // id 1
+};
+// class Foo with version 2
+class Foo2 {
+ // id 0, but will have id 2 in process A
+ bool v0;
+ // id 1, but will have id 0 in process A
+ int32_t v1;
+ // id 2, but will have id 3 in process A
+ int64_t long_value;
+ // id 3, but will have id 1 in process A
+ std::string v2;
+ // id 4, but will have id 4 in process A
+ std::vector<std::string> list;
+};
+```
+
+When process A received serialized `Foo2` from process B, here is how it
deserialize the data:
+
+```c++
+Foo1 foo1 = ...;
+const std::vector<fory::FieldInfo> &field_infos = type_meta.field_infos;
+for (const auto &field_info : field_infos) {
+ switch (field_info.field_id) {
+ case 0:
+ foo1.v1 = buffer.read_varint32();
+ break;
+ case 1:
+ foo1.v2 = fory.read_string();
+ break;
+ default:
+ fory.skip_data(field_info);
+ }
+}
+```
+
+## Implementation Checklist for New Languages
+
+This section provides a step-by-step guide for implementing Fory xlang
serialization in a new language.
+
+### Phase 1: Core Infrastructure
+
+1. **Buffer Implementation**
+ - [ ] Create a byte buffer with read/write cursor tracking
+ - [ ] Implement little-endian byte order for all multi-byte writes
+ - [ ] Implement `write_int8`, `write_int16`, `write_int32`, `write_int64`
+ - [ ] Implement `write_float32`, `write_float64`
+ - [ ] Implement `read_*` counterparts for all write methods
+ - [ ] Implement buffer growth strategy (e.g., doubling)
+
+2. **Varint Encoding**
+ - [ ] Implement `write_varuint32` / `read_varuint32`
+ - [ ] Implement `write_varint32` / `read_varint32` (with ZigZag)
+ - [ ] Implement `write_varuint64` / `read_varuint64`
+ - [ ] Implement `write_varint64` / `read_varint64` (with ZigZag)
+ - [ ] Implement `write_varuint36_small` / `read_varuint36_small` (for
strings)
+ - [ ] Optionally implement Hybrid encoding (TAGGED_INT64/TAGGED_UINT64) for
int64
+
+3. **Header Handling**
+ - [ ] Write/read bitmap flags (null, xlang, oob)
+
+### Phase 2: Basic Type Serializers
+
+4. **Primitive Types**
+ - [ ] bool (1 byte: 0 or 1)
+ - [ ] int8, int16, int32, int64 (little endian)
+ - [ ] float32, float64 (IEEE 754, little endian)
+
+5. **String Serialization**
+ - [ ] Implement string header: `(byte_length << 2) | encoding`
+ - [ ] Support UTF-8 encoding (required for xlang)
+ - [ ] Optionally support LATIN1 and UTF-16
+
+6. **Temporal Types**
+ - [ ] Duration (seconds + nanoseconds)
+ - [ ] Timestamp (seconds + nanoseconds since epoch)
+ - [ ] Date (days since epoch)
+
+7. **Reference Tracking**
+ - [ ] Implement write-side object tracking (object → ref_id map)
+ - [ ] Implement read-side object tracking (ref_id → object list)
+ - [ ] Handle all four reference flags: NULL(-3), REF(-2), NOT_NULL(-1),
REF_VALUE(0)
+ - [ ] Support disabling reference tracking per-type or globally
+
+### Phase 3: Collection Types
+
+8. **List/Array Serialization**
+ - [ ] Write length as varuint32
+ - [ ] Write elements header byte
+ - [ ] Handle homogeneous vs heterogeneous elements
+ - [ ] Handle null elements
+
+9. **Map Serialization**
+ - [ ] Write total size as varuint32
+ - [ ] Implement chunk-based format (max 255 pairs per chunk)
+ - [ ] Write KV header byte per chunk
+ - [ ] Handle key and value type variations
+
+10. **Set Serialization**
+ - [ ] Same format as List (reuse implementation)
+
+### Phase 4: Meta String Encoding
+
+Meta strings are required for enum and struct serialization (encoding field
names, type names, namespaces).
+
+11. **Meta String Compression**
+ - [ ] Implement LOWER_SPECIAL encoding (5 bits/char)
+ - [ ] Implement LOWER_UPPER_DIGIT_SPECIAL encoding (6 bits/char)
+ - [ ] Implement FIRST_TO_LOWER_SPECIAL encoding
+ - [ ] Implement ALL_TO_LOWER_SPECIAL encoding
+ - [ ] Implement encoding selection algorithm
+ - [ ] Implement meta string deduplication
+
+### Phase 5: Enum Serialization
+
+12. **Enum Serialization**
+ - [ ] Write ordinal as varuint32
+ - [ ] Support named enum (namespace + type name)
+
+### Phase 6: Struct Serialization
+
+13. **Type Registration**
+ - [ ] Support registration by numeric ID
+ - [ ] Support registration by namespace + type name
+ - [ ] Maintain type → serializer mapping
+ - [ ] Generate type IDs: write internal type ID, then `user_type_id` as
varuint32
+
+14. **Field Ordering**
+ - [ ] Implement the spec-defined grouping and ordering
(primitive/boxed/built-in, collections/maps, other)
+ - [ ] Use a stable comparator within each group (type ID and name)
+ - [ ] Use tag ID or snake_case field name as field identifier for
fingerprints
+
+15. **Schema Consistent Mode**
+ - [ ] If class-version check is enabled, compute schema hash from field
identifiers
+ - [ ] Write 4-byte schema hash before fields
+ - [ ] Serialize fields in Fory order
+
+16. **Compatible/Meta Share Mode**
+ - [ ] Implement shared TypeDef stream (inline new TypeDefs, index
references)
+ - [ ] Map fields by name or tag ID, skip unknown fields
+ - [ ] Apply nullable/ref flags from TypeDef metadata
+
+### Phase 7: Other types
+
+17. **Binary/Array Types**
+
+- [ ] Primitive arrays (direct buffer copy)
+- [ ] Multi-dimensional arrays as nested lists (no tensor encoding)
+
+### Testing Strategy
+
+18. **Cross-Language Compatibility Tests**
+ - [ ] Serialize in new language, deserialize in Java/Python
+ - [ ] Serialize in Java/Python, deserialize in new language
+ - [ ] Test all primitive types
+ - [ ] Test strings with various encodings
+ - [ ] Test collections (empty, single, multiple elements)
+ - [ ] Test maps with various key/value types
+ - [ ] Test nested structs
+ - [ ] Test circular references (if supported)
+
+## Language-Specific Implementation Notes
+
+### Java
+
+- Uses runtime code generation (JIT) for maximum performance
+- Supports all reference tracking modes
+- Uses internal String coder for encoding selection
+- Thread-safe via `ThreadSafeFory` wrapper
+
+### Python
+
+- Two modes: Pure Python (debugging) and Cython (performance)
+- Uses `id(obj)` for reference tracking
+- Latin1/UTF-16/UTF-8 encoding for all strings in xlang mode
+- `dataclass` support via code generation
+
+### C++
+
+- Compile-time reflection via macros (`FORY_STRUCT`)
+- Template meta programming for type dispatch and serializer selection
+- Uses `std::shared_ptr` for reference tracking
+- Compile-time field ordering
+- No runtime code generation
+
+### Rust
+
+- Derive macros for automatic serialization (`#[derive(ForyObject)]`)
+- Uses `Rc<T>` / `Arc<T>` for reference tracking
+- Thread-local context caching for performance
+- Compile-time field ordering
+
+### Go
+
+- Reflection-based and codegen-based modes
+- Struct tags for field annotations
+- Interface types for polymorphism
diff --git a/docs/specification/xlang_serialization_spec.md
b/docs/specification/xlang_serialization_spec.md
index b611d8fd8a..084bc6cd94 100644
--- a/docs/specification/xlang_serialization_spec.md
+++ b/docs/specification/xlang_serialization_spec.md
@@ -1445,243 +1445,6 @@ standard `skipValue(type_id)`.
Type will be serialized using type meta format.
-## Implementation guidelines
-
-### How to reduce memory read/write code
-
-- Try to merge multiple bytes into an int/long write before writing to reduce
memory IO and bound check cost.
-- Read multiple bytes as an int/long, then split into multiple bytes to reduce
memory IO and bound check cost.
-- Try to use one varint/long to write flags and length together to save one
byte cost and reduce memory io.
-- Condition branches are less expensive compared to memory IO cost unless
there are too many branches.
-
-### Fast deserialization for static languages without runtime codegen support
-
-For type evolution, the serializer will encode the type meta into the
serialized data. The deserializer will compare
-this meta with class meta in the current process, and use the diff to
determine how to deserialize the data.
-
-For java/javascript/python, we can use the diff to generate serializer code at
runtime and load it as class/function for
-deserialization. In this way, the type evolution will be as fast as type
consist mode.
-
-For C++/Rust, we can't generate the serializer code at runtime. So we need to
generate the code at compile-time using
-meta programming. But at that time, we don't know the type schema in other
processes, so we can't generate the
-serializer code for such inconsistent types. We may need to generate the code
which has a loop and compare field name
-one by one to decide whether to deserialize and assign the field or skip the
field value.
-
-One fast way is that we can optimize the string comparison into `jump`
instructions:
-
-- Assume the current type has `n` fields, and the peer type has `n1` fields.
-- Generate an auto growing `field id` from `0` for every sorted field in the
current type at the compile time.
-- Compare the received type meta with current type, generate same id if the
field name is same, otherwise generate an
- auto growing id starting from `n`, cache this meta at runtime.
-- Iterate the fields of received type meta, use a `switch` to compare the
`field id` to deserialize data
- and `assign/skip` field value. **Continuous** field id will be optimized
into `jump` in `switch` block, so it will
- very fast.
-
-Here is an example, suppose process A has a class `Foo` with version 1 defined
as `Foo1`, process B has a class `Foo`
-with version 2 defined as `Foo2`:
-
-```c++
-// class Foo with version 1
-class Foo1 {
- int32_t v1; // id 0
- std::string v2; // id 1
-};
-// class Foo with version 2
-class Foo2 {
- // id 0, but will have id 2 in process A
- bool v0;
- // id 1, but will have id 0 in process A
- int32_t v1;
- // id 2, but will have id 3 in process A
- int64_t long_value;
- // id 3, but will have id 1 in process A
- std::string v2;
- // id 4, but will have id 4 in process A
- std::vector<std::string> list;
-};
-```
-
-When process A received serialized `Foo2` from process B, here is how it
deserialize the data:
-
-```c++
-Foo1 foo1 = ...;
-const std::vector<fory::FieldInfo> &field_infos = type_meta.field_infos;
-for (const auto &field_info : field_infos) {
- switch (field_info.field_id) {
- case 0:
- foo1.v1 = buffer.read_varint32();
- break;
- case 1:
- foo1.v2 = fory.read_string();
- break;
- default:
- fory.skip_data(field_info);
- }
-}
-```
-
-## Implementation Checklist for New Languages
-
-This section provides a step-by-step guide for implementing Fory xlang
serialization in a new language.
-
-### Phase 1: Core Infrastructure
-
-1. **Buffer Implementation**
- - [ ] Create a byte buffer with read/write cursor tracking
- - [ ] Implement little-endian byte order for all multi-byte writes
- - [ ] Implement `write_int8`, `write_int16`, `write_int32`, `write_int64`
- - [ ] Implement `write_float32`, `write_float64`
- - [ ] Implement `read_*` counterparts for all write methods
- - [ ] Implement buffer growth strategy (e.g., doubling)
-
-2. **Varint Encoding**
- - [ ] Implement `write_varuint32` / `read_varuint32`
- - [ ] Implement `write_varint32` / `read_varint32` (with ZigZag)
- - [ ] Implement `write_varuint64` / `read_varuint64`
- - [ ] Implement `write_varint64` / `read_varint64` (with ZigZag)
- - [ ] Implement `write_varuint36_small` / `read_varuint36_small` (for
strings)
- - [ ] Optionally implement Hybrid encoding (TAGGED_INT64/TAGGED_UINT64) for
int64
-
-3. **Header Handling**
- - [ ] Write/read bitmap flags (null, xlang, oob)
-
-### Phase 2: Basic Type Serializers
-
-4. **Primitive Types**
- - [ ] bool (1 byte: 0 or 1)
- - [ ] int8, int16, int32, int64 (little endian)
- - [ ] float32, float64 (IEEE 754, little endian)
-
-5. **String Serialization**
- - [ ] Implement string header: `(byte_length << 2) | encoding`
- - [ ] Support UTF-8 encoding (required for xlang)
- - [ ] Optionally support LATIN1 and UTF-16
-
-6. **Temporal Types**
- - [ ] Duration (seconds + nanoseconds)
- - [ ] Timestamp (seconds + nanoseconds since epoch)
- - [ ] Date (days since epoch)
-
-7. **Reference Tracking**
- - [ ] Implement write-side object tracking (object → ref_id map)
- - [ ] Implement read-side object tracking (ref_id → object list)
- - [ ] Handle all four reference flags: NULL(-3), REF(-2), NOT_NULL(-1),
REF_VALUE(0)
- - [ ] Support disabling reference tracking per-type or globally
-
-### Phase 3: Collection Types
-
-8. **List/Array Serialization**
- - [ ] Write length as varuint32
- - [ ] Write elements header byte
- - [ ] Handle homogeneous vs heterogeneous elements
- - [ ] Handle null elements
-
-9. **Map Serialization**
- - [ ] Write total size as varuint32
- - [ ] Implement chunk-based format (max 255 pairs per chunk)
- - [ ] Write KV header byte per chunk
- - [ ] Handle key and value type variations
-
-10. **Set Serialization**
- - [ ] Same format as List (reuse implementation)
-
-### Phase 4: Meta String Encoding
-
-Meta strings are required for enum and struct serialization (encoding field
names, type names, namespaces).
-
-11. **Meta String Compression**
- - [ ] Implement LOWER_SPECIAL encoding (5 bits/char)
- - [ ] Implement LOWER_UPPER_DIGIT_SPECIAL encoding (6 bits/char)
- - [ ] Implement FIRST_TO_LOWER_SPECIAL encoding
- - [ ] Implement ALL_TO_LOWER_SPECIAL encoding
- - [ ] Implement encoding selection algorithm
- - [ ] Implement meta string deduplication
-
-### Phase 5: Enum Serialization
-
-12. **Enum Serialization**
- - [ ] Write ordinal as varuint32
- - [ ] Support named enum (namespace + type name)
-
-### Phase 6: Struct Serialization
-
-13. **Type Registration**
- - [ ] Support registration by numeric ID
- - [ ] Support registration by namespace + type name
- - [ ] Maintain type → serializer mapping
- - [ ] Generate type IDs: write internal type ID, then `user_type_id` as
varuint32
-
-14. **Field Ordering**
- - [ ] Implement the spec-defined grouping and ordering
(primitive/boxed/built-in, collections/maps, other)
- - [ ] Use a stable comparator within each group (type ID and name)
- - [ ] Use tag ID or snake_case field name as field identifier for
fingerprints
-
-15. **Schema Consistent Mode**
- - [ ] If class-version check is enabled, compute schema hash from field
identifiers
- - [ ] Write 4-byte schema hash before fields
- - [ ] Serialize fields in Fory order
-
-16. **Compatible/Meta Share Mode**
- - [ ] Implement shared TypeDef stream (inline new TypeDefs, index
references)
- - [ ] Map fields by name or tag ID, skip unknown fields
- - [ ] Apply nullable/ref flags from TypeDef metadata
-
-### Phase 7: Other types
-
-17. **Binary/Array Types**
-
-- [ ] Primitive arrays (direct buffer copy)
-- [ ] Multi-dimensional arrays as nested lists (no tensor encoding)
-
-### Testing Strategy
-
-18. **Cross-Language Compatibility Tests**
- - [ ] Serialize in new language, deserialize in Java/Python
- - [ ] Serialize in Java/Python, deserialize in new language
- - [ ] Test all primitive types
- - [ ] Test strings with various encodings
- - [ ] Test collections (empty, single, multiple elements)
- - [ ] Test maps with various key/value types
- - [ ] Test nested structs
- - [ ] Test circular references (if supported)
-
-## Language-Specific Implementation Notes
-
-### Java
-
-- Uses runtime code generation (JIT) for maximum performance
-- Supports all reference tracking modes
-- Uses internal String coder for encoding selection
-- Thread-safe via `ThreadSafeFory` wrapper
-
-### Python
-
-- Two modes: Pure Python (debugging) and Cython (performance)
-- Uses `id(obj)` for reference tracking
-- Latin1/UTF-16/UTF-8 encoding for all strings in xlang mode
-- `dataclass` support via code generation
-
-### C++
-
-- Compile-time reflection via macros (`FORY_STRUCT`)
-- Template meta programming for type dispatch and serializer selection
-- Uses `std::shared_ptr` for reference tracking
-- Compile-time field ordering
-- No runtime code generation
-
-### Rust
-
-- Derive macros for automatic serialization (`#[derive(ForyObject)]`)
-- Uses `Rc<T>` / `Arc<T>` for reference tracking
-- Thread-local context caching for performance
-- Compile-time field ordering
-
-### Go
-
-- Reflection-based and codegen-based modes
-- Struct tags for field annotations
-- Interface types for polymorphism
-
## Common Pitfalls
1. **Byte Order**: Always use little-endian for multi-byte values
@@ -1692,3 +1455,7 @@ Meta strings are required for enum and struct
serialization (encoding field name
6. **Null Handling**: Different languages represent null differently
7. **Empty Collections**: Still write length (0) and header byte
8. **Schema Hash Calculation**: Must use the same fingerprint and MurmurHash3
algorithm across languages when enabled
+
+## Language Implementation Guidelines
+
+See [Xlang Implementation Guide](xlang_implementation_guide.md) documentation.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
