This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory-site.git
The following commit(s) were added to refs/heads/main by this push:
new 25e4b8ae6 π synced local 'docs/docs/guide/' with remote 'docs/guide/'
25e4b8ae6 is described below
commit 25e4b8ae669b4a26a3ddfb45e61a8b3c7277bfb6
Author: chaokunyang <[email protected]>
AuthorDate: Tue Dec 30 04:56:42 2025 +0000
π synced local 'docs/docs/guide/' with remote 'docs/guide/'
---
docs/docs/guide/xlang/field-nullability.md | 251 +++++++++++++++++++++
docs/docs/guide/xlang/field-reference-tracking.md | 260 ++++++++++++++++++++++
docs/docs/guide/xlang/getting-started.md | 2 +-
docs/docs/guide/xlang/row_format.md | 2 +-
docs/docs/guide/xlang/serialization.md | 2 +-
docs/docs/guide/xlang/troubleshooting.md | 2 +-
docs/docs/guide/xlang/zero-copy.md | 2 +-
7 files changed, 516 insertions(+), 5 deletions(-)
diff --git a/docs/docs/guide/xlang/field-nullability.md
b/docs/docs/guide/xlang/field-nullability.md
new file mode 100644
index 000000000..d3ba65bdc
--- /dev/null
+++ b/docs/docs/guide/xlang/field-nullability.md
@@ -0,0 +1,251 @@
+---
+title: Field Nullability
+sidebar_position: 40
+id: xlang_field_nullability
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+This page explains how Fory handles field nullability in cross-language
(xlang) serialization mode.
+
+## Default Behavior
+
+In xlang mode, **fields are non-nullable by default**. This means:
+
+- Values must always be present (non-null)
+- No null flag byte is written for the field
+- Serialization is more compact
+
+The following types are nullable by default:
+
+- `Optional<T>` (Java, C++)
+- Java boxed types (`Integer`, `Long`, `Double`, etc.)
+- Go pointer types (`*int32`, `*string`, etc.)
+- Rust `Option<T>`
+- Python `Optional[T]`
+
+| Field Type | Default Nullable | Null Flag
Written |
+| ------------------------------------------ | ---------------- |
----------------- |
+| Primitives (`int`, `bool`, `float`, etc.) | No | No
|
+| `String` | No | No
|
+| `List<T>`, `Map<K,V>`, `Set<T>` | No | No
|
+| Custom structs | No | No
|
+| Enums | No | No
|
+| Java boxed types (`Integer`, `Long`, etc.) | Yes | Yes
|
+| Go pointer types (`*int32`, `*string`) | Yes | Yes
|
+| `Optional<T>` / `Option<T>` | Yes | Yes
|
+
+## Wire Format
+
+The nullable flag controls whether a **null flag byte** is written before the
field value:
+
+```
+Non-nullable field: [value data]
+Nullable field: [null_flag] [value data if not null]
+```
+
+Where `null_flag` is:
+
+- `-1` (NULL_FLAG): Value is null
+- `-2` (NOT_NULL_VALUE_FLAG): Value is present
+
+## Nullable vs Reference Tracking
+
+These are related but distinct concepts:
+
+| Concept | Purpose | Flag Values
|
+| ---------------------- | ------------------------------------ |
------------------------------------------- |
+| **Nullable** | Allow null values for a field | `-1` (null),
`-2` (not null) |
+| **Reference Tracking** | Deduplicate shared object references | `-1` (null),
`-2` (not null), `β₯0` (ref ID) |
+
+Key differences:
+
+- **Nullable only**: Writes `-1` or `-2` flag, no reference deduplication
+- **Reference tracking**: Extends nullable semantics with reference IDs (`β₯0`)
for previously seen objects
+- Both use the same flag byte positionβref tracking is a superset of nullable
+
+When `refTracking=true`, the null flag byte doubles as a ref flag:
+
+```
+ref_flag = -1 β null value
+ref_flag = -2 β new object (first occurrence)
+ref_flag >= 0 β reference to object at index ref_flag
+```
+
+For detailed reference tracking behavior, see [Reference
Tracking](field-reference-tracking.md).
+
+## Language-Specific Examples
+
+### Java
+
+```java
+public class Person {
+ // Non-nullable by default in xlang mode
+ String name; // Must not be null
+ int age; // Primitive, always non-nullable
+ List<String> tags; // Must not be null
+
+ // Explicitly nullable
+ @ForyField(nullable = true)
+ String nickname; // Can be null
+
+ // Optional wrapper - nullable by default
+ Optional<String> bio; // Can be empty/null
+}
+
+Fory fory = Fory.builder()
+ .withLanguage(Language.XLANG)
+ .build();
+fory.register(Person.class, "example.Person");
+```
+
+### Python
+
+```python
+from dataclasses import dataclass
+from typing import Optional, List
+import pyfory
+
+@dataclass
+class Person:
+ # Non-nullable by default
+ name: str # Must have a value
+ age: pyfory.int32 # Primitive
+ tags: List[str] # Must not be None
+
+ # Optional makes it nullable
+ nickname: Optional[str] = None # Can be None
+ bio: Optional[str] = None # Can be None
+
+fory = pyfory.Fory(xlang=True)
+fory.register_type(Person, typename="example.Person")
+```
+
+### Rust
+
+```rust
+use fory::Fory;
+
+#[derive(Fory)]
+#[tag("example.Person")]
+struct Person {
+ // Non-nullable by default
+ name: String,
+ age: i32,
+ tags: Vec<String>,
+
+ // Option<T> is nullable
+ nickname: Option<String>, // Can be None
+ bio: Option<String>, // Can be None
+}
+```
+
+### Go
+
+```go
+type Person struct {
+ // Non-nullable by default
+ Name string
+ Age int32
+ Tags []string
+
+ // Pointer types for nullable fields
+ Nickname *string // Can be nil
+ Bio *string // Can be nil
+}
+
+fory := forygo.NewFory()
+fory.RegisterTagType("example.Person", Person{})
+```
+
+### C++
+
+```cpp
+struct Person {
+ // Non-nullable by default
+ std::string name;
+ int32_t age;
+ std::vector<std::string> tags;
+
+ // std::optional for nullable
+ std::optional<std::string> nickname;
+ std::optional<std::string> bio;
+};
+FORY_STRUCT(Person, name, age, tags, nickname, bio);
+```
+
+## Customizing Nullability
+
+### Java: @ForyField Annotation
+
+```java
+public class Config {
+ @ForyField(nullable = true)
+ String optionalSetting; // Explicitly nullable
+
+ @ForyField(nullable = false)
+ String requiredSetting; // Explicitly non-nullable (default)
+}
+```
+
+### C++: fory::field Wrapper
+
+```cpp
+struct Config {
+ // Explicitly mark as nullable
+ fory::field<std::string, 1, fory::nullable<true>> optional_setting;
+
+ // Explicitly mark as non-nullable (default)
+ fory::field<std::string, 2, fory::nullable<false>> required_setting;
+};
+FORY_STRUCT(Config, optional_setting, required_setting);
+```
+
+## Null Value Handling
+
+When a non-nullable field receives a null value:
+
+| Language | Behavior |
+| -------- | ---------------------------------------------------- |
+| Java | Throws `NullPointerException` or serialization error |
+| Python | Raises `TypeError` or serialization error |
+| Rust | Compile-time error (non-Option types can't be None) |
+| Go | Zero value is used (empty string, 0, etc.) |
+| C++ | Default-constructed value or undefined behavior |
+
+## Schema Compatibility
+
+The nullable flag is part of the struct schema fingerprint. Changing a field's
nullability is a **breaking change** that will cause schema version mismatch
errors.
+
+```
+Schema A: { name: String (non-nullable) }
+Schema B: { name: String (nullable) }
+// These have different fingerprints and are incompatible
+```
+
+## Best Practices
+
+1. **Use non-nullable by default**: Only make fields nullable when null is a
valid semantic value
+2. **Use Optional/Option wrappers**: Instead of raw types with nullable
annotation
+3. **Be consistent across languages**: Use the same nullability for
corresponding fields
+4. **Document nullable fields**: Make it clear which fields can be null in
your API
+
+## See Also
+
+- [Reference Tracking](field-reference-tracking.md) - Shared and circular
reference handling
+- [Serialization](serialization.md) - Basic cross-language serialization
+- [Type
Mapping](https://fory.apache.org/docs/specification/xlang_type_mapping) -
Cross-language type mapping reference
+- [Xlang
Specification](https://fory.apache.org/docs/specification/fory_xlang_serialization_spec)
- Binary protocol details
diff --git a/docs/docs/guide/xlang/field-reference-tracking.md
b/docs/docs/guide/xlang/field-reference-tracking.md
new file mode 100644
index 000000000..e003c59f5
--- /dev/null
+++ b/docs/docs/guide/xlang/field-reference-tracking.md
@@ -0,0 +1,260 @@
+---
+title: Reference Tracking
+sidebar_position: 45
+id: xlang_reference_tracking
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+This page explains how Fory handles reference tracking for shared and circular
references in cross-language serialization.
+
+## Overview
+
+Reference tracking enables:
+
+- **Shared references**: Same object referenced multiple times is serialized
once
+- **Circular references**: Objects that reference themselves or form cycles
+- **Memory efficiency**: No duplicate data for repeated objects
+
+## Enabling Reference Tracking
+
+### Java
+
+```java
+Fory fory = Fory.builder()
+ .withLanguage(Language.XLANG)
+ .withRefTracking(true)
+ .build();
+```
+
+### Python
+
+```python
+fory = pyfory.Fory(xlang=True, ref_tracking=True)
+```
+
+### Go
+
+```go
+fory := forygo.NewFory(true) // true enables ref tracking
+```
+
+### C++
+
+```cpp
+auto fory = fory::Fory::create(fory::Config{
+ .ref_tracking = true
+});
+```
+
+### Rust
+
+```rust
+let fory = Fory::builder()
+ .with_ref_tracking(true)
+ .build();
+```
+
+## Wire Format
+
+When reference tracking is enabled, nullable fields write a **ref flag byte**
before the value:
+
+```
+[ref_flag] [value data if not null/ref]
+```
+
+Where `ref_flag` is:
+
+| Value | Meaning
|
+| -------------------------- |
----------------------------------------------------- |
+| `-1` (NULL_FLAG) | Value is null
|
+| `-2` (NOT_NULL_VALUE_FLAG) | Value is present, first occurrence
|
+| `β₯0` | Reference ID pointing to previously serialized
object |
+
+## Reference Tracking vs Nullability
+
+These are **independent** concepts:
+
+| Concept | Purpose |
Controlled By |
+| ---------------------- | ------------------------------------------ |
---------------------------------------- |
+| **Nullability** | Whether a field can hold null values | Field
type (`Optional<T>`) or annotation |
+| **Reference Tracking** | Whether duplicate objects are deduplicated | Global
`refTracking` option |
+
+Key behavior:
+
+- Ref flag bytes are **only written for nullable fields**
+- Non-nullable fields skip ref flags entirely, even with `refTracking=true`
+- Reference deduplication only applies to objects that appear multiple times
+
+```java
+// Reference tracking enabled, but non-nullable fields still skip ref flags
+Fory fory = Fory.builder()
+ .withLanguage(Language.XLANG)
+ .withRefTracking(true)
+ .build();
+```
+
+## Per-Field Reference Tracking
+
+By default, **most fields do not track references** even when global
`refTracking=true`. Only specific pointer/smart pointer types track references
by default.
+
+### Default Behavior by Language
+
+| Language | Default Ref Tracking | Types That Track Refs by Default |
+| -------- | -------------------- | -------------------------------- |
+| Java | No | None (use annotation to enable) |
+| Python | No | None (use annotation to enable) |
+| Go | No | Pointer types (`*T`) |
+| C++ | No | `std::shared_ptr<T>` |
+| Rust | No | `Rc<T>`, `Arc<T>`, `Weak<T>` |
+
+### Customizing Per-Field Ref Tracking
+
+#### Java: @ForyField Annotation
+
+```java
+public class Document {
+ // Default: no ref tracking
+ String title;
+
+ // Enable ref tracking for this field
+ @ForyField(trackingRef = true)
+ Author author;
+
+ // Shared across documents, track refs to avoid duplicates
+ @ForyField(trackingRef = true)
+ List<Tag> tags;
+}
+```
+
+#### C++: fory::field Wrapper
+
+```cpp
+struct Document {
+ std::string title;
+
+ // shared_ptr tracks refs by default
+ std::shared_ptr<Author> author;
+
+ // Explicitly enable ref tracking
+ fory::field<std::vector<Tag>, 1, fory::track_ref<true>> tags;
+
+ // Explicitly disable ref tracking
+ fory::field<std::shared_ptr<Data>, 2, fory::track_ref<false>> data;
+};
+FORY_STRUCT(Document, title, author, tags, data);
+```
+
+#### Rust: Field Attributes
+
+```rust
+#[derive(Fory)]
+#[tag("example.Document")]
+struct Document {
+ title: String,
+
+ // Rc/Arc track refs by default
+ author: Rc<Author>,
+
+ // Explicitly enable ref tracking
+ #[track_ref]
+ tags: Vec<Tag>,
+}
+```
+
+#### Go: Struct Tags
+
+```go
+type Document struct {
+ Title string
+
+ // Pointer types track refs by default
+ Author *Author
+
+ // Use struct tag to control ref tracking
+ Tags []Tag `fory:"trackRef"`
+}
+```
+
+### When to Enable Per-Field Ref Tracking
+
+Enable ref tracking for fields that:
+
+- May contain the same object instance multiple times
+- Are part of circular reference chains
+- Hold large objects that might be shared
+
+Disable (or leave default) for fields that:
+
+- Always contain unique values
+- Are primitives or simple value types
+- Don't participate in object sharing
+
+## Example: Shared References
+
+```java
+public class Container {
+ List<String> data;
+ List<String> sameData; // Points to same list
+}
+
+Container obj = new Container();
+obj.data = Arrays.asList("a", "b", "c");
+obj.sameData = obj.data; // Shared reference
+
+// With refTracking=true: data serialized once, sameData stores reference ID
+// With refTracking=false: data serialized twice (duplicate)
+```
+
+## Example: Circular References
+
+```java
+public class Node {
+ String value;
+ Node next;
+}
+
+Node a = new Node("A");
+Node b = new Node("B");
+a.next = b;
+b.next = a; // Circular reference
+
+// With refTracking=true: works correctly
+// With refTracking=false: infinite recursion error
+```
+
+## Language Support
+
+| Language | Shared Refs | Circular Refs |
+| ---------- | ----------- | -------------------- |
+| Java | Yes | Yes |
+| Python | Yes | Yes |
+| Go | Yes | Yes |
+| C++ | Yes | Yes |
+| JavaScript | Yes | Yes |
+| Rust | Yes | No (ownership rules) |
+
+## Performance Considerations
+
+- **Overhead**: Reference tracking adds a hash map lookup per object
+- **When to enable**: Use when data has shared/circular references
+- **When to disable**: Use for simple data structures without sharing
+
+## See Also
+
+- [Field Nullability](field-nullability.md) - How nullability affects
serialization
+- [Serialization](serialization.md) - Basic cross-language serialization
examples
+- [Xlang
Specification](https://fory.apache.org/docs/specification/fory_xlang_serialization_spec)
- Binary protocol details
diff --git a/docs/docs/guide/xlang/getting-started.md
b/docs/docs/guide/xlang/getting-started.md
index adf77b353..f2eb5e41f 100644
--- a/docs/docs/guide/xlang/getting-started.md
+++ b/docs/docs/guide/xlang/getting-started.md
@@ -1,6 +1,6 @@
---
title: Getting Started
-sidebar_position: 1
+sidebar_position: 10
id: xlang_getting_started
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
diff --git a/docs/docs/guide/xlang/row_format.md
b/docs/docs/guide/xlang/row_format.md
index 58b20fe5c..c17dc9800 100644
--- a/docs/docs/guide/xlang/row_format.md
+++ b/docs/docs/guide/xlang/row_format.md
@@ -1,6 +1,6 @@
---
title: Row Format
-sidebar_position: 5
+sidebar_position: 60
id: row_format
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
diff --git a/docs/docs/guide/xlang/serialization.md
b/docs/docs/guide/xlang/serialization.md
index 0b2085ab0..fa8d1cc3b 100644
--- a/docs/docs/guide/xlang/serialization.md
+++ b/docs/docs/guide/xlang/serialization.md
@@ -1,6 +1,6 @@
---
title: Serialization
-sidebar_position: 3
+sidebar_position: 30
id: xlang_serialization
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
diff --git a/docs/docs/guide/xlang/troubleshooting.md
b/docs/docs/guide/xlang/troubleshooting.md
index 26c4707ed..dd3eed952 100644
--- a/docs/docs/guide/xlang/troubleshooting.md
+++ b/docs/docs/guide/xlang/troubleshooting.md
@@ -1,6 +1,6 @@
---
title: Troubleshooting
-sidebar_position: 6
+sidebar_position: 70
id: xlang_troubleshooting
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
diff --git a/docs/docs/guide/xlang/zero-copy.md
b/docs/docs/guide/xlang/zero-copy.md
index 3d2e83b1a..18e5b736a 100644
--- a/docs/docs/guide/xlang/zero-copy.md
+++ b/docs/docs/guide/xlang/zero-copy.md
@@ -1,6 +1,6 @@
---
title: Zero-Copy Serialization
-sidebar_position: 4
+sidebar_position: 50
id: xlang_zero_copy
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
