This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/fory-site.git
commit 90a2d25f6e44f542b91a2617d1b210af10d16cd4 Author: chaokunyang <[email protected]> AuthorDate: Mon Aug 3 17:14:50 2026 +0000 π synced local 'docs/guide/' with remote 'docs/guide/' --- docs/guide/java/row-format.md | 27 +++--- docs/guide/python/row-format.md | 16 ++-- docs/guide/rust/basic-serialization.md | 13 ++- docs/guide/rust/index.md | 35 ++----- docs/guide/rust/row-format.md | 169 ++++++++++++++++++++------------- docs/guide/xlang/row_format.md | 63 ++++++++++-- 6 files changed, 195 insertions(+), 128 deletions(-) diff --git a/docs/guide/java/row-format.md b/docs/guide/java/row-format.md index 477f9ec136..471bfb2da9 100644 --- a/docs/guide/java/row-format.md +++ b/docs/guide/java/row-format.md @@ -28,7 +28,7 @@ Row format is a cache-friendly binary random access format that supports: - **Zero-copy access**: Read fields directly from binary without allocating objects - **Partial deserialization**: Access only the fields you need - **Skipping serialization**: Skip serialization of fields you don't need -- **Cross-language compatibility**: Works across Python, Java, C++, and other languages +- **Cross-language compatibility**: Standard rows work across Python, Java, C++, and Rust - **Column format conversion**: Can convert to Apache Arrow columnar format automatically ## Basic Usage @@ -62,7 +62,7 @@ for (int i = 0; i < 1_000_000; i++) { } foo.f4 = bars; -// Encode to row format (cross-language compatible with Python/C++) +// Encode to row format (cross-language compatible with Python/C++/Rust) BinaryRow binaryRow = encoder.toRow(foo); // Zero-copy random access without full deserialization @@ -95,13 +95,13 @@ straight to the ordinal row getter without another schema map lookup or typed ha ## Key Benefits -| Feature | Description | -| ----------------------- | ------------------------------------------------------ | -| Zero-Copy Access | Read nested fields without deserializing entire object | -| Memory Efficiency | Memory-map large datasets directly from disk | -| Cross-Language | Binary format compatible between Java, Python, C++ | -| Partial Deserialization | Deserialize only specific elements you need | -| High Performance | Skip unnecessary data parsing for analytics workloads | +| Feature | Description | +| ----------------------- | -------------------------------------------------------- | +| Zero-Copy Access | Read nested fields without deserializing entire object | +| Memory Efficiency | Memory-map large datasets directly from disk | +| Cross-Language | Binary format compatible between Java, Python, C++, Rust | +| Partial Deserialization | Deserialize only specific elements you need | +| High Performance | Skip unnecessary data parsing for analytics workloads | ## When to Use Row Format @@ -121,20 +121,19 @@ Row format works seamlessly across languages. The same binary data can be access ```python import pyfory -import pyarrow as pa from dataclasses import dataclass from typing import List, Dict @dataclass class Bar: f1: str - f2: List[pa.int64] + f2: List[pyfory.Int64] @dataclass class Foo: - f1: pa.int32 - f2: List[pa.int32] - f3: Dict[str, pa.int32] + f1: pyfory.Int32 + f2: List[pyfory.Int32] + f3: Dict[str, pyfory.Int32] f4: List[Bar] encoder = pyfory.encoder(Foo) diff --git a/docs/guide/python/row-format.md b/docs/guide/python/row-format.md index a9f7219e41..61266be590 100644 --- a/docs/guide/python/row-format.md +++ b/docs/guide/python/row-format.md @@ -27,13 +27,13 @@ Row format drastically reduces overhead when working with large objects where on **Key Benefits:** -| Feature | Description | -| ----------------------- | ------------------------------------------------------ | -| Zero-Copy Access | Read nested fields without deserializing entire object | -| Memory Efficiency | Memory-map large datasets directly from disk | -| Cross-Language | Binary format compatible between Python, Java, C++ | -| Partial Deserialization | Deserialize only specific elements you need | -| High Performance | Skip unnecessary data parsing for analytics workloads | +| Feature | Description | +| ----------------------- | -------------------------------------------------------- | +| Zero-Copy Access | Read nested fields without deserializing entire object | +| Memory Efficiency | Memory-map large datasets directly from disk | +| Cross-Language | Binary format compatible between Python, Java, C++, Rust | +| Partial Deserialization | Deserialize only specific elements you need | +| High Performance | Skip unnecessary data parsing for analytics workloads | ## Basic Usage @@ -102,7 +102,7 @@ standard-library Python `array.array` carrier, not PyArrow. ## Cross-Language Compatibility -Row format works seamlessly across languages. The same binary data can be accessed from Java and C++. +Row format works seamlessly across languages. The same binary data can be accessed from Java, C++, and Rust. ### Java diff --git a/docs/guide/rust/basic-serialization.md b/docs/guide/rust/basic-serialization.md index ea18eaafc8..0d1737e65e 100644 --- a/docs/guide/rust/basic-serialization.md +++ b/docs/guide/rust/basic-serialization.md @@ -152,10 +152,12 @@ fory = { version = "1.5.0", features = ["chrono"] } ### Custom Types -| Macro | Description | -| ----------------------- | -------------------------- | -| `#[derive(ForyStruct)]` | Object graph serialization | -| `#[derive(ForyRow)]` | Row-based serialization | +| Macro | Description | +| ----------------------- | ------------------------------------- | +| `#[derive(ForyStruct)]` | Object graph serialization | +| `#[derive(ForyRow)]` | Standard Row Format for named structs | + +`ForyRow` has a separate type set and returns borrowed field views. Row reads and field access use `Result` to report invalid row data. See [Row Format](row-format.md) for supported types, nullability, and examples. ## Serialization APIs @@ -205,7 +207,7 @@ all supported carriers, and registration. ## Performance Tips -- **Zero-Copy Deserialization**: Row format enables direct memory access without copying +- **Selective Zero-Copy Access**: Row Format returns borrowed views for direct field and element access - **Buffer Pre-allocation**: Minimizes memory allocations during serialization - **Compact Encoding**: Variable-length encoding for space efficiency - **Little-Endian**: Optimized for modern CPU architectures @@ -217,3 +219,4 @@ all supported carriers, and registration. - [References](references.md) - Shared and circular references - [Custom Serializers](custom-serializers.md) - Custom serialization - [External-Type Serialization](external-types.md) - Third-party values and carrier roots +- [Row Format](row-format.md) - Standard Row Format and zero-copy borrowed views diff --git a/docs/guide/rust/index.md b/docs/guide/rust/index.md index 6115e7ff7e..3f676f0dad 100644 --- a/docs/guide/rust/index.md +++ b/docs/guide/rust/index.md @@ -19,7 +19,7 @@ license: | limitations under the License. --- -**Apache Foryβ’** is a blazing fast multi-language serialization framework powered by **JIT compilation** and **zero-copy** techniques, providing up to **ultra-fast performance** while maintaining ease of use and safety. +**Apache Foryβ’** is a high-performance multi-language serialization framework. The Rust implementation uses compile-time code generation for object serialization and borrowed views for zero-copy Row Format access. The Rust implementation provides versatile and high-performance serialization with automatic memory management and compile-time type safety. It supports both xlang mode for cross-language payloads and native mode for Rust-only payloads. @@ -32,7 +32,7 @@ The Rust implementation provides versatile and high-performance serialization wi - **Circular references**: Automatic tracking of shared and circular references with `Rc`/`Arc` and weak pointers - **Polymorphic**: Serialize trait objects with `Box<dyn Trait>`, `Rc<dyn Trait>`, and `Arc<dyn Trait>` - **Schema evolution**: Compatible mode for independent schema changes -- **Two formats**: Object graph serialization and zero-copy row-based format +- **Two formats**: Object graph serialization and the Standard Row Format shared with Java, C++, and Python ## Crates @@ -146,30 +146,6 @@ fn main() -> Result<(), Error> { **Tip:** Perform registrations (such as `fory.register::<T>(id)`) before spawning threads so every worker sees the same metadata. Once configured, wrapping the instance in `Arc` is enough to fan out serialization and deserialization tasks safely. -## Architecture - -The Rust implementation consists of three main crates: - -``` -fory/ # High-level API -βββ src/lib.rs # Public API exports - -fory-core/ # Core serialization engine -βββ src/ -β βββ fory.rs # Main serialization entry point -β βββ buffer.rs # Binary buffer management -β βββ serializer/ # Type-specific serializers -β βββ resolver/ # Type resolution and metadata -β βββ meta/ # Meta string compression -β βββ row/ # Row format implementation -β βββ types.rs # Type definitions - -fory-derive/ # Procedural macros -βββ src/ -β βββ object/ # ForyStruct macro -β βββ fory_row.rs # ForyRow macro -``` - ## Use Cases ### Object Serialization @@ -180,13 +156,14 @@ fory-derive/ # Procedural macros - Schema evolution with compatible mode - Graph-like data structures with circular references -### Row-Based Serialization +### Standard Row Format - High-throughput data processing - Analytics workloads requiring fast field access - Memory-constrained environments - Real-time data streaming applications -- Zero-copy scenarios +- Zero-copy field and collection access +- Standard Row Format interchange with Java, C++, and Python ## Next Steps @@ -199,5 +176,5 @@ fory-derive/ # Procedural macros - [Custom Serializers](custom-serializers.md) - Implement custom serialization behavior - [External-Type Serialization](external-types.md) - External structural and custom serializers plus carrier composition -- [Row Format](row-format.md) - Zero-copy row-based format +- [Row Format](row-format.md) - Standard Row Format with borrowed views - [gRPC Support](grpc-support.md) - Fory payloads over tonic diff --git a/docs/guide/rust/row-format.md b/docs/guide/rust/row-format.md index 9c5c545525..5ae2e3703f 100644 --- a/docs/guide/rust/row-format.md +++ b/docs/guide/rust/row-format.md @@ -19,19 +19,13 @@ license: | limitations under the License. --- -Apache Foryβ’ provides a high-performance **row format** for zero-copy deserialization. +Apache Foryβ’ Rust implements the Standard Row Format used by Java, C++, and Python. It provides zero-copy borrowed views and random field access without reconstructing the complete value. ## Overview -Unlike traditional object serialization that reconstructs entire objects in memory, row format enables **random access** to fields directly from binary data without full deserialization. +Use Row Format when readers need selected fields or collection elements rather than an owned copy of the complete value. The view borrows the input bytes, so the bytes must remain alive while the view is in use. -**Key benefits:** - -- **Zero-copy access**: Read fields without allocating or copying data -- **Partial deserialization**: Access only the fields you need -- **Memory-mapped files**: Work with data larger than RAM -- **Cache-friendly**: Sequential memory layout for better CPU cache utilization -- **Lazy evaluation**: Defer expensive operations until field access +Row Format is schema-driven: the Rust type supplied to `from_row` determines the field types and declaration order. Cross-language readers and writers must use the same schema. ## When to Use Row Format @@ -39,88 +33,133 @@ Unlike traditional object serialization that reconstructs entire objects in memo - Large datasets where only a subset of fields is needed - Memory-constrained environments - High-throughput data pipelines -- Reading from memory-mapped files or shared memory +- Sharing Standard Row Format bytes with Java, C++, or Python ## Basic Usage ```rust -use fory::{to_row, from_row}; -use fory::ForyRow; +use fory::{from_row, to_row, Error, ForyRow}; use std::collections::BTreeMap; #[derive(ForyRow)] struct UserProfile { id: i64, username: String, - email: String, + email: Option<String>, scores: Vec<i32>, preferences: BTreeMap<String, String>, is_active: bool, } -let profile = UserProfile { - id: 12345, - username: "alice".to_string(), - email: "[email protected]".to_string(), - scores: vec![95, 87, 92, 88], - preferences: BTreeMap::from([ - ("theme".to_string(), "dark".to_string()), - ("language".to_string(), "en".to_string()), - ]), - is_active: true, -}; - -// Serialize to row format -let row_data = to_row(&profile).unwrap(); - -// Zero-copy deserialization - no object allocation! -let row = from_row::<UserProfile>(&row_data); - -// Access fields directly from binary data -assert_eq!(row.id(), 12345); -assert_eq!(row.username(), "alice"); -assert_eq!(row.email(), "[email protected]"); -assert_eq!(row.is_active(), true); - -// Access collections efficiently -let scores = row.scores(); -assert_eq!(scores.size(), 4); -assert_eq!(scores.get(0).unwrap(), 95); -assert_eq!(scores.get(1).unwrap(), 87); - -let prefs = row.preferences(); -assert_eq!(prefs.keys().size(), 2); -assert_eq!(prefs.keys().get(0).unwrap(), "language"); -assert_eq!(prefs.values().get(0).unwrap(), "en"); +fn main() -> Result<(), Error> { + let profile = UserProfile { + id: 12345, + username: "alice".to_string(), + email: Some("[email protected]".to_string()), + scores: vec![95, 87, 92, 88], + preferences: BTreeMap::from([ + ("theme".to_string(), "dark".to_string()), + ("language".to_string(), "en".to_string()), + ]), + is_active: true, + }; + + let row_data = to_row(&profile)?; + let row = from_row::<UserProfile>(&row_data)?; + + // Field methods return Result and validate the referenced bytes. + assert_eq!(row.id()?, 12345); + assert_eq!(row.username()?, "alice"); + assert_eq!(row.email()?, Some("[email protected]")); + assert!(row.is_active()?); + + let scores = row.scores()?; + assert_eq!(scores.len(), 4); + assert_eq!(scores.get(0)?, 95); + assert_eq!(scores.get(1)?, 87); + + let preferences = row.preferences()?; + assert_eq!(preferences.keys().len(), 2); + assert_eq!(preferences.keys().get(0)?, "language"); + assert_eq!(preferences.values().get(0)?, "en"); + Ok(()) +} ``` -## How It Works +`to_row` accepts Row Format roots: derived structs, supported arrays, and `BTreeMap` values. Scalar, string, binary, and `Option<T>` values are field or element values rather than standalone roots. + +## Nullability and Field Order + +`Option<T>` declares a nullable field or array element. `None` sets the corresponding null bit, and the field method returns `None` without reading a value body. `Some(value)` uses the same fixed slot width as `T`. + +`#[derive(ForyRow)]` supports named structs, including generic structs. Fields are encoded in source declaration order. The derive generates a borrowed `StructNameRowView` type whose visibility matches the source struct. Each generated field method preserves the corresponding field's visibility and returns `Result<_, Error>`. + +Changing field order or field types changes the Row Format schema. Coordinate such changes across all producers and consumers. + +## Supported Types + +| Rust type | Standard Row Format encoding | +| ---------------------------------------------- | -------------------------------- | +| `bool`, `i8`, `i16`, `i32`, `i64` | Fixed-width scalar | +| `f32`, `f64` | Fixed-width IEEE 754 scalar | +| `Date` | Fixed-width date32 in epoch days | +| `Timestamp` | Fixed-width epoch microseconds | +| `Duration` | Fixed-width microseconds | +| `String`, `&str` | Variable-width UTF-8 | +| `Vec<u8>`, `&[u8]` | Variable-width binary | +| `Vec<T>`, `[T; N]` for supported element types | Standard array | +| `BTreeMap<K, V>` | Standard map | +| Named structs with `#[derive(ForyRow)]` | Nested Standard Row | +| `Option<T>` | Nullable field or array element | + +`Float16` and `Decimal` are not supported because the Standard Row Format specification does not define complete interoperable encodings for them. + +Fixed arrays require the encoded element count to equal `N`. `BTreeMap` keys must implement `Ord`; map values do not need to implement `Ord`. -- Fields are encoded in a binary row with fixed offsets for primitives -- Variable-length data (strings, collections) stored with offset pointers -- Null bitmap tracks which fields are present -- Nested structures supported through recursive row encoding +`Vec<u8>` is encoded as binary rather than as a Standard Array. Use another supported element type when an array representation is required. + +## Standard Binary Layout + +- A row starts with an 8-byte-aligned null bitmap followed by one 8-byte slot per field. +- Fixed-width values are stored little-endian at the low address of their slot. Unused slot bytes are zero. +- A variable-width slot is the little-endian `u64` value `(relative_offset << 32) | size`. The value body and its zero padding follow the fixed region. +- An array starts with a `u64` element count and an 8-byte-aligned null bitmap. Fixed-width elements use contiguous natural-width slots; variable-width elements use 8-byte offset-size slots. +- A map contains the key-array byte size followed by complete key and value arrays. Nested structs, arrays, and maps are complete child structures. +- Variable bodies and array slot regions are padded with zeroes to an 8-byte boundary. Offsets are relative to the immediate containing row or array. + +For the normative layout and size formulas, see the [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec). + +## Validation and Errors + +`from_row`, generated field methods, and array `get` calls return `Result`. They reject truncated fixed regions, invalid counts, out-of-range offsets and sizes, invalid UTF-8, fixed-array length mismatches, and mismatched map key/value counts. + +Array access is also bounds-checked: + +```rust +let scores = row.scores()?; +assert!(scores.get(scores.len()).is_err()); +``` ## Performance Comparison -| Operation | Object Format | Row Format | -| -------------------- | ----------------------------- | ------------------------------- | -| Full deserialization | Allocates all objects | Zero allocation | -| Single field access | Full deserialization required | Direct offset read | -| Memory usage | Full object graph in memory | Only accessed fields in memory | -| Suitable for | Small objects, full access | Large objects, selective access | +| Operation | Object Format | Row Format | +| ----------------- | ----------------------------------- | ---------------------------------------- | +| Open encoded data | Reconstructs an owned value | Creates a borrowed view | +| Read one field | Accesses the reconstructed object | Validates and reads the field directly | +| Collection access | Uses an owned collection | Uses a borrowed array or map view | +| Suitable for | Full object use and graph semantics | Selective access and cross-language rows | ## ForyRow vs ForyStruct -| Feature | `#[derive(ForyRow)]` | `#[derive(ForyStruct)]` | -| --------------- | --------------------- | -------------------------- | -| Deserialization | Zero-copy, lazy | Full object reconstruction | -| Field access | Direct from binary | Normal struct access | -| Memory usage | Minimal | Full object | -| Best for | Analytics, large data | General serialization | +| Feature | `#[derive(ForyRow)]` | `#[derive(ForyStruct)]` | +| ------------ | -------------------------------------- | ---------------------------- | +| Read result | Borrowed view | Owned Rust value | +| Field access | Field method returning `Result` | Normal struct access | +| Schema order | Source declaration order | Object-format schema rules | +| Best for | Selective access to Standard Row bytes | General object serialization | ## Related Topics - [Basic Serialization](basic-serialization.md) - Object graph serialization -- [Xlang Serialization](xlang-serialization.md) - Row format across languages +- [Cross-Language Row Format](../xlang/row_format.md) - Row format across languages - [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec) - Protocol details diff --git a/docs/guide/xlang/row_format.md b/docs/guide/xlang/row_format.md index afc256b7b3..9ff1e4baa3 100644 --- a/docs/guide/xlang/row_format.md +++ b/docs/guide/xlang/row_format.md @@ -19,15 +19,20 @@ license: | limitations under the License. --- -Fory Row Format is a cache-friendly binary format designed for efficient random access and partial serialization. Unlike object graph serialization, row format allows you to read individual fields without deserializing the entire object. +Fory Row Format is a cache-friendly binary format designed for efficient random access. Unlike object graph serialization, Row Format allows readers to access individual fields without reconstructing the complete object. ## Features - **Zero-Copy Random Access**: Read specific fields directly from binary data -- **Partial Serialization**: Skip unnecessary fields during serialization -- **Cross-Language Compatible**: Row format data can be shared between Java, Python, and C++ +- **Selective Access**: Read only the fields or collection elements an application needs +- **Cross-Language Compatible**: Standard Row Format data can be shared between Java, Python, C++, and Rust - **Apache Arrow Integration**: Convert row format to/from Arrow RecordBatch for analytics (Java/Python) +| Format | Implementations | Purpose | +| ------------------- | ----------------------- | -------------------------- | +| Standard Row Format | Java, Python, C++, Rust | Cross-language interchange | +| Compact Row Format | Java | Smaller Java-only rows | + ## Java ```java @@ -78,15 +83,22 @@ Bar newBar2 = barEncoder.fromRow(binaryArray4.getStruct(20)); ## Python ```python +import datetime +import pickle +from dataclasses import dataclass +from typing import Dict, List + +import pyfory + @dataclass class Bar: f1: str - f2: List[pa.int64] + f2: List[pyfory.Int64] @dataclass class Foo: - f1: pa.int32 - f2: List[pa.int32] - f3: Dict[str, pa.int32] + f1: pyfory.Int32 + f2: List[pyfory.Int32] + f3: Dict[str, pyfory.Int32] f4: List[Bar] encoder = pyfory.encoder(Foo) @@ -106,6 +118,42 @@ print(new_foo.f2[100000], new_foo.f4[100000].f1, new_foo.f4[200000].f2[5]) print(f"pickle end: {datetime.datetime.now()}") ``` +## Rust + +Rust derives a static Row Format schema from a named struct. Struct fields use source declaration order, and `Option<T>` declares nullable fields or array elements. The Rust type used by `from_row` must match the producer's field types and order. + +```rust +use fory::{from_row, to_row, Error, ForyRow}; + +#[derive(ForyRow)] +struct Metric { + id: i64, + label: String, + samples: Vec<f64>, + note: Option<String>, +} + +fn main() -> Result<(), Error> { + let bytes = to_row(&Metric { + id: 7, + label: "latency".to_string(), + samples: vec![1.5, 2.0], + note: None, + })?; + + let row = from_row::<Metric>(&bytes)?; + assert_eq!(row.id()?, 7); + assert_eq!(row.label()?, "latency"); + assert_eq!(row.note()?, None); + + let samples = row.samples()?; + assert_eq!(samples.get(1)?, 2.0); + Ok(()) +} +``` + +Rust reads and writes the exact Standard Row Format, including natural-width primitive array elements, little-endian offset-size slots, null bitmaps, and 8-byte zero padding. See the [Rust Row Format Guide](../rust/row-format.md) for supported types and API details. + ## Apache Arrow Support Fory Row Format supports automatic conversion from/to Arrow Table/RecordBatch for analytics workloads. @@ -188,3 +236,4 @@ Parent decoded = encoder.fromRow(row); - [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec) - Binary format details - [Java Row Format Guide](../java/row-format.md) - Java-specific row format documentation - [Python Row Format Guide](../python/row-format.md) - Python-specific row format documentation +- [Rust Row Format Guide](../rust/row-format.md) - Rust-specific row format documentation --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
