This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/fory-site.git
commit a1957d3d3ed23cbbdf77f0444ae8502583a9edd3 Author: chaokunyang <[email protected]> AuthorDate: Mon May 4 13:02:28 2026 +0000 🔄 synced local 'docs/compiler/' with remote 'docs/compiler/' --- docs/compiler/generated-code.md | 28 ++++++- docs/compiler/index.md | 5 +- docs/compiler/protobuf-idl.md | 20 ++--- docs/compiler/schema-idl.md | 171 ++++++++++++++++++++++++++-------------- 4 files changed, 151 insertions(+), 73 deletions(-) diff --git a/docs/compiler/generated-code.md b/docs/compiler/generated-code.md index 3462ba02dd..6fcdc5f373 100644 --- a/docs/compiler/generated-code.md +++ b/docs/compiler/generated-code.md @@ -954,7 +954,33 @@ final class Animal { Nested types use flat underscore naming (e.g., `Person_PhoneNumber`, `Person_PhoneType`). -Non-optional, non-ref lists of primitive types use typed arrays for zero-copy performance (e.g., `list<int32>` → `Int32List`). +`list<T>` fields generate ordered collection carriers and use the Fory list +protocol. `array<T>` fields generate dense one-dimensional bool or numeric +carriers and use the specialized dense-array protocol. Generated code must not +choose `array<T>` only because a language has an optimized list-like carrier; +the schema kind comes from the IDL. + +| IDL schema | Dart generated carrier | Notes | +| ------------------- | ---------------------- | ------------------------------------------ | +| `list<int32>` | `List<int>` | List protocol, varint element encoding | +| `list<fixed int32>` | `List<int>` | List protocol, fixed-width element segment | +| `array<bool>` | `BoolList` | One byte per bool | +| `array<int8>` | `Int8List` | Dense signed bytes | +| `array<int16>` | `Int16List` | Dense little-endian int16 | +| `array<int32>` | `Int32List` | Dense little-endian int32 | +| `array<int64>` | `Int64List` | Dense little-endian int64 | +| `array<uint8>` | `Uint8List` | Dense unsigned bytes | +| `array<uint16>` | `Uint16List` | Dense little-endian uint16 | +| `array<uint32>` | `Uint32List` | Dense little-endian uint32 | +| `array<uint64>` | `Uint64List` | Dense little-endian uint64 | +| `array<float16>` | `Float16List` | Dense binary16 storage | +| `array<bfloat16>` | `Bfloat16List` | Dense bfloat16 storage | +| `array<float32>` | `Float32List` | Dense little-endian float32 | +| `array<float64>` | `Float64List` | Dense little-endian float64 | + +Generated Dart fields that use `ArrayType(element: BoolType())` must use +`BoolList`; plain `List<bool>` remains the generated and handwritten carrier +for `list<bool>`. Reference tracking on list elements or map values uses the container sugar annotations: diff --git a/docs/compiler/index.md b/docs/compiler/index.md index 605a4e6097..707a679fcb 100644 --- a/docs/compiler/index.md +++ b/docs/compiler/index.md @@ -182,7 +182,8 @@ data = bytes(person) # or `person.to_bytes()` - **`optional`**: Field can be null/None - **`ref`**: Enable reference tracking for shared/circular references -- **`list`**: Field is a list/array (alias: `repeated`) +- **`list`**: Field is an ordered collection (alias: `repeated`) +- **`array`**: Field is dense one-dimensional bool or numeric data ```protobuf message Example { @@ -198,7 +199,7 @@ Fory IDL types map to native types in each language: | Fory IDL Type | Java | Python | Go | Rust | C++ | C# | JavaScript | Swift | Dart | | ------------- | --------- | -------------- | -------- | -------- | ------------- | -------- | ---------- | -------- | -------- | -| `int32` | `int` | `pyfory.int32` | `int32` | `i32` | `int32_t` | `int` | `number` | `Int32` | `int` | +| `int32` | `int` | `pyfory.Int32` | `int32` | `i32` | `int32_t` | `int` | `number` | `Int32` | `int` | | `string` | `String` | `str` | `string` | `String` | `std::string` | `string` | `string` | `String` | `String` | | `bool` | `boolean` | `bool` | `bool` | `bool` | `bool` | `bool` | `boolean` | `Bool` | `bool` | diff --git a/docs/compiler/protobuf-idl.md b/docs/compiler/protobuf-idl.md index f0dc62971b..60dabe49f0 100644 --- a/docs/compiler/protobuf-idl.md +++ b/docs/compiler/protobuf-idl.md @@ -189,8 +189,8 @@ Protobuf imports are supported. Common well-known types map directly: | `map<K, V>` | `map<K, V>` | | `optional T` | `optional T` | | `oneof` | `union` + optional union reference field | -| `int64 [(fory).type = "tagged_int64"]` | `tagged_int64` encoding | -| `uint64 [(fory).type = "tagged_uint64"]` | `tagged_uint64` encoding | +| `int64 [(fory).type = "tagged int64"]` | `tagged int64` encoding | +| `uint64 [(fory).type = "tagged uint64"]` | `tagged uint64` encoding | ## Fory Extension Options (Protobuf) @@ -229,14 +229,14 @@ message TreeNode { ### Field-Level Options -| Option | Type | Description | -| ---------------------------- | ------ | ------------------------------------------------------------ | -| `(fory).ref` | bool | Enable reference tracking for this field | -| `(fory).nullable` | bool | Treat field as nullable (`optional`) | -| `(fory).weak_ref` | bool | Generate weak pointer semantics (C++/Rust codegen) | -| `(fory).thread_safe_pointer` | bool | Rust pointer flavor for ref fields (`Arc` vs `Rc`) | -| `(fory).deprecated` | bool | Mark field as deprecated | -| `(fory).type` | string | Primitive override, currently `tagged_int64`/`tagged_uint64` | +| Option | Type | Description | +| ---------------------------- | ------ | ----------------------------------------------------- | +| `(fory).ref` | bool | Enable reference tracking for this field | +| `(fory).nullable` | bool | Treat field as nullable (`optional`) | +| `(fory).weak_ref` | bool | Generate weak pointer semantics (C++/Rust codegen) | +| `(fory).thread_safe_pointer` | bool | Rust pointer flavor for ref fields (`Arc` vs `Rc`) | +| `(fory).deprecated` | bool | Mark field as deprecated | +| `(fory).type` | string | Primitive override for tagged 64-bit integer encoding | Reference option behavior: diff --git a/docs/compiler/schema-idl.md b/docs/compiler/schema-idl.md index 315fc97e92..68bcf70d8d 100644 --- a/docs/compiler/schema-idl.md +++ b/docs/compiler/schema-idl.md @@ -854,8 +854,9 @@ list<ref Node> nodes = 4; // Elements tracked as references ``` field_def := [modifiers] field_type IDENTIFIER '=' INTEGER ';' modifiers := { 'optional' | 'ref' } -field_type := primitive_type | named_type | list_type | map_type -list_type := 'list' '<' { 'optional' | 'ref' } field_type '>' +field_type := primitive_type | named_type | list_type | array_type | map_type +list_type := 'list' '<' { 'optional' | 'ref' | scalar_encoding } field_type '>' +array_type := 'array' '<' array_element_type '>' ``` Modifiers apply to the field/collection. Use `list<...>` to describe element @@ -929,7 +930,7 @@ to customize pointer types. For protobuf option syntax, see #### `list` -Marks the field as a list/array: +Marks the field as an ordered collection: ```protobuf message Document { @@ -1002,38 +1003,34 @@ message Example { ## Type System Fory IDL provides a cross-language type system for primitives, named types, and -collections. Field modifiers (`optional`, `list`, `ref`) control nullability, -collection behavior, and reference tracking (see -[Field Modifiers](#field-modifiers)). +collections. Field modifiers (`optional`, `ref`) control nullability and +reference tracking, while `list<T>` and `array<T>` choose collection schema kind +(see [Field Modifiers](#field-modifiers)). ### Primitive Types -| Type | Description | Size | -| --------------- | ----------------------------------------- | -------- | -| `bool` | Boolean value | 1 byte | -| `int8` | Signed 8-bit integer | 1 byte | -| `int16` | Signed 16-bit integer | 2 bytes | -| `int32` | Signed 32-bit integer (varint encoding) | 4 bytes | -| `int64` | Signed 64-bit integer (varint encoding) | 8 bytes | -| `uint8` | Unsigned 8-bit integer | 1 byte | -| `uint16` | Unsigned 16-bit integer | 2 bytes | -| `uint32` | Unsigned 32-bit integer (varint encoding) | 4 bytes | -| `uint64` | Unsigned 64-bit integer (varint encoding) | 8 bytes | -| `fixed_int32` | Signed 32-bit integer (fixed encoding) | 4 bytes | -| `fixed_int64` | Signed 64-bit integer (fixed encoding) | 8 bytes | -| `fixed_uint32` | Unsigned 32-bit integer (fixed encoding) | 4 bytes | -| `fixed_uint64` | Unsigned 64-bit integer (fixed encoding) | 8 bytes | -| `tagged_int64` | Signed 64-bit integer (tagged encoding) | 8 bytes | -| `tagged_uint64` | Unsigned 64-bit integer (tagged encoding) | 8 bytes | -| `float32` | 32-bit floating point | 4 bytes | -| `float64` | 64-bit floating point | 8 bytes | -| `string` | UTF-8 string | Variable | -| `bytes` | Binary data | Variable | -| `date` | Calendar date | Variable | -| `timestamp` | Date and time with timezone | Variable | -| `duration` | Duration | Variable | -| `decimal` | Decimal value | Variable | -| `any` | Dynamic value (runtime type) | Variable | +| Type | Description | Size | +| ----------- | ---------------------------------------------- | -------- | +| `bool` | Boolean value | 1 byte | +| `int8` | Signed 8-bit integer | 1 byte | +| `int16` | Signed 16-bit integer | 2 bytes | +| `int32` | Signed 32-bit integer, varint by default | 4 bytes | +| `int64` | Signed 64-bit integer, PVL varint by default | 8 bytes | +| `uint8` | Unsigned 8-bit integer | 1 byte | +| `uint16` | Unsigned 16-bit integer | 2 bytes | +| `uint32` | Unsigned 32-bit integer, varint by default | 4 bytes | +| `uint64` | Unsigned 64-bit integer, PVL varint by default | 8 bytes | +| `float16` | IEEE 754 binary16 floating point | 2 bytes | +| `bfloat16` | Brain floating point | 2 bytes | +| `float32` | 32-bit floating point | 4 bytes | +| `float64` | 64-bit floating point | 8 bytes | +| `string` | UTF-8 string | Variable | +| `bytes` | Binary data | Variable | +| `date` | Calendar date | Variable | +| `timestamp` | Date and time with timezone | Variable | +| `duration` | Duration | Variable | +| `decimal` | Decimal value | Variable | +| `any` | Dynamic value (runtime type) | Variable | #### Boolean @@ -1062,10 +1059,10 @@ Fory IDL provides fixed-width signed integers (varint encoding for 32/64-bit by | Fory IDL | Java | Python | Go | Rust | C++ | JavaScript | Dart | | -------- | ------- | -------------- | ------- | ----- | --------- | ------------------ | ------- | -| `int8` | `byte` | `pyfory.int8` | `int8` | `i8` | `int8_t` | `number` | `int` | -| `int16` | `short` | `pyfory.int16` | `int16` | `i16` | `int16_t` | `number` | `int` | -| `int32` | `int` | `pyfory.int32` | `int32` | `i32` | `int32_t` | `number` | `int` | -| `int64` | `long` | `pyfory.int64` | `int64` | `i64` | `int64_t` | `bigint \| number` | `Int64` | +| `int8` | `byte` | `pyfory.Int8` | `int8` | `i8` | `int8_t` | `number` | `int` | +| `int16` | `short` | `pyfory.Int16` | `int16` | `i16` | `int16_t` | `number` | `int` | +| `int32` | `int` | `pyfory.Int32` | `int32` | `i32` | `int32_t` | `number` | `int` | +| `int64` | `long` | `pyfory.Int64` | `int64` | `i64` | `int64_t` | `bigint \| number` | `Int64` | Fory IDL provides fixed-width unsigned integers (varint encoding for 32/64-bit by default): @@ -1080,24 +1077,32 @@ Fory IDL provides fixed-width unsigned integers (varint encoding for 32/64-bit b | Fory IDL | Java | Python | Go | Rust | C++ | JavaScript | Dart | | -------- | ------- | --------------- | -------- | ----- | ---------- | ------------------ | -------- | -| `uint8` | `short` | `pyfory.uint8` | `uint8` | `u8` | `uint8_t` | `number` | `int` | -| `uint16` | `int` | `pyfory.uint16` | `uint16` | `u16` | `uint16_t` | `number` | `int` | -| `uint32` | `long` | `pyfory.uint32` | `uint32` | `u32` | `uint32_t` | `number` | `int` | -| `uint64` | `long` | `pyfory.uint64` | `uint64` | `u64` | `uint64_t` | `bigint \| number` | `Uint64` | +| `uint8` | `short` | `pyfory.UInt8` | `uint8` | `u8` | `uint8_t` | `number` | `int` | +| `uint16` | `int` | `pyfory.UInt16` | `uint16` | `u16` | `uint16_t` | `number` | `int` | +| `uint32` | `long` | `pyfory.UInt32` | `uint32` | `u32` | `uint32_t` | `number` | `int` | +| `uint64` | `long` | `pyfory.UInt64` | `uint64` | `u64` | `uint64_t` | `bigint \| number` | `Uint64` | -#### Integer Encoding Variants +#### Integer Encoding Modifiers -For 32/64-bit integers, Fory IDL uses varint encoding by default. Use explicit types when -you need fixed-width or tagged encoding: +For 32/64-bit integers, Fory IDL uses variable-length encoding by default. Add a +scalar encoding modifier when you need a different wire encoding: -| Fory IDL Type | Encoding | Notes | -| --------------- | -------- | ------------------------ | -| `fixed_int32` | fixed | Signed 32-bit | -| `fixed_int64` | fixed | Signed 64-bit | -| `fixed_uint32` | fixed | Unsigned 32-bit | -| `fixed_uint64` | fixed | Unsigned 64-bit | -| `tagged_int64` | tagged | Signed 64-bit (hybrid) | -| `tagged_uint64` | tagged | Unsigned 64-bit (hybrid) | +| Modifier | Valid types | Notes | +| -------- | ------------------------------------ | ---------------------------- | +| `varint` | `int32`, `int64`, `uint32`, `uint64` | Explicit spelling of default | +| `fixed` | `int32`, `int64`, `uint32`, `uint64` | Fixed-width little-endian | +| `tagged` | `int64`, `uint64` | Tagged 64-bit encoding | + +Modifiers are part of the scalar type expression, so they can be used in nested +list and map positions: + +```protobuf +fixed int32 id = 1; +list<fixed int32> offsets = 2; +map<string, tagged uint64> counters = 3; +``` + +Underscore spellings for integer encoding are not FDL type names. #### Floating-Point Types @@ -1108,10 +1113,12 @@ you need fixed-width or tagged encoding: **Language Mapping:** -| Fory IDL | Java | Python | Go | Rust | C++ | JavaScript | Dart | -| --------- | -------- | ---------------- | --------- | ----- | -------- | ---------- | --------- | -| `float32` | `float` | `pyfory.float32` | `float32` | `f32` | `float` | `number` | `Float32` | -| `float64` | `double` | `pyfory.float64` | `float64` | `f64` | `double` | `number` | `double` | +| Fory IDL | Java | Python annotation/value | Go | Rust | C++ | JavaScript | Dart | +| ---------- | ---------- | --------------------------- | ------------------- | ---------- | ------------------ | ---------- | ---------- | +| `float16` | `Float16` | `pyfory.Float16` / `float` | `float16.Float16` | `Float16` | `fory::float16_t` | `number` | `Float16` | +| `bfloat16` | `BFloat16` | `pyfory.BFloat16` / `float` | `bfloat16.BFloat16` | `BFloat16` | `fory::bfloat16_t` | `BFloat16` | `BFloat16` | +| `float32` | `float` | `pyfory.Float32` | `float32` | `f32` | `float` | `number` | `Float32` | +| `float64` | `double` | `pyfory.Float64` | `float64` | `f64` | `double` | `number` | `double` | #### String Type @@ -1243,6 +1250,50 @@ nested collection specs such as `list<list<...>>`, `list<map<...>>`, and continue to reject them. Use a message wrapper when you need portable schemas across all targets. +#### Array (`array`) + +Use `array<T>` for dynamic-length dense numeric data. `array<T>` is a distinct +schema kind from `list<T>` and uses the packed primitive-array wire payload. + +```protobuf +message Embedding { + array<int32> indices = 1; + array<float32> values = 2; + array<uint8> pixels = 3; +} +``` + +`array<T>` accepts `bool`, integer, and floating-point element domains only. It +does not accept `optional`, `ref`, named/object types, `string`, `bytes`, maps, +or scalar integer encoding modifiers such as `array<fixed int32>`; array +elements are always fixed-width by the array contract. + +Generated carriers are language-specific, but the schema kind is not: + +| IDL schema | Java default | Python default | Dart default | JavaScript/TypeScript | +| ----------------- | ---------------------------- | ---------------------- | -------------- | ------------------------ | +| `list<bool>` | `BoolList` / `List<Boolean>` | `List[bool]` | `List<bool>` | `Type.list(Type.bool())` | +| `array<bool>` | `boolean[]` | `pyfory.BoolArray` | `BoolList` | `Type.boolArray()` | +| `array<int8>` | `byte[]` | `pyfory.Int8Array` | `Int8List` | `Type.int8Array()` | +| `array<int16>` | `short[]` | `pyfory.Int16Array` | `Int16List` | `Type.int16Array()` | +| `array<int32>` | `int[]` | `pyfory.Int32Array` | `Int32List` | `Type.int32Array()` | +| `array<int64>` | `long[]` | `pyfory.Int64Array` | `Int64List` | `Type.int64Array()` | +| `array<uint8>` | `@UInt8Type byte[]` | `pyfory.UInt8Array` | `Uint8List` | `Type.uint8Array()` | +| `array<uint16>` | `@UInt16Type short[]` | `pyfory.UInt16Array` | `Uint16List` | `Type.uint16Array()` | +| `array<uint32>` | `@UInt32Type int[]` | `pyfory.UInt32Array` | `Uint32List` | `Type.uint32Array()` | +| `array<uint64>` | `@UInt64Type long[]` | `pyfory.UInt64Array` | `Uint64List` | `Type.uint64Array()` | +| `array<float16>` | `Float16Array` | `pyfory.Float16Array` | `Float16List` | `Type.float16Array()` | +| `array<bfloat16>` | `BFloat16Array` | `pyfory.BFloat16Array` | `Bfloat16List` | `Type.bfloat16Array()` | +| `array<float32>` | `float[]` | `pyfory.Float32Array` | `Float32List` | `Type.float32Array()` | +| `array<float64>` | `double[]` | `pyfory.Float64Array` | `Float64List` | `Type.float64Array()` | + +For handwritten Dart models, `array<bool>` requires `BoolList` plus +`@ArrayField(element: BoolType())` or +`@ForyField(type: ArrayType(element: BoolType()))`; `List<bool>` remains +`list<bool>`. For handwritten Java models, unsigned primitive arrays use +type-use annotations on the element type, for example +`private @UInt32Type int[] ids;`. + #### Map Maps with typed keys and values: @@ -1469,19 +1520,19 @@ reserved_item := INTEGER | INTEGER 'to' INTEGER | INTEGER 'to' 'max' | STRING modifiers := { 'optional' | 'ref' | 'repeated' } -field_type := primitive_type | named_type | list_type | map_type +field_type := [scalar_encoding] (primitive_type | named_type | list_type | array_type | map_type) primitive_type := 'bool' | 'int8' | 'int16' | 'int32' | 'int64' | 'uint8' | 'uint16' | 'uint32' | 'uint64' - | 'fixed_int32' | 'fixed_int64' | 'fixed_uint32' | 'fixed_uint64' - | 'tagged_int64' | 'tagged_uint64' - | 'float32' | 'float64' + | 'float16' | 'bfloat16' | 'float32' | 'float64' | 'string' | 'bytes' | 'date' | 'timestamp' | 'duration' | 'decimal' | 'any' +scalar_encoding := 'varint' | 'fixed' | 'tagged' named_type := qualified_name qualified_name := IDENTIFIER ('.' IDENTIFIER)* // e.g., Parent.Child -list_type := 'list' '<' { 'optional' | 'ref' } field_type '>' +list_type := 'list' '<' { 'optional' | 'ref' | scalar_encoding } field_type '>' +array_type := 'array' '<' array_element_type '>' map_type := 'map' '<' field_type ',' field_type '>' type_options := '[' type_option (',' type_option)* ']' --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
