asadjan4611 opened a new pull request, #3329:
URL: https://github.com/apache/fory/pull/3329
## Why?
This PR implements bfloat16 (Brain Float 16) and bfloat16_array support for
Fory Python runtime and codegen, addressing issue #3289. This enables using
bfloat16 in FDL to reduce payload size while keeping a wide exponent range,
which is common in ML/AI workflows.
## What does this PR do?
This PR adds comprehensive bfloat16 support to Fory Python:
### Core Implementation
- **BFloat16 Type**: Cython implementation with IEEE 754 compliant
float32↔bfloat16 conversions (round-to-nearest, ties-to-even)
- **BFloat16Array**: Python-visible array type backed by `array.array('H')`
for packed contiguous storage
- **Serializers**: Both scalar (`BFloat16Serializer`) and array
(`BFloat16ArraySerializer`) serializers
- **Type Registration**: Registered with TypeId.BFLOAT16 (18) and
TypeId.BFLOAT16_ARRAY (54)
### Integration Points
- **Buffer Operations**: Added `write_bfloat16()` and `read_bfloat16()`
methods
- **Codegen Support**: Added bfloat16 to codegen type mapping
- **Row Format**: Added `bfloat16()` factory function (temporarily maps to
float16 until C++ row format supports it)
- **Type System**: Fully integrated into Fory type resolver
### Testing
- **11 comprehensive test cases** covering:
- Basic operations and conversions
- Special values (NaN, ±Inf, ±0)
- Serialization round-trips
- Array operations
- Integration with dataclasses, lists, and maps
- Type registration verification
### Code Quality
- Follows existing `float16` implementation patterns
- Matches project code standards and style
- All files include proper Apache 2.0 license headers
- No linter errors
## Related issues
- Fixes #3289
## Does this PR introduce any user-facing change?
- [x] Does this PR introduce any public API change?
- Yes: Adds `BFloat16`, `BFloat16Array` types and `bfloat16()` factory
function
- [ ] Does this PR introduce any binary protocol compatibility change?
- No: Uses existing TypeId.BFLOAT16 (18) already defined in protocol spec
## Implementation Details
### Wire Format
- Encodes bfloat16 as 2 bytes representing raw IEEE 754 bfloat16 bit pattern
- Little endian byte order (matches existing float32/float64 behavior)
- NaN/Inf/±0/subnormal values round-trip correctly at bit level
### Type System
- Type ID: 18 (BFLOAT16) - already defined in xlang serialization spec
- Array Type ID: 54 (BFLOAT16_ARRAY)
- Protocol compliant with existing xlang serialization format
### Performance
- Uses Cython for performance-critical conversion operations
- Zero-copy array operations using `array.array('H')`
- Follows same optimization patterns as existing float16 implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
