fresh-borzoni opened a new pull request, #156:
URL: https://github.com/apache/fluss-rust/pull/156
### Purpose
Linked issue: close #111
This PR implements the complete KV (Key-Value) record batch format for
Fluss-Rust, enabling serialization and deserialization of KV records compatible
with the Java implementation.
### Brief change log
**Core Implementation:**
- **KvRecord** Implements immutable key-value record format with
variable-length encoding for keys and optional values (tombstones for deletions)
- **KvRecordBatch**: Provides read-only access to serialized KV record
batches with CRC32C checksum validation, schema versioning, and iterator support
- **KvRecordBatchBuilder** : Implements batch building with configurable
write limits, exactly-once semantics (writer ID + sequence), and direct
CompactedRow integration
- **Varint utilities** : Complete variable-length integer
encoding/decoding matching Protocol Buffers format, with optimized variants for
`Write`, `BufMut`, and raw slices
**Record Format:**
KvRecord:
[Length:I32][KeyLength:VarInt][Key:bytes][ValueLength:VarInt?][Value:bytes?]
KvBatch: [Length:I32][Magic:I8][CRC:U32][SchemaId:I16][Attributes:I8]
[WriterId:I64][BatchSequence:I32][RecordCount:I32][Records...]
**API Updates:**
- Exposed `write_unsigned_varint_to_slice()` for `CompactedRowWriter`
integration
- Updated `CompactedRowWriter::finish()` to return slice instead of copying
### API and Format
```rust
pub struct KvRecord { ... }
pub struct KvRecordBatch { ... }
pub struct KvRecordBatchBuilder { ... }
pub mod varint { ... }
```
Storage Format:
- Introduces KV record batch format (100% binary-compatible with Java)
- Uses unsigned varint encoding (NOT zigzag) for space efficiency.
- Format version tracked via magic byte (CURRENT_KV_MAGIC_VALUE = 0)
Breaking Changes: None (new module)
Compatibility: Rust implementation produces byte-identical output to Java
implementation for all field types and edge cases.
Documentation
New Feature: Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
