vustef opened a new pull request, #575:
URL: https://github.com/apache/arrow-julia/pull/575
Implements support for Arrow's RunEndEncoded (REE) layout as specified in
the Arrow format specification. REE is
a run-length encoding variant that efficiently stores arrays with repeated
values using two child arrays:
run_ends (indices where runs terminate) and values (the actual run values).
Implementation
- Core type: Added Arrow.RunEndEncoded{T,R,A} struct with O(log n) binary
search indexing
- Type system: Registered RunEndEncodedKind in ArrowTypes module
- Serialization: Implemented arrowvector() and makenodesbuffers!() for
writing REE arrays
- Deserialization: Added build() function and juliaeltype() for reading REE
arrays from Arrow IPC format
- Interoperability: Validated against PyArrow-generated test files (included
as fixtures)
Testing
- Cross-language validation using PyArrow 20.0.0-generated test files
- Round-trip tests for various data types (integers, floats, strings,
booleans, with nulls)
- Edge cases: single runs, alternating values, long runs
Files Changed
- src/ArrowTypes/src/ArrowTypes.jl - Added RunEndEncodedKind
- src/arraytypes/runendencoded.jl - New 260-line implementation
- src/arraytypes/arraytypes.jl - Added include
- src/eltypes.jl - Added type mapping functions
- src/table.jl - Added deserialization logic
- test/generate_ree_test_data.py - Test data generator (reference)
- test/test_ree_{simple,data}.arrow - PyArrow test fixtures
- test/test_runendencoded.jl - Comprehensive test suite
Closes #476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
