zeroshade opened a new pull request, #701:
URL: https://github.com/apache/arrow-go/pull/701
### Rationale for this change
Legacy Go map-based memo table implementations exist alongside newer
xxh3-based implementations, but the performance advantages of xxh3 (2-3x faster
for Float types, 75-89% fewer allocations for all types) are not clearly
documented or communicated to users.
**Current situation:**
- Production code uses xxh3-based dictionary implementations
(`NewInt32Dictionary()`, etc.)
- Legacy Go map-based constructors (`NewInt32MemoTable()`, etc.) still exist
without deprecation
- No clear guidance on which implementation to use
- Performance characteristics not documented
**Performance evidence:**
- **Float64:** xxh3 is 1.18-1.64x faster than Go maps
- **Float32:** xxh3 is 1.26-1.59x faster than Go maps
- **Int types:** xxh3 has 75-89% fewer allocations (critical for GC pressure)
- **All types:** Consistent 2-5 allocations vs 9-46 for Go maps
**Need for change:**
- Prevent users from accidentally using slower legacy implementations
- Document performance characteristics for informed decision-making
- Establish clear deprecation path for future cleanup
- Expand benchmark coverage to validate xxh3 advantages
### What changes are included in this PR?
Added deprecation notices and expanded benchmark functions
**Deprecation notice format:**
```go
// Deprecated: Use NewInt32Dictionary instead. This implementation uses Go's
// built-in map and has 75-89% more allocations than xxh3-based dictionary,
// increasing GC pressure. For Float types, xxh3 is also 1.2-2x faster.
// Will be removed in a future release.
func NewInt32MemoTable() *Int32MemoTable { ... }
```
### Are these changes tested?
Yes, extensively tested and benchmarked:
New benchmark validation (6 benchmarks, 28 total):
**Float64 performance (xxh3 vs Go map):**
```
100 unique: 1.285 ms (map) → 1.082 ms (xxh3) = 1.18x faster, 78% fewer
allocs
1,000 unique: 1.539 ms (map) → 939.8 µs (xxh3) = 1.64x faster, 80% fewer
allocs
5,000 unique: 1.992 ms (map) → 1.250 ms (xxh3) = 1.59x faster, 89% fewer
allocs
```
**Float32 performance (xxh3 vs Go map):**
```
100 unique: 1.264 ms (map) → 998.3 µs (xxh3) = 1.26x faster, 78% fewer
allocs
1,000 unique: 1.544 ms (map) → 1.034 ms (xxh3) = 1.49x faster, 80% fewer
allocs
5,000 unique: 2.044 ms (map) → 1.282 ms (xxh3) = 1.59x faster, 89% fewer
allocs
```
**Int64/Int32 allocation comparison:**
```
100 unique: 9 allocs (map) → 2 allocs (xxh3) = 78% fewer
1,000 unique: 20 allocs (map) → 4 allocs (xxh3) = 80% fewer
5,000 unique: 46 allocs (map) → 5 allocs (xxh3) = 89% fewer
```
**Edge case validation:**
- NaN values: Consistent hashing across all NaN representations ✓
- Infinity values: +Inf and -Inf handled correctly ✓
- Null values: Proper null tracking for all types ✓
- High cardinality: Tested up to 1M unique values ✓
**Benchmark coverage expanded:**
- Original: 22 benchmarks
- New: 28 benchmarks (+6, 27% increase)
- All data types covered (Int32, Int64, Float32, Float64, Binary)
### Are there any user-facing changes?
only deprecation notices and performance guidance:
**Benefits of migrating to xxh3-based implementations:**
**No immediate action required:**
- Deprecated functions still work (no breaking changes)
- Legacy implementations will be removed in future release
- Migration is straightforward (simple constructor swap)
- No behavior changes, only performance improvements
**Performance guidance:**
- **Always use xxh3** for Float32/Float64 (clear speed + allocation wins)
- **Use xxh3** for Int32/Int64 (allocation benefits outweigh slight speed
trade-off)
- **Use xxh3** for high cardinality data (>5,000 unique values)
- **Use xxh3** for long-running applications (GC benefits compound over time)
**Documentation improvements:**
- Clear deprecation notices in code
- Performance characteristics documented in comments
- Migration path clearly specified
- Benchmark results validate recommendations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
