FMX opened a new pull request, #3825:
URL: https://github.com/apache/celeborn/pull/3825
### What changes were proposed in this pull request?
This PR reduces the memory footprint of `PartitionLocation` metadata while
preserving the existing wire protocol and public getter behavior.
The main changes are:
- Introduce an immutable, weakly interned `WorkerEndpoint` to share worker
host and port information across partition locations.
- Move the cached `host:pushPort` and `host:fetchPort` strings into the
shared endpoint.
- Lazily initialize `StorageInfo` and `RoaringBitmap` using thread-safe
initialization.
- Keep `getMapIdBitMap()` non-null for compatibility, and add a
non-materializing accessor for serialization, HTTP API, and read-side
inspection.
- Avoid allocating empty bitmaps when locations are created by
`LifecycleManager` and `SlotsAllocator`.
- Preserve the current `storageTypes.head` selection behavior in
`LifecycleManager`.
- Keep string keys in `WorkerPartitionLocationInfo` to avoid parsing and
boxing overhead on the Worker hot path.
- Add allocation-free helpers for converting partition location IDs into
packed `long` values.
- Avoid materializing optional metadata from logging, protobuf
serialization, HTTP rendering, and range-read paths.
- Re-intern `WorkerEndpoint` instances after same-version Java
deserialization.
- Fix peer bitmap serialization so that the peer protobuf contains the
peer's bitmap instead of the primary location's bitmap.
- Make range reads fall back to the peer bitmap and fail open when bitmap
metadata is unavailable.
- Add JOL-based manual memory benchmarks and focused unit tests.
No protobuf schema is changed by this PR.
### Why are the changes needed?
A shuffle may contain millions of `PartitionLocation` instances.
Previously, every instance independently retained:
- Worker host and port fields.
- Cached host/port strings.
- An empty `StorageInfo`.
- An empty `RoaringBitmap`.
Most of this metadata is identical across locations or unused for the
majority of their lifetime. The eager allocations therefore contribute
significant heap usage and GC pressure on the Master, Worker, and client.
The JOL benchmark includes both the live location graph and the weak
interner overhead:
| Scenario | Before | After, including interner | Reduction |
|---|---:|---:|---:|
| 10,000 primary/replica pairs across 2,000 endpoints | 5,672,304 bytes |
1,187,928 bytes | 79.06% |
| 1,000,000 primary/replica pairs across 2,000 endpoints | 567,200,304
bytes | 88,307,928 bytes | 84.43% |
The additional serialization and range-read fixes are needed to ensure
that lazy bitmap allocation does not introduce data loss, incorrect peer
metadata, or compatibility regressions.
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed
automatically. -->
- [x] Yes
It fixes peer bitmap serialization and ensures that missing bitmap
metadata does not cause an NPE or silently exclude a partition location during
range reads.
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [ ] Yes
There are no configuration, wire-protocol, or intended behavioral changes
for users.
### How was this patch tested?
The following verification was completed:
- Ran the `common`, `client`, and `master` module regression suites:
- 366 JUnit tests, with 1 ignored.
- 325 ScalaTest tests.
- 691 tests in total, with no failures or errors.
- Ran the final `PartitionLocationSuiteJ`:
- 20 tests passed.
- Covers lazy initialization, endpoint interning, endpoint setters, ID
parsing, peer references, compatibility getters, and same-version Java
serialization.
- Ran Worker partition and storage regression suites:
- 27 Java tests passed.
- 9 Scala tests passed.
- Ran the Spark 3.5 `LifecycleManagerReserveSlotsSuite` integration test:
- 1 test passed.
- All 11 reactor modules completed successfully.
- Compiled the affected modules through SBT:
- `celeborn-common/Test/compile`
- `celeborn-service/Compile/compile`
- Ran `spotless:check` and `git diff --check`.
- Manually ran `PartitionLocationMemorySuiteJ` with JOL for both the
10,000-pair and 1,000,000-pair scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
