KKcorps opened a new pull request, #19444:
URL: https://github.com/apache/pinot/pull/19444
## Summary
Fix the realtime offload handoff so the stream consumer is closed first,
upsert/dedup metadata is removed while the partition semaphore is still held,
and only then the semaphore is released.
This prevents the next consuming segment from taking its snapshot or
replaying against primary-key state that still points to the mutable segment
being offloaded. For partial upsert in `PROTECTED` mode, that stale state can
merge an update twice.
```mermaid
flowchart LR
A[stop and close consumer] --> B[remove segment metadata / revert]
B --> C[release partition semaphore]
C --> D[next segment snapshots and consumes]
```
Issue: [DATA-3301](https://startree.atlassian.net/browse/DATA-3301).
Supersedes #19433; unlike that version, this keeps stream-consumer close before
segment offload.
## Behavior and scope
- Splits consumer close from semaphore release.
- Keeps early release unchanged for `ALLOW_DURING_BUILD_ONLY` and
`ALLOW_ALWAYS` during build/download.
- Releases in `finally` if metadata removal fails, preserving partition
liveness and current failure semantics.
- Moves metric cleanup after release so it is outside the protected handoff
window.
The only performance impact is on offload handoff: when the semaphore has
not already been released, successor startup waits for metadata removal. There
is no steady-state ingestion cost.
## Tests
- `./mvnw -pl pinot-core -Dtest=RealtimeSegmentDataManagerTest test`
- Focused offload ordering, early-release idempotency, and failure-path tests
- `./mvnw spotless:apply -pl pinot-core`
- `./mvnw checkstyle:check -pl pinot-core`
- `./mvnw license:format -pl pinot-core`
- `./mvnw license:check -pl pinot-core`
- `./mvnw test-compile -pl pinot-core -Dmaven.compiler.showDeprecation=true
-Dmaven.compiler.showWarnings=true`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
