KKcorps opened a new pull request, #19505:
URL: https://github.com/apache/pinot/pull/19505
When realtime metadata removal throws, the consumer semaphore is still
released. A successor can
then consume against metadata left by the failed offload. The segment's
one-shot offload latch also
makes a retry return without retrying metadata removal.
This builds on merged #19444 and records that failure before releasing the
semaphore. Subsequent
consumer and query admission fail explicitly for the affected table on that
server.
### Behavior
- Keep a sticky table-level failure marker containing only the failed
segment's name.
- Check it before loading a new consuming segment and before waiting for
consumer admission.
Recheck after acquiring the semaphore to cover a waiter that started
before the failure; return
the acquired permit on rejection.
- Reject query segment acquisition before taking references. Return a
`QUERY_EXECUTION` error block
from the server executor, including for callers used by hybrid multi-stage
queries.
- If a logical-table query acquired references from earlier physical tables,
release them when a
later table rejects acquisition.
- Preserve the failure across registration and repeated offload callbacks. A
no-op retry cannot
make the same table manager healthy again.
```mermaid
sequenceDiagram
participant O as Offloading segment
participant T as Realtime table manager
participant S as Consumer semaphore
participant N as Successor
O->>O: Metadata removal throws
O->>T: Record failed segment name
O->>S: Release permit in finally
O-->>O: Propagate original failure
S-->>N: Waiting successor acquires permit
N->>T: Recheck metadata health
T-->>N: Reject admission
N->>S: Return acquired permit
```
### Scope and recovery
This intentionally trades availability for correctness after a reported
offload failure: the
failure boundary covers the whole realtime table on that server, including
other partitions.
Queries and consumers that already passed admission are not cancelled. The
serial-consumption case
is the target; this does not establish safe recovery for overlapping
consumers.
The marker is in memory and has no reset API. A new table manager starts
with an empty marker, so
safe recovery still requires rebuilding and validating metadata. Simply
reopening potentially
inconsistent persistent metadata is not a recovery guarantee. This PR does
not add automatic
reconstruction, retry failed metadata work, or change errors that metadata
implementations currently
swallow. Broader rollback and recovery work remains separate.
There are no row-loop changes, new per-key fields, segment pins, or retained
exception objects.
The healthy query path adds one volatile read per segment-list acquisition;
consumer checks run at
lifecycle boundaries. No throughput or latency benchmark was run.
### Validation
**67 tests passed, zero failures/errors/skips**, using JDK 25. Coverage
includes the injected offload
failure, a concurrent waiter, repeated offload/registration, acquisition for
full and partial upsert
under `NONE`/`SYNC`/`SNAPSHOT`, the server error block, logical-table
reference cleanup, and existing
consumer/query regressions. These are lifecycle and query-path tests, not
full data-level upsert
certification.
```sh
./mvnw -pl pinot-core -am \
-Dtest=RealtimeMetadataFailureTest,RealtimeSegmentDataManagerTest,ConsumerCoordinatorTest,RealtimeTableDataManagerTest,SingleTableExecutionInfoTest,LogicalTableExecutionInfoTest,QueryExecutorTest
\
-Dsurefire.failIfNoSpecifiedTests=false \
-Dspotless.check.skip=true -Dcheckstyle.skip=true -Dlicense.skip=true test
./mvnw spotless:apply -pl pinot-core
./mvnw license:format -pl pinot-core
./mvnw checkstyle:check -pl pinot-core
./mvnw license:check -pl pinot-core
git diff --check
```
All four module-scoped pre-commit checks and the diff whitespace check
passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
