KKcorps commented on PR #20157:
URL: https://github.com/apache/pinot/pull/20157#issuecomment-6680275089
This comes from a model of upsert thread orders that StarTree runs on upsert
PRs.
**The model agrees that the reset this PR removes puts a replica with wrong
rows back into routing. It also shows that a replica left in ERROR can still
win the next commit, so its wrong rows reach the deep store anyway.**
I built the ERROR replica by hand, as a failed ONLINE load and as a
COMMITTING download that timed out, and a 73-minute search ran 1,177 searches
over 44 table shapes with the reset allowed and blocked. With the reset
blocked, this was the only bad order it found.
**The order the gate misses.** While N sits in ERROR, the server consumes
N+1 without N's keys, and nothing stops it from committing N+1. Every replica
that downloads N+1 then gets a partial upsert row merged without its base, or a
dedup repeat. A reset sent with the override does not fix that server either,
because it reloads N and leaves N+1 as it is.
```mermaid
sequenceDiagram
participant S as Server 1
participant C as Controller
participant D as Deep store
participant R as Server 2
S->>S: N fails to load and stays in ERROR
S->>S: N+1 consumes A, merges with no base row
C-->>S: no reset, so S stays out of routing
S->>C: wins the commit of N+1
S->>D: uploads N+1 with A = 1
R->>D: downloads N+1
Note over R: serves A = 1, the right value is 2
```
<details><summary>Model history</summary>
```
join_fresh_replica (the partition moves back to this server)
append A cmp=0 +1 (offset 0), N0 commits on another replica at offset 0
fail_load N0 (Helix ERROR)
acquire N1', consume A cmp=0 +1 (offset 1), merged with no base, A = 1
commit_local N1' at offset 1: build, upload, commit end
N1 in deep store holds A = 1, a correct build holds A = 2
```
</details>
Holding each consumer until the older segments of its partition load, as
#20106 does for these tables, stops both orders in the model. The model has one
replica and no broker, so it treats a server with a segment in ERROR as out of
routing.
<!-- upsert-model-check head=5abbf61be212b0e4b976649aceb6743a772c2b8c -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
