gortiz opened a new issue, #20062:
URL: https://github.com/apache/pinot/issues/20062
## Summary
`PinotHelixResourceManager.updateInstance` (for example on `PUT
/instances/{name}`) reads the instance config, changes it, and writes the full
record back with `HelixDataAccessor.setProperty`. The write has no version
check. If an instance restarts between the read and the write, the controller
overwrites the fields that the instance wrote at startup.
## Affected fields
- The `pinotVersion` simple field. Every instance writes it at startup. A
lost update keeps the version of the previous build in ZooKeeper.
- The `capabilities` map field, from
[apache/pinot#20058](https://github.com/apache/pinot/pull/20058). A lost update
restores the capabilities of the previous build, or removes the map.
For the capabilities, a removed map is fail-safe: readers see revision 0
until the next restart of the instance. A restored map from the previous build
is also caught in most cases, because the rollback guard compares the map's
version stamp with `pinotVersion`. But the race can restore both fields
together, and then the guard does not see the difference.
## Sequence
```
controller server (restart, new build)
---------- ---------------------------
read instance config (old build)
write pinotVersion + capabilities (new
build)
write full record (old build)
```
## Possible fixes
- Write with a version check: read the `ZNRecord` version and use a
conditional write, or use `HelixDataAccessor.updateProperty` with a
`DataUpdater` that applies only the fields of the request.
- Or keep the fields that the instance owns (`pinotVersion`, `capabilities`)
from the live record, the same way `updateInstance` already keeps
`queriesDisabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
