gortiz opened a new pull request, #20058:
URL: https://github.com/apache/pinot/pull/20058
## Summary
**What.** Pinot instances can now advertise *capabilities* in their Helix
instance config, and other components can read them. A capability is a named
feature with an integer revision, for example `{"MSE_MV_AGGREGATE": "1"}`.
**Why.** During a rolling upgrade, a new broker can send work to servers of
an older version. A new feature that old servers cannot run must stay off until
every server supports it. Pinot has no generic way to know this today.
Comparing Pinot versions does not work well: different distributions and
backport branches use their own version numbers.
**How.**
- `InstanceCapability` (in `pinot-common`) is the registry of capabilities.
Each one has a name, the instance type that advertises it, and a revision. This
PR adds the mechanism with an empty registry. The first capability comes in a
follow-up PR (the multi-stage engine MV aggregate).
- At startup, each instance writes its capabilities to the new map field
`capabilities` of its own instance config, with
`HelixHelper.updateCapabilities`. Today only servers call it.
- `InstanceCapabilityTracker` listens to instance config changes and gives
the minimum revision of a capability over all the instances of one type.
`MultiClusterInstanceCapabilityTracker` gives the minimum over several
clusters, for a broker that federates remote clusters.
## Contract
- A revision only increases. Revision `n` includes all the function of the
revisions before it.
- A revision is not a Pinot version. The mechanism compares no version
numbers.
- A missing map, a missing key, or a value that is not a valid integer
counts as revision 0: not supported. Every failure is fail-safe.
- The names are persisted in ZooKeeper. A capability is never renamed or
removed.
## Rollback guard
An old build does not know the `capabilities` field, so it does not remove
it after a rollback. The map has a reserved entry `pinotVersion` with the
version of the build that wrote it. Every build since 1.4.0 writes the
`pinotVersion` simple field at startup. If the two values are not equal, the
tracker ignores the capabilities of that instance. This is a string equality
between two fields of one instance config, not a comparison of versions.
One gap remains. The controller writes the full instance config record, for
example on `PUT /instances/{name}`. If this write races with an instance
restart, it can restore the `pinotVersion` field and the `capabilities` map of
the previous build. The `pinotVersion` field has the same problem today. A
follow-up can make the controller keep the fields that the instance owns.
## ZooKeeper cost
Writes:
- Each instance writes only its own instance config znode,
`/<cluster>/CONFIGS/PARTICIPANT/<instanceId>`. There is no shared znode, so
instances never write the same node.
- The write is part of the instance config update that `BaseServerStarter`
already does at startup. It adds about 60 bytes. On an upgrade, the
`pinotVersion` field changes anyway, so the capabilities add no extra write. A
restart with the same version writes nothing.
- The controller writes the full instance config record (for example on `PUT
/instances/{name}`). If such a write races with a server restart, the map can
disappear. The readers then see revision 0 until the next restart of that
server. This is fail-safe.
Reads (only when a component creates a tracker; this PR creates none):
- One `InstanceConfigChangeListener` for each tracked cluster.
- On the first notification, and when an instance is added or removed, the
tracker reads the config of each instance of the tracked type, about 2
ZooKeeper operations each.
- When one instance config changes, it reads that config only.
The broker routing manager already reads all the instance configs on every
instance config change. A tracker on a broker adds traffic of the same order,
and a second watch on the same path. A possible follow-up removes this cost:
the routing manager can feed the tracker from the records that it already reads.
## Thread safety
Helix callback threads update the tracker under a lock. Query threads read
an immutable snapshot through a volatile field, without a lock.
## Testing
- `InstanceCapabilityTrackerTest`: revision 0 before the first notification,
missing and malformed values, the minimum over mixed revisions, config changes,
removed instances, failed reads, and the rollback guard with matching,
different and missing version stamps.
- `MultiClusterInstanceCapabilityTrackerTest`: the minimum over clusters, a
cluster that does not connect, a failed registration, and `close()`.
- `HelixHelperTest`: `updateCapabilities` writes the capabilities and the
version stamp, and reports a change only when the map changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
