mihaibercea opened a new pull request, #3112:
URL: https://github.com/apache/jackrabbit-oak/pull/3112
## OAK-12394
### Problem
The `<lane>.LAST_INDEXED_TIME` metric exposed by `AsyncIndexUpdate` does not
reliably
reflect the true last-indexed time. Two independent sources of this
information diverge
in practice:
| Source | Backing store | Behaviour |
|--------|--------------|-----------|
| `IndexStatsMBean.getLastIndexedTime()` | Durable repository property
`:async/<lane>-LastIndexedTo` | Ground truth — survives restarts, always
current |
| `<lane>.LAST_INDEXED_TIME` metric (`CounterStats`) | In-memory Dropwizard
counter, updated in `ExecutionStats.doneOneCycle()` | Only updated on the
**success path**; resets to `0` on restart |
`doneOneCycle()` is reachable only via `AsyncIndexStats.done()` ←
`postAsyncRunStatsStatus()`. Any run that takes the failure path — e.g. a
lane stuck on
a lease / concurrent-update exception — calls `indexStats.failed()` instead
and **never
reaches `done()`**. As a result:
- A lane that fails on every cycle keeps the counter at whatever value it
held at the
last successful cycle on *this process instance*.
- A process restart resets the counter to `0`. If the lane continues to
fail, the
metric stays at `0` indefinitely — not a stale-but-real timestamp, but a
value
indistinguishable from "never indexed".
- Standby cluster members never run async indexing, so their counters are
permanently
`0` regardless of what the repository state says.
The metric therefore cannot be used to reliably determine when a lane last
indexed, which
is its entire purpose.
### Fix
Replace the `CounterStats` with a `GaugeStats<Long>` whose supplier is
evaluated on
every metric scrape:
- **Feature enabled (default):** reads the durable
`:async/<lane>-LastIndexedTo`
property from the repository and returns epoch millis (`0` if absent or
unparseable,
e.g. before the first index cycle has completed).
- **Feature disabled:** returns the legacy in-memory value (wall-clock of
the last
successful cycle), preserving the previous behaviour for a clean revert.
Reading from the shared `NodeStore` means the value is consistent across all
cluster
members (leader and standby), survives process restarts, and reflects the
actual
repository state even while the lane is failing. The metric name and units
(epoch millis)
are unchanged, so any tooling built on top of this metric continues to work;
only the
metric type changes from counter to gauge.
### Feature toggle
`FT_OAK_ASYNC_METRIC_DURABLE`, enabled by default (bug-fix convention per Oak
conventions). Registered on the OSGi Whiteboard via `Feature.newFeature()`
and
runtime-flippable. When no `Whiteboard` is provided (embedded/test use), the
durable
behaviour applies unconditionally.
### Changes
**`AsyncIndexUpdate`**
- New constant `FT_OAK_ASYNC_METRIC_DURABLE` and an optional `Whiteboard`
constructor
parameter; existing constructors delegate with `null` — no call-site
changes required.
- `ExecutionStats`: registers a `GaugeStats<Long>` instead of a
`CounterStats`;
adds `getLastIndexedTimeMetricValue()` (feature-gated dispatch),
`readLastIndexedToMillis()` (reads and parses the durable property), and
`close()`
(releases the feature toggle).
- `AsyncIndexUpdate.close()` now calls `indexStats.close()` to release the
feature.
**`AsyncIndexerService`**
- Passes the activation `Whiteboard` to the `AsyncIndexUpdate` constructor
for all
configured lanes and the async-reindex lane.
**`AsyncIndexUpdateTest`**
- New test `lastIndexedTimeGaugeReflectsDurableProperty()`: constructs a
fresh indexer
instance against a store that already has the `LastIndexedTo` property
written (from a
prior cycle), and asserts the gauge returns the persisted epoch millis
without any
cycle having run — directly testing the restart / in-memory-reset scenario.
### Testing
```bash
mvn test -pl oak-core -Dtest=AsyncIndexUpdateTest
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
