sergehuber opened a new pull request, #855: URL: https://github.com/apache/unomi/pull/855
A scheduler lock's renewal cadence follows its owner's configured lock timeout (lockTimeout/3), but expiry was judged against the observer's timeout. A node configured shorter than a peer's renewal cadence saw every renewal gap as a dead lock: it marked the live execution CRASHED, cleared the lock, and the next peer tick re-dispatched the task while the original execution was still running. Reproduced deterministically (LOCK-DIAG traces show a 1s-timeout observer crash-marking a lock 2.9s into its owner's 3.3s renewal interval, and a second worker dispatching while the first still runs). Configuration drift or a rolling upgrade produces the same double execution in production, and the divergently-configured setUp() scheduler in the unit tests is what made SchedulerServiceImplTest.testConcurrentLockAcquisition flake on loaded CI runners. Locks now record the lease their owner granted itself (ScheduledTask.lockLeaseMillis, stamped on acquire and every renewal, cleared on release), and isLockExpired() judges against that lease. Documents without a lease - written before this change, or with a corrupt negative value - fall back to the observer's timeout, the exact pre-change behaviour, so same-config clusters and existing locks are bit-for-bit unaffected. Both ES and OpenSearch mappings gain the field. ScheduledTask now tolerates unknown JSON properties so an older node keeps reading task documents written by a newer one during a rolling upgrade, and startLockRenewal() warns on the one configuration (lockTimeout at or below the renewal floor) where a node cannot keep its own lease alive. The steal is pinned end to end by a mutation-validated regression test (fails "expected: <RUNNING> but was: <CRASHED>" with the fix reverted); the recovery direction - a patient survivor recovers a dead owner's task as soon as the OWNER's lease expires - has its own end-to-end test; unit coverage exercises lease stamping on all three acquire paths, renewal re-stamping after a runtime timeout change, clearing on release, both override directions, legacy/corrupt fallbacks, boundary equality, overflow, and the persistence format through both real read paths. The second commit deflakes the scheduler suites themselves: the setUp() scheduler no longer runs with a divergent 1s timeout as an uninvited extra cluster node (multi-node tests stop it, per the existing testNodeFailure pattern); testClusteringSupport no longer demands one runOnAllNodes task execute on all three nodes - not a property the implementation promises, since all nodes share the task's single schedule and each period has one phase-dependent winner - and the regression it actually protected (non-executors poll and run runOnAllNodes tasks) is pinned deterministically in a new single-node test; exact counts on still-firing periodic tasks now cancel-and-quiesce or assert lower bounds; sleep-then-assert waits became bounded polls, Mockito timeout() verifies, or test-released latches, while deliberate quiet windows for negative assertions keep their sleeps and say why; the dead configureDebugLogging() (slf4j-simple properties under a logback binding) is replaced by documenting -DTEST_LOG_LEVEL=DEBUG, so the next failure arrives with LOCK-DIAG traces attached. Validated with seven consecutive green runs of the four scheduler suites under ~2.5x CPU oversubscription - the protocol that reproduced both original CI failures on demand. Note for a follow-up ticket: runOnAllNodes as implemented means "any node, including non-executors, may run the task", not "each node runs it every period" as the name suggests. Jira: https://issues.apache.org/jira/browse/UNOMI-979 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
