allthingssecurity opened a new pull request, #30252: URL: https://github.com/apache/camel/pull/30252
# Description [CAMEL-27066](https://issues.apache.org/jira/browse/CAMEL-27066) Follow-up of CAMEL-26969 (the lock renewer keeps a lock after a transient renewal error). Found by the DST world in #30130 (CAMEL-26956) once the CAMEL-26969 fix was on main: with `-Ddst.servicebus.strictRenew=true` it still failed 11 of 2000 seeds. The JIRA suggested a short bounded retry of the renewal; this PR instead runs the lock renewer more often, which retries at the next run without a timer and without more calls to Service Bus (see "Why this" below). The world still excuses two renewal faults in a row, see the end. The lock renewer of `ServiceBusConsumer` (peek-lock, `maxAutoLockRenewDuration` > 0) ran every 10 seconds or a second later (the period task scheduler ticks once per second) and renewed a lock that expired within 20 seconds. Since CAMEL-26969 a transient renewal error keeps the entry, so the next run tries again. But depending on where the runs fell, the first renewal of a lock could come with only 9 to 11 seconds left, for any lock duration, and when it failed the next run could come after the lock expired. Seed 113 of the world, 30 second lock: - t=37 s: the lock is renewed until 68.9 s; - t=48 s: nothing to do, the lock expires later than now + 20 s; - t=59 s: the renewal fails with a transient error; - t=70 s: the next run renews, but the lock expired at 68.9 s. Service Bus delivers the message again while the first exchange is still running, and its `complete()` fails with a lost lock. The same can happen with longer locks: a 42 second lock with 11 second gaps, or a 60 second lock with four 10 second gaps in a row. Change (`ServiceBusConsumer`): the renewer runs every 5 seconds (`LOCK_RENEW_INTERVAL_SECONDS`) and still renews a lock that expires within 20 seconds, now a constant of its own (`LOCK_RENEW_BEFORE_EXPIRY_SECONDS`) instead of two intervals. When the runs come on time, the first renewal comes with more than 14 seconds left, a retry after a transient error with more than 8 seconds left, and a second retry with more than 2 seconds left. These margins hold only when the runs come on time: the renewer runs on the one timer thread that all period tasks of the CamelContext share, so a slow task or a GC pause makes a gap longer, and the call to Service Bus and any clock skew between client and server (`lockedUntil` is server time) take from them too. Why this and not renewing earlier or retrying in the error callback: - renewing when the lock expires within three intervals (30 seconds, the first version of this change) also survives one error, but renews more: a 60 second lock at every third run instead of every fourth (about every 33 seconds instead of 44), locks of about 31 to 41 seconds at every run instead of every other run, and every message still in flight on a 30 second lock, even one about to complete. In the world it still failed 9 of 2000 strict seeds (two faults in a row), against 1 of 2000 with this change; - with the shorter interval the renewal calls stay as they were when renewals succeed: a 60 second lock (the Service Bus default) about every 41 to 46 seconds (usually 42) (before: about every 44, up to 50), a 30 second lock about every 11 to 16 seconds (usually 12) (before: 11 to 20), a 5 minute lock about every 281 to 286 seconds (before: 286 to 290), and a message with more than 20 seconds of lock left is not renewed, as before; - no new thread, timer or Reactor time operator (`Mono.delay`, `Retry.backoff` run on Reactor's parallel scheduler); the renewal is still one call per lock per run, from the same task; - the cost: the renewer task runs twice as often (a scan of the exchanges in flight, with no call to Service Bus unless a lock is due), and a failed renewal is retried 5 or 6 seconds later, sooner than the back-off of about 10 seconds that the Service Bus documentation describes after a ServerBusy error (the CAMEL-26969 PR kept the 10 second wait for that reason). A retry in the error callback, right away, would be sooner still. - for comparison, the processor client of the SDK renews its own locks 10 seconds before they expire (half the remaining time for a lock with less than 20 seconds left) and stops at the first failed renewal (`LockRenewalOperation` in azure-messaging-servicebus 7.17.16), so it has no retry at all. What users can notice: nothing in the number of renewal calls when renewals succeed (a failing renewal is now tried again every 5 or 6 seconds, so up to 3 or 4 attempts in the last 20 seconds of a lock instead of 1 or 2); a transient renewal error no longer loses a lock (when the renewer runs on time), and two in a row usually do not either. No option or default changes, so no upgrade guide entry. What is not changed: - three renewal faults in a row, or two when the first renewal came late, can still lose a lock; - a lock shorter than 20 seconds is renewed at every run; a failed renewal of it is retried in time only for a lock longer than about 13 seconds (two gaps plus the call), and a lock shorter than about 6 seconds can never be held by the Camel renewer (the Service Bus minimum lock duration is 5 seconds). Before, these limits were about 23 and 11 seconds. Tests, in `ServiceBusConsumerTest` (Mockito, no Service Bus, no waits): - `lockRenewerRetriesATransientErrorBeforeTheLockExpires` (was `lockRenewerRenewsAgainAfterATransientError`): a lock that expires 2 seconds after two of the longest gaps between runs (the period plus a second, read from the scheduled task) must be renewed now; the renewal fails with a transient `ServiceBusException` (SERVER_BUSY), and the next run renews again. Without the change it fails: `Wanted but not invoked: renewalClient.renewMessageLock(message)` (with 10 second runs that lock has 24 seconds left and is left alone). - `lockRenewerDoesNotRenewALockThatOutlastsTheNextRuns` now uses a lock with 22 seconds left (instead of a minute), so it pins the 20 second threshold from above. - With the change the camel-azure-servicebus unit tests pass: 81 run, 0 failures. With the DST world of #30130, this change merged locally: - `-Ddst.servicebus.strictRenew=true` fails 1 of 2000 seeds instead of 11 (seed 1312: three renewal faults in a row, at 30, 36 and 42 s, the lock expires at 44.9 s); - the world then drops `dst.servicebus.strictRenew` and excuses a lost lock only after two renewal faults in a row; one fault is no excuse any more. That world passes 1000 of 1000 in the module run (1081 tests, 0 failures) and 2000 of 2000. That world change is test-only and goes in with whichever of #30130 and this PR is merged second. The ITs need a Service Bus namespace and were not run locally (CI runs them). Backport: camel-4.18.x has the same lock renewer, together with CAMEL-26969. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-27066](https://issues.apache.org/jira/browse/CAMEL-27066). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-azure-servicebus and the core modules with the formatter and import-sort plugins and ran the camel-azure-servicebus unit tests. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
