allthingssecurity opened a new pull request, #30252:
URL: https://github.com/apache/camel/pull/30252

   # Description
   
   [CAMEL-27066](https://issues.apache.org/jira/browse/CAMEL-27066)
   
   Follow-up of CAMEL-26969 (the lock renewer keeps a lock after a transient 
renewal error). Found by the DST world in #30130 (CAMEL-26956) once the 
CAMEL-26969 fix was on main: with `-Ddst.servicebus.strictRenew=true` it still 
failed 11 of 2000 seeds. The JIRA suggested a short bounded retry of the 
renewal; this PR instead runs the lock renewer more often, which retries at the 
next run without a timer and without more calls to Service Bus (see "Why this" 
below). The world still excuses two renewal faults in a row, see the end.
   
   The lock renewer of `ServiceBusConsumer` (peek-lock, 
`maxAutoLockRenewDuration` > 0) ran every 10 seconds or a second later (the 
period task scheduler ticks once per second) and renewed a lock that expired 
within 20 seconds. Since CAMEL-26969 a transient renewal error keeps the entry, 
so the next run tries again. But depending on where the runs fell, the first 
renewal of a lock could come with only 9 to 11 seconds left, for any lock 
duration, and when it failed the next run could come after the lock expired. 
Seed 113 of the world, 30 second lock:
   - t=37 s: the lock is renewed until 68.9 s;
   - t=48 s: nothing to do, the lock expires later than now + 20 s;
   - t=59 s: the renewal fails with a transient error;
   - t=70 s: the next run renews, but the lock expired at 68.9 s. Service Bus 
delivers the message again while the first exchange is still running, and its 
`complete()` fails with a lost lock.
   
   The same can happen with longer locks: a 42 second lock with 11 second gaps, 
or a 60 second lock with four 10 second gaps in a row.
   
   Change (`ServiceBusConsumer`): the renewer runs every 5 seconds 
(`LOCK_RENEW_INTERVAL_SECONDS`) and still renews a lock that expires within 20 
seconds, now a constant of its own (`LOCK_RENEW_BEFORE_EXPIRY_SECONDS`) instead 
of two intervals. When the runs come on time, the first renewal comes with more 
than 14 seconds left, a retry after a transient error with more than 8 seconds 
left, and a second retry with more than 2 seconds left. These margins hold only 
when the runs come on time: the renewer runs on the one timer thread that all 
period tasks of the CamelContext share, so a slow task or a GC pause makes a 
gap longer, and the call to Service Bus and any clock skew between client and 
server (`lockedUntil` is server time) take from them too.
   
   Why this and not renewing earlier or retrying in the error callback:
   - renewing when the lock expires within three intervals (30 seconds, the 
first version of this change) also survives one error, but renews more: a 60 
second lock at every third run instead of every fourth (about every 33 seconds 
instead of 44), locks of about 31 to 41 seconds at every run instead of every 
other run, and every message still in flight on a 30 second lock, even one 
about to complete. In the world it still failed 9 of 2000 strict seeds (two 
faults in a row), against 1 of 2000 with this change;
   - with the shorter interval the renewal calls stay as they were when 
renewals succeed: a 60 second lock (the Service Bus default) about every 41 to 
46 seconds (usually 42) (before: about every 44, up to 50), a 30 second lock 
about every 11 to 16 seconds (usually 12) (before: 11 to 20), a 5 minute lock 
about every 281 to 286 seconds (before: 286 to 290), and a message with more 
than 20 seconds of lock left is not renewed, as before;
   - no new thread, timer or Reactor time operator (`Mono.delay`, 
`Retry.backoff` run on Reactor's parallel scheduler); the renewal is still one 
call per lock per run, from the same task;
   - the cost: the renewer task runs twice as often (a scan of the exchanges in 
flight, with no call to Service Bus unless a lock is due), and a failed renewal 
is retried 5 or 6 seconds later, sooner than the back-off of about 10 seconds 
that the Service Bus documentation describes after a ServerBusy error (the 
CAMEL-26969 PR kept the 10 second wait for that reason). A retry in the error 
callback, right away, would be sooner still.
   - for comparison, the processor client of the SDK renews its own locks 10 
seconds before they expire (half the remaining time for a lock with less than 
20 seconds left) and stops at the first failed renewal (`LockRenewalOperation` 
in azure-messaging-servicebus 7.17.16), so it has no retry at all.
   
   What users can notice: nothing in the number of renewal calls when renewals 
succeed (a failing renewal is now tried again every 5 or 6 seconds, so up to 3 
or 4 attempts in the last 20 seconds of a lock instead of 1 or 2); a transient 
renewal error no longer loses a lock (when the renewer runs on time), and two 
in a row usually do not either. No option or default changes, so no upgrade 
guide entry.
   
   What is not changed:
   - three renewal faults in a row, or two when the first renewal came late, 
can still lose a lock;
   - a lock shorter than 20 seconds is renewed at every run; a failed renewal 
of it is retried in time only for a lock longer than about 13 seconds (two gaps 
plus the call), and a lock shorter than about 6 seconds can never be held by 
the Camel renewer (the Service Bus minimum lock duration is 5 seconds). Before, 
these limits were about 23 and 11 seconds.
   
   Tests, in `ServiceBusConsumerTest` (Mockito, no Service Bus, no waits):
   - `lockRenewerRetriesATransientErrorBeforeTheLockExpires` (was 
`lockRenewerRenewsAgainAfterATransientError`): a lock that expires 2 seconds 
after two of the longest gaps between runs (the period plus a second, read from 
the scheduled task) must be renewed now; the renewal fails with a transient 
`ServiceBusException` (SERVER_BUSY), and the next run renews again. Without the 
change it fails: `Wanted but not invoked: 
renewalClient.renewMessageLock(message)` (with 10 second runs that lock has 24 
seconds left and is left alone).
   - `lockRenewerDoesNotRenewALockThatOutlastsTheNextRuns` now uses a lock with 
22 seconds left (instead of a minute), so it pins the 20 second threshold from 
above.
   - With the change the camel-azure-servicebus unit tests pass: 81 run, 0 
failures.
   
   With the DST world of #30130, this change merged locally:
   - `-Ddst.servicebus.strictRenew=true` fails 1 of 2000 seeds instead of 11 
(seed 1312: three renewal faults in a row, at 30, 36 and 42 s, the lock expires 
at 44.9 s);
   - the world then drops `dst.servicebus.strictRenew` and excuses a lost lock 
only after two renewal faults in a row; one fault is no excuse any more. That 
world passes 1000 of 1000 in the module run (1081 tests, 0 failures) and 2000 
of 2000. That world change is test-only and goes in with whichever of #30130 
and this PR is merged second.
   
   The ITs need a Service Bus namespace and were not run locally (CI runs them).
   
   Backport: camel-4.18.x has the same lock renewer, together with CAMEL-26969.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-27066](https://issues.apache.org/jira/browse/CAMEL-27066).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built camel-azure-servicebus and the core modules with the formatter 
and import-sort plugins and ran the camel-azure-servicebus unit tests. I did 
not run the full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to