[
https://issues.apache.org/jira/browse/CAMEL-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shashank updated CAMEL-27066:
-----------------------------
Description:
Found by the DST world of CAMEL-26956 once the CAMEL-26969 fix was on main.
h3. What happens
The {{LockRenewer}} of {{ServiceBusConsumer}} runs every 10 s (or a second
later) and renews a message lock when it expires within 20 s. Since CAMEL-26969
a transient renewal error keeps the entry, so the next run tries again. But
depending on where the runs fall, the first renewal of a lock can come with
only 9 to 11 s left, for any lock duration, and the next run then comes after
the lock expired.
Example with a queue lock duration of 30 s (seed 113 of the world):
* t=37 s: the lock is renewed until 68.9 s.
* t=48 s: nothing to do, the lock expires later than now + 20 s.
* t=59 s: the renewal fails with a transient error.
* t=70 s: the next run renews, but the lock expired at 68.9 s. The renewal
fails with a lost lock and the message is delivered again while the first
exchange is still being processed.
The same can happen with longer locks (for example 42 s with 11 s gaps, or 60 s
with four 10 s gaps in a row). In the world, 11 of 2000 seeds fail this way
with {{-Ddst.servicebus.strictRenew=true}}.
h3. Fix
Run the lock renewer every 5 s instead of 10 s and keep the 20 s threshold (as
a constant of its own). When the runs come on time, the first renewal comes
with more than 14 s left and a retry after a transient error with more than 8 s
left, without a timer and without more renewal calls when renewals succeed (a
60 s lock is still renewed about every 42 s). A short bounded retry, or
renewing within 30 s with 10 s runs, were considered: the first needs a timer
or a retry in the error callback, the second renews more often (a 60 s lock
about every 33 s instead of 44) and still failed 9 of 2000 seeds in the world.
With the fix, the world fails 1 of 2000 with strictRenew (three faults in a
row). The world then drops strictRenew and excuses a lost lock only after two
renewal faults in a row.
_Filed with Claude Code on behalf of allthingssecurity._
was:
Found by the DST world of CAMEL-26956 once the CAMEL-26969 fix was on main.
h3. What happens
The {{LockRenewer}} of {{ServiceBusConsumer}} runs every 10 s (or a second
later) and renews a message lock when it expires within two intervals. Since
CAMEL-26969 a transient renewal error keeps the entry, so the next run tries
again. But the next run comes about 10 s later, and by then the lock may be
gone.
Example with a queue lock duration of 30 s (seed 113 of the world):
* t=37 s: the lock is renewed until 68.9 s.
* t=48 s: nothing to do, the lock expires later than now + 20 s.
* t=59 s: the renewal fails with a transient error.
* t=70 s: the next run renews, but the lock expired at 68.9 s. The renewal
fails with a lost lock and the message is delivered again while the first
exchange is still being processed.
So with a lock duration below about 40 s, one transient error is enough to lose
the lock. In the world, 11 of 2000 seeds fail this way (all with a 30 s lock
and a renewal fault rate of 0.05 or 0.2).
h3. Fix
Try a transient failure again soon, while the lock is still held, instead of
waiting for the next run (a short bounded retry of the renewal), and keep the
next run as the fallback. The world then runs with
{{-Ddst.servicebus.strictRenew=true}} as its default, and the excuse for
renewal faults goes away.
_Filed with Claude Code on behalf of allthingssecurity._
> camel-azure-servicebus - the Camel lock renewer retries a failed renewal only
> on its next run, which can come after the lock expired
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27066
> URL: https://issues.apache.org/jira/browse/CAMEL-27066
> Project: Camel
> Issue Type: Bug
> Components: camel-azure-servicebus
> Reporter: shashank
> Assignee: shashank
> Priority: Minor
>
> Found by the DST world of CAMEL-26956 once the CAMEL-26969 fix was on main.
> h3. What happens
> The {{LockRenewer}} of {{ServiceBusConsumer}} runs every 10 s (or a second
> later) and renews a message lock when it expires within 20 s. Since
> CAMEL-26969 a transient renewal error keeps the entry, so the next run tries
> again. But depending on where the runs fall, the first renewal of a lock can
> come with only 9 to 11 s left, for any lock duration, and the next run then
> comes after the lock expired.
> Example with a queue lock duration of 30 s (seed 113 of the world):
> * t=37 s: the lock is renewed until 68.9 s.
> * t=48 s: nothing to do, the lock expires later than now + 20 s.
> * t=59 s: the renewal fails with a transient error.
> * t=70 s: the next run renews, but the lock expired at 68.9 s. The renewal
> fails with a lost lock and the message is delivered again while the first
> exchange is still being processed.
> The same can happen with longer locks (for example 42 s with 11 s gaps, or 60
> s with four 10 s gaps in a row). In the world, 11 of 2000 seeds fail this way
> with {{-Ddst.servicebus.strictRenew=true}}.
> h3. Fix
> Run the lock renewer every 5 s instead of 10 s and keep the 20 s threshold
> (as a constant of its own). When the runs come on time, the first renewal
> comes with more than 14 s left and a retry after a transient error with more
> than 8 s left, without a timer and without more renewal calls when renewals
> succeed (a 60 s lock is still renewed about every 42 s). A short bounded
> retry, or renewing within 30 s with 10 s runs, were considered: the first
> needs a timer or a retry in the error callback, the second renews more often
> (a 60 s lock about every 33 s instead of 44) and still failed 9 of 2000 seeds
> in the world. With the fix, the world fails 1 of 2000 with strictRenew (three
> faults in a row). The world then drops strictRenew and excuses a lost lock
> only after two renewal faults in a row.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)