Federico Mariani created CAMEL-27013:
----------------------------------------

             Summary: camel-sql - JdbcOrphanLockAwareIdempotentRepository: 
after a database outage longer than lockMaxAgeMillis, an instance takes over 
the keys it still processes, so their messages are processed twice
                 Key: CAMEL-27013
                 URL: https://issues.apache.org/jira/browse/CAMEL-27013
             Project: Camel
          Issue Type: Bug
          Components: camel-sql
            Reporter: Federico Mariani
             Fix For: 4.24.0


{{JdbcOrphanLockAwareIdempotentRepository}} keeps the leases of the keys an 
instance holds alive: a keep-alive task sets their {{createdAt}} to now every 
{{lockKeepAliveIntervalMillis}}. When the instance cannot reach the database 
for longer than {{lockMaxAgeMillis}}, every keep-alive round fails (logged at 
ERROR, {{keepAlive()}}, line 426 of 
{{components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java}}
 on main at 21b8ed8e9613), and the leases grow older than {{lockMaxAgeMillis}}. 
Once the database is back, the orphan check treats them as orphans by their age 
only, also on the instance that holds them:

* {{queryForInt}} (line 127) counts the row only when {{createdAt >= now - 
lockMaxAgeMillis OR createdAt IS NULL}}, so {{contains()}} returns false and 
{{add()}} goes on to {{insert()}} ({{AbstractJdbcMessageIdRepository}}, lines 
142-145);
* {{insert()}} (line 285) takes the row over with {{UPDATE ... AND createdAt < 
?}}, and {{add()}} returns true.

Neither looks at {{processorNameMessageIdSet}}, the keys this instance holds. A 
copy of a message that the instance is still processing (a redelivery, the same 
file polled again, a duplicate sent by the producer) is accepted and processed 
a second time on the same instance, until the next keep-alive refreshes the 
lease.

Found with fault-injection tests (Toxiproxy between Camel and a real 
PostgreSQL, Idempotent Consumer in the default eager mode, {{lockMaxAgeMillis}} 
30 s, {{lockKeepAliveIntervalMillis}} 3 s, the database unreachable for 35 s 
while a message is processed): ids processed twice in every such run. On 4.23.0 
the ids processed before the outage were taken over too, as they stayed in the 
keep-alive set; CAMEL-26816 (confirm sets {{createdAt}} to null) fixed that 
part on main. The keys in flight during the outage are still taken over on main.

h3. Steps to reproduce (main, H2, no network)

A repository on a {{DataSource}} whose {{getConnection()}} throws while an 
"outage" flag is set:
# {{add("ORDER_1")}} returns true: the message is being processed.
# Outage: {{keepAlive()}} fails. Set the {{createdAt}} of the row older than 
{{lockMaxAgeMillis}}, as when the outage lasts longer than that.
# End of the outage, before the next keep-alive: {{contains("ORDER_1")}} and 
{{add("ORDER_1")}} on the same repository.

h3. Actual

{{contains("ORDER_1")}} returns false and {{add("ORDER_1")}} returns true: the 
instance takes over its own key, and the message is processed twice.

h3. Expected

A key the instance still holds is not an orphan, whatever the age of its lease: 
{{contains}} returns true and {{add}} false; the next keep-alive refreshes the 
lease.

h3. Another instance (a limitation of the lease: documented, not changed)

Another instance that reaches the database while this one cannot (a network 
partition, an exhausted pool), or right after the outage and before the next 
keep-alive of this instance, cannot tell this instance from one that crashed: 
it takes the lease over and processes the message while this instance still 
does. Without a fencing token this is inherent to a lease. The fix documents it 
in the Orphan Lock section of {{sql-component.adoc}} (choose 
{{lockMaxAgeMillis}} longer than the longest database outage to survive), and 
the keep-alive logs once (WARN) when it has failed for longer than 
{{lockMaxAgeMillis}}, as the leases may then be taken over, and once (INFO) 
when it works again; today it only logs the same ERROR every round.

h3. Possible follow-ups (not in this fix)

* An owner column (an id of the instance, or of the lease), written by the 
insert and the takeover and checked by the keep-alive, {{confirm}} and 
{{remove}}. Today, once another instance took a lease over, the keep-alive of 
the first instance refreshes the row of the new owner, and a {{remove}} of the 
first instance (its processing failed) deletes the lock of the new owner, so a 
third copy can be accepted. This needs a schema change.
* The instance could report (for example in a health check) that its leases may 
have expired, but it cannot stop a message it already processes; only the 
system the route writes to can reject a stale owner (a fencing token).

_Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to