[ 
https://issues.apache.org/jira/browse/CAMEL-27013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Federico Mariani resolved CAMEL-27013.
--------------------------------------
    Resolution: Fixed

> camel-sql - JdbcOrphanLockAwareIdempotentRepository: after a database outage 
> longer than lockMaxAgeMillis, an instance takes over the keys it still 
> processes, so their messages are processed twice
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27013
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27013
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-sql
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>             Fix For: 4.24.0
>
>
> {{JdbcOrphanLockAwareIdempotentRepository}} keeps the leases of the keys an 
> instance holds alive: a keep-alive task sets their {{createdAt}} to now every 
> {{lockKeepAliveIntervalMillis}}. When the instance cannot reach the database 
> for longer than {{lockMaxAgeMillis}}, every keep-alive round fails (logged at 
> ERROR, {{keepAlive()}}, line 426 of 
> {{components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java}}
>  on main at 21b8ed8e9613), and the leases grow older than 
> {{lockMaxAgeMillis}}. Once the database is back, the orphan check treats them 
> as orphans by their age only, also on the instance that holds them:
> * {{queryForInt}} (line 127) counts the row only when {{createdAt >= now - 
> lockMaxAgeMillis OR createdAt IS NULL}}, so {{contains()}} returns false and 
> {{add()}} goes on to {{insert()}} ({{AbstractJdbcMessageIdRepository}}, lines 
> 142-145);
> * {{insert()}} (line 285) takes the row over with {{UPDATE ... AND createdAt 
> < ?}}, and {{add()}} returns true.
> Neither looks at {{processorNameMessageIdSet}}, the keys this instance holds. 
> A copy of a message that the instance is still processing (a redelivery, the 
> same file polled again, a duplicate sent by the producer) is accepted and 
> processed a second time on the same instance, until the next keep-alive 
> refreshes the lease.
> Found with fault-injection tests (Toxiproxy between Camel and a real 
> PostgreSQL, Idempotent Consumer in the default eager mode, 
> {{lockMaxAgeMillis}} 30 s, {{lockKeepAliveIntervalMillis}} 3 s, the database 
> unreachable for 35 s while a message is processed): ids processed twice in 
> every such run. On 4.23.0 the ids processed before the outage were taken over 
> too, as they stayed in the keep-alive set; CAMEL-26816 (confirm sets 
> {{createdAt}} to null) fixed that part on main. The keys in flight during the 
> outage are still taken over on main.
> h3. Steps to reproduce (main, H2, no network)
> A repository on a {{DataSource}} whose {{getConnection()}} throws while an 
> "outage" flag is set:
> # {{add("ORDER_1")}} returns true: the message is being processed.
> # Outage: {{keepAlive()}} fails. Set the {{createdAt}} of the row older than 
> {{lockMaxAgeMillis}}, as when the outage lasts longer than that.
> # End of the outage, before the next keep-alive: {{contains("ORDER_1")}} and 
> {{add("ORDER_1")}} on the same repository.
> h3. Actual
> {{contains("ORDER_1")}} returns false and {{add("ORDER_1")}} returns true: 
> the instance takes over its own key, and the message is processed twice.
> h3. Expected
> A key the instance still holds is not an orphan, whatever the age of its 
> lease: {{contains}} returns true and {{add}} false; the next keep-alive 
> refreshes the lease.
> h3. Another instance (a limitation of the lease: documented, not changed)
> Another instance that reaches the database while this one cannot (a network 
> partition, an exhausted pool), or right after the outage and before the next 
> keep-alive of this instance, cannot tell this instance from one that crashed: 
> it takes the lease over and processes the message while this instance still 
> does. Without a fencing token this is inherent to a lease. The fix documents 
> it in the Orphan Lock section of {{sql-component.adoc}} (choose 
> {{lockMaxAgeMillis}} longer than the longest database outage to survive), and 
> the keep-alive logs once (WARN) when it has failed for longer than 
> {{lockMaxAgeMillis}}, as the leases may then be taken over, and once (INFO) 
> when it works again; today it only logs the same ERROR every round.
> h3. Possible follow-ups (not in this fix)
> * An owner column (an id of the instance, or of the lease), written by the 
> insert and the takeover and checked by the keep-alive, {{confirm}} and 
> {{remove}}. Today, once another instance took a lease over, the keep-alive of 
> the first instance refreshes the row of the new owner, and a {{remove}} of 
> the first instance (its processing failed) deletes the lock of the new owner, 
> so a third copy can be accepted. This needs a schema change.
> * The instance could report (for example in a health check) that its leases 
> may have expired, but it cannot stop a message it already processes; only the 
> system the route writes to can reject a stale owner (a fencing token).
> _Claude Code on behalf of Croway_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to