Federico Mariani created CAMEL-27013:
----------------------------------------
Summary: camel-sql - JdbcOrphanLockAwareIdempotentRepository:
after a database outage longer than lockMaxAgeMillis, an instance takes over
the keys it still processes, so their messages are processed twice
Key: CAMEL-27013
URL: https://issues.apache.org/jira/browse/CAMEL-27013
Project: Camel
Issue Type: Bug
Components: camel-sql
Reporter: Federico Mariani
Fix For: 4.24.0
{{JdbcOrphanLockAwareIdempotentRepository}} keeps the leases of the keys an
instance holds alive: a keep-alive task sets their {{createdAt}} to now every
{{lockKeepAliveIntervalMillis}}. When the instance cannot reach the database
for longer than {{lockMaxAgeMillis}}, every keep-alive round fails (logged at
ERROR, {{keepAlive()}}, line 426 of
{{components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java}}
on main at 21b8ed8e9613), and the leases grow older than {{lockMaxAgeMillis}}.
Once the database is back, the orphan check treats them as orphans by their age
only, also on the instance that holds them:
* {{queryForInt}} (line 127) counts the row only when {{createdAt >= now -
lockMaxAgeMillis OR createdAt IS NULL}}, so {{contains()}} returns false and
{{add()}} goes on to {{insert()}} ({{AbstractJdbcMessageIdRepository}}, lines
142-145);
* {{insert()}} (line 285) takes the row over with {{UPDATE ... AND createdAt <
?}}, and {{add()}} returns true.
Neither looks at {{processorNameMessageIdSet}}, the keys this instance holds. A
copy of a message that the instance is still processing (a redelivery, the same
file polled again, a duplicate sent by the producer) is accepted and processed
a second time on the same instance, until the next keep-alive refreshes the
lease.
Found with fault-injection tests (Toxiproxy between Camel and a real
PostgreSQL, Idempotent Consumer in the default eager mode, {{lockMaxAgeMillis}}
30 s, {{lockKeepAliveIntervalMillis}} 3 s, the database unreachable for 35 s
while a message is processed): ids processed twice in every such run. On 4.23.0
the ids processed before the outage were taken over too, as they stayed in the
keep-alive set; CAMEL-26816 (confirm sets {{createdAt}} to null) fixed that
part on main. The keys in flight during the outage are still taken over on main.
h3. Steps to reproduce (main, H2, no network)
A repository on a {{DataSource}} whose {{getConnection()}} throws while an
"outage" flag is set:
# {{add("ORDER_1")}} returns true: the message is being processed.
# Outage: {{keepAlive()}} fails. Set the {{createdAt}} of the row older than
{{lockMaxAgeMillis}}, as when the outage lasts longer than that.
# End of the outage, before the next keep-alive: {{contains("ORDER_1")}} and
{{add("ORDER_1")}} on the same repository.
h3. Actual
{{contains("ORDER_1")}} returns false and {{add("ORDER_1")}} returns true: the
instance takes over its own key, and the message is processed twice.
h3. Expected
A key the instance still holds is not an orphan, whatever the age of its lease:
{{contains}} returns true and {{add}} false; the next keep-alive refreshes the
lease.
h3. Another instance (a limitation of the lease: documented, not changed)
Another instance that reaches the database while this one cannot (a network
partition, an exhausted pool), or right after the outage and before the next
keep-alive of this instance, cannot tell this instance from one that crashed:
it takes the lease over and processes the message while this instance still
does. Without a fencing token this is inherent to a lease. The fix documents it
in the Orphan Lock section of {{sql-component.adoc}} (choose
{{lockMaxAgeMillis}} longer than the longest database outage to survive), and
the keep-alive logs once (WARN) when it has failed for longer than
{{lockMaxAgeMillis}}, as the leases may then be taken over, and once (INFO)
when it works again; today it only logs the same ERROR every round.
h3. Possible follow-ups (not in this fix)
* An owner column (an id of the instance, or of the lease), written by the
insert and the takeover and checked by the keep-alive, {{confirm}} and
{{remove}}. Today, once another instance took a lease over, the keep-alive of
the first instance refreshes the row of the new owner, and a {{remove}} of the
first instance (its processing failed) deletes the lock of the new owner, so a
third copy can be accepted. This needs a schema change.
* The instance could report (for example in a health check) that its leases may
have expired, but it cannot stop a message it already processes; only the
system the route writes to can reject a stale owner (a fencing token).
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)