This is an automated email from the ASF dual-hosted git repository.
Croway pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 3b9793e25917 CAMEL-27013: camel-sql -
JdbcOrphanLockAwareIdempotentRepository does not take over its own keys after
an outage longer than lockMaxAgeMillis (#30162)
3b9793e25917 is described below
commit 3b9793e25917bed5493f4baef2d3e1441e19a412
Author: Federico Mariani <[email protected]>
AuthorDate: Mon Nov 16 14:10:06 2026 +0100
CAMEL-27013: camel-sql - JdbcOrphanLockAwareIdempotentRepository does not
take over its own keys after an outage longer than lockMaxAgeMillis (#30162)
* CAMEL-27013: camel-sql - JdbcOrphanLockAwareIdempotentRepository: a key
the instance still holds is not taken over by that instance after a database
outage longer than lockMaxAgeMillis
When the keep-alive could not refresh the leases for longer than
lockMaxAgeMillis
(the database could not be reached), the orphan check treated the keys the
instance
still processed as orphans by age only, also on that instance: contains()
returned
false and add() took the row over, so a copy of a message in flight was
processed
a second time.
- queryForInt counts a key of processorNameMessageIdSet as present,
whatever the
age of its lease; the next keep-alive refreshes it as before
- the keep-alive logs one WARN when it has not refreshed the leases for
longer than
lockMaxAgeMillis (another instance may take them over), and one INFO when
it
refreshes them again
- docs: the Orphan Lock section says what an outage longer than
lockMaxAgeMillis
means across instances (another instance takes the lease over; choose
lockMaxAgeMillis longer than the longest outage to survive)
* CAMEL-27013: camel-sql - JdbcOrphanLockAwareIdempotentRepository: a key
whose confirm runs is still held by the instance
The confirm takes the key out of the keep-alive before its transaction
starts and
before its UPDATE, so that the keep-alive does not wait for its row lock.
Right after
an outage longer than lockMaxAgeMillis, before the next keep-alive, a copy
of the
message on the same instance could still take the aged lease over in that
window.
- confirm() adds the key to a confirming set before it takes it out of the
keep-alive set, and removes it when the confirm ends; queryForInt checks
both
sets, the keep-alive still ignores it
- remove() has no such window: its DELETE runs before the key leaves the
keep-alive
set; a test guards that order
- docs: the WARN of the keep-alive is logged while the instance holds keys
- tests: RecordingLogger, a package-private helper shared by the confirm
and outage
tests
---
.../apache/camel/catalog/docs/sql-component.adoc | 6 +-
.../camel-sql/src/main/docs/sql-component.adoc | 6 +-
.../JdbcOrphanLockAwareIdempotentRepository.java | 64 ++++
...anLockAwareIdempotentRepositoryConfirmTest.java | 45 +--
...hanLockAwareIdempotentRepositoryOutageTest.java | 332 +++++++++++++++++++++
...anLockAwareIdempotentRepositoryRestartTest.java | 10 +-
...dbcOrphanLockAwareIdempotentRepositoryTest.java | 17 +-
.../processor/idempotent/jdbc/RecordingLogger.java | 86 ++++++
8 files changed, 518 insertions(+), 48 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sql-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sql-component.adoc
index ad7f4eec26e2..272619d4b03f 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sql-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/sql-component.adoc
@@ -847,18 +847,22 @@ One of the limitations of
`org.apache.camel.processor.idempotent.jdbc.JdbcMessag
* lock already exists and its `createdAt` is null: the key was confirmed, so
it is a duplicate, as in `JdbcMessageIdRepository`.
+* the instance requesting the lock holds it already (it added the key and has
not confirmed or removed it yet): the key is a duplicate, whatever its
`createdAt`.
+
A key that is confirmed (the message was processed, for example by the
Idempotent Consumer, by the file consumer with `idempotent=true` and the
default `idempotentEager=true`, or by the file consumer with
`readLock=idempotent` (also `idempotent-changed` and `idempotent-rename`) and
the default `readLockRemoveOnCommit=false`) is no longer a lock: its
`createdAt` is set to null and it is no longer kept alive, so another instance
does not take it over when the instance that added it stops. [...]
A confirmed key has no age, so a purge by `createdAt` does not delete it. To
purge confirmed keys by age, add a column of your own to the table, for example
`insertedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP` (the repository does not write
it), or set `confirmQuery` to a statement that also stores the time of the
confirm in such a column, and purge by that column.
The `createdAt` column must store null. The repository checks this once when
it starts, in a transaction that it rolls back: it inserts a row, confirms it
with `confirmQuery`, and checks that its `createdAt` is null. When the check
fails (a `NOT NULL` column, a MySQL or MariaDB `TIMESTAMP` column with
`explicit_defaults_for_timestamp=OFF`, or an error of the database), it logs a
WARN and does not confirm keys until the repository is started again: they stay
locks, as before, and a confir [...]
+A lock is kept alive only while its instance reaches the database. When the
keep-alive fails (the database is down or cannot be reached, a failover, a
connection pool with no free connection), it logs an ERROR on every round; when
the instance holds keys and the keep-alive has not refreshed their locks for
longer than `lockMaxAgeMillis`, it logs a WARN, as they may now be taken over,
and an INFO when it refreshes them again. The instance itself keeps treating
the keys it holds as duplica [...]
+
This repository has these additional configuration parameters
[cols="1,1"]
|===
|Parameter | Description
-|`lockMaxAgeMillis` | This refers to the duration after which the lock is
considered orphaned, i.e., if the `currentTimestamp - createdAt >=
lockMaxAgeMillis` then lock is orphaned.
+|`lockMaxAgeMillis` | This refers to the duration after which the lock is
considered orphaned, i.e., if the `currentTimestamp - createdAt >=
lockMaxAgeMillis` then lock is orphaned. Choose it longer than the longest
database outage to survive, see above.
|`lockKeepAliveIntervalMillis` | The frequency at which keep-alive updates are
done to createdAt Timestamp column.
|`confirmQuery` | The query that confirms a key by setting its `createdAt` to
null; it may set other columns of the table too. It follows `tableName`; set it
when the table is set through custom queries instead. Set it to null or empty
to keep confirmed keys as locks. It is checked when the repository starts; a
query set later is used after the next start.
|===
diff --git a/components/camel-sql/src/main/docs/sql-component.adoc
b/components/camel-sql/src/main/docs/sql-component.adoc
index ad7f4eec26e2..272619d4b03f 100644
--- a/components/camel-sql/src/main/docs/sql-component.adoc
+++ b/components/camel-sql/src/main/docs/sql-component.adoc
@@ -847,18 +847,22 @@ One of the limitations of
`org.apache.camel.processor.idempotent.jdbc.JdbcMessag
* lock already exists and its `createdAt` is null: the key was confirmed, so
it is a duplicate, as in `JdbcMessageIdRepository`.
+* the instance requesting the lock holds it already (it added the key and has
not confirmed or removed it yet): the key is a duplicate, whatever its
`createdAt`.
+
A key that is confirmed (the message was processed, for example by the
Idempotent Consumer, by the file consumer with `idempotent=true` and the
default `idempotentEager=true`, or by the file consumer with
`readLock=idempotent` (also `idempotent-changed` and `idempotent-rename`) and
the default `readLockRemoveOnCommit=false`) is no longer a lock: its
`createdAt` is set to null and it is no longer kept alive, so another instance
does not take it over when the instance that added it stops. [...]
A confirmed key has no age, so a purge by `createdAt` does not delete it. To
purge confirmed keys by age, add a column of your own to the table, for example
`insertedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP` (the repository does not write
it), or set `confirmQuery` to a statement that also stores the time of the
confirm in such a column, and purge by that column.
The `createdAt` column must store null. The repository checks this once when
it starts, in a transaction that it rolls back: it inserts a row, confirms it
with `confirmQuery`, and checks that its `createdAt` is null. When the check
fails (a `NOT NULL` column, a MySQL or MariaDB `TIMESTAMP` column with
`explicit_defaults_for_timestamp=OFF`, or an error of the database), it logs a
WARN and does not confirm keys until the repository is started again: they stay
locks, as before, and a confir [...]
+A lock is kept alive only while its instance reaches the database. When the
keep-alive fails (the database is down or cannot be reached, a failover, a
connection pool with no free connection), it logs an ERROR on every round; when
the instance holds keys and the keep-alive has not refreshed their locks for
longer than `lockMaxAgeMillis`, it logs a WARN, as they may now be taken over,
and an INFO when it refreshes them again. The instance itself keeps treating
the keys it holds as duplica [...]
+
This repository has these additional configuration parameters
[cols="1,1"]
|===
|Parameter | Description
-|`lockMaxAgeMillis` | This refers to the duration after which the lock is
considered orphaned, i.e., if the `currentTimestamp - createdAt >=
lockMaxAgeMillis` then lock is orphaned.
+|`lockMaxAgeMillis` | This refers to the duration after which the lock is
considered orphaned, i.e., if the `currentTimestamp - createdAt >=
lockMaxAgeMillis` then lock is orphaned. Choose it longer than the longest
database outage to survive, see above.
|`lockKeepAliveIntervalMillis` | The frequency at which keep-alive updates are
done to createdAt Timestamp column.
|`confirmQuery` | The query that confirms a key by setting its `createdAt` to
null; it may set other columns of the table too. It follows `tableName`; set it
when the table is set through custom queries instead. Set it to null or empty
to keep confirmed keys as locks. It is checked when the repository starts; a
query set later is used after the next start.
|===
diff --git
a/components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java
b/components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java
index 66d9a8514014..608fba264aef 100644
---
a/components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java
+++
b/components/camel-sql/src/main/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepository.java
@@ -51,6 +51,12 @@ import
org.springframework.transaction.support.TransactionTemplate;
* CAMEL_MESSAGEPROCESSED table or if in case the instance holding the lock
has crashed. This is determined if the
* timestamp on the createdAt column is more than the lockMaxAge.
*
+ * A key that this instance holds (it was added and is not confirmed or
removed yet) is never an orphan for this
+ * instance, whatever the age of its createdAt: when the keep-alive cannot
refresh it for longer than the lockMaxAge
+ * (the database cannot be reached), the message may still be processed by
this instance, which then still treats the
+ * key as a duplicate. Another instance cannot tell this instance from one
that crashed, so it does take the key over
+ * then: the lockMaxAge must be longer than the longest database outage that
must not lead to a message processed twice.
+ *
* A key that is confirmed (the message was processed) is no longer a lock:
its createdAt is set to null, which the
* orphan check never treats as old, and it is no longer kept alive. So it
stays a duplicate after the instance that
* added it stops, as in {@link JdbcMessageIdRepository}. A confirmed key has
no age then, so a purge by createdAt does
@@ -75,6 +81,10 @@ public class JdbcOrphanLockAwareIdempotentRepository extends
JdbcMessageIdReposi
// run, and the keep-alive needs one too, so a lock held across SQL could
deadlock with a small pool
private final Set<ProcessorNameAndMessageId> processorNameMessageIdSet =
ConcurrentHashMap.newKeySet();
+ // the keys whose confirm runs: the confirm takes them out of
processorNameMessageIdSet before its UPDATE, so that the
+ // keep-alive does not wait for the row lock of the confirm, but this
instance still holds them until the confirm ends
+ private final Set<ProcessorNameAndMessageId> confirming =
ConcurrentHashMap.newKeySet();
+
private ExecutorServiceManager executorServiceManager;
private ScheduledExecutorService executorService;
@@ -94,6 +104,11 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
// set when the repository starts: whether the confirm query stores null
in createdAt
private volatile boolean confirmChecked;
+ // when the keep-alive last refreshed the leases of this instance (or the
repository started), and whether it logged
+ // that they may have expired since then
+ private volatile long leasesRefreshedAt = System.currentTimeMillis();
+ private volatile boolean leasesMayHaveExpired;
+
public JdbcOrphanLockAwareIdempotentRepository(CamelContext camelContext) {
super();
this.context = camelContext;
@@ -118,6 +133,12 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
@Override
protected int queryForInt(String key) {
+ // a key this instance holds is not an orphan, whatever the age of its
lease: when the keep-alive could not refresh
+ // it for longer than lockMaxAge (the database could not be reached),
this instance may still process the message,
+ // and taking the key over would process it a second time
+ if (holds(new ProcessorNameAndMessageId(processorName, key))) {
+ return 1;
+ }
/**
* If the update timestamp time is more than lockMaxAge then assume
that the lock is orphan and the process
* which had acquired the lock has died
@@ -130,6 +151,17 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
xMillisAgo);
}
+ // whether this instance holds the key: it is kept alive, or its confirm
runs. The confirm adds the key to confirming
+ // before it takes it out of processorNameMessageIdSet, so the first two
reads find a key whose confirm starts
+ private boolean holds(ProcessorNameAndMessageId id) {
+ if (processorNameMessageIdSet.contains(id) || confirming.contains(id))
{
+ return true;
+ }
+ // a confirm that failed puts the key back in
processorNameMessageIdSet before it takes it out of confirming,
+ // possibly between the two reads above
+ return processorNameMessageIdSet.contains(id);
+ }
+
@Override
protected int delete(String key) {
int result = super.delete(key);
@@ -162,6 +194,8 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
// the row until that transaction ends, and a later keep-alive that
updated the row would wait for it (and could
// deadlock with a later confirm or remove in the same transaction).
It is kept alive again when the confirm fails
// or its transaction does not commit; not when no row matched, as
then there is no lease of this instance.
+ // Until the confirm ends, this instance still holds the key (see
queryForInt), but does not keep it alive.
+ confirming.add(id);
boolean keptAlive = processorNameMessageIdSet.remove(id);
boolean completed = false;
try {
@@ -190,6 +224,7 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
if (keptAlive && !completed) {
processorNameMessageIdSet.add(id);
}
+ confirming.remove(id);
}
}
@@ -376,6 +411,8 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
String query = getConfirmQuery();
confirmChecked = ObjectHelper.isNotEmpty(query) &&
checkConfirmQuery(query);
+ leasesRefreshedAt = System.currentTimeMillis();
+ leasesMayHaveExpired = false;
executorServiceManager = context.getExecutorServiceManager();
executorService =
executorServiceManager.newSingleThreadScheduledExecutor(this,
this.getClass().getSimpleName());
@@ -410,6 +447,8 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
}
void keepAlive() {
+ // the leases get a later timestamp, taken once the connection was
obtained
+ long started = System.currentTimeMillis();
try {
transactionTemplate.execute(status -> {
// take the timestamp and the keys inside the transaction,
after the connection was obtained (with the
@@ -424,6 +463,31 @@ public class JdbcOrphanLockAwareIdempotentRepository
extends JdbcMessageIdReposi
});
} catch (Exception e) {
log.error("failed updating createdAt in keepAlive due to ", e);
+ warnOnceWhenLeasesMayHaveExpired();
+ return;
+ }
+ if (leasesMayHaveExpired) {
+ leasesMayHaveExpired = false;
+ log.info("The keep-alive of processor {} refreshes the leases of
this instance again, {} ms after it last did:"
+ + " another instance may have taken over some of its keys
meanwhile",
+ processorName, started - leasesRefreshedAt);
+ }
+ leasesRefreshedAt = started;
+ }
+
+ // once per outage: the keep-alive has not refreshed the leases for longer
than lockMaxAge, so other instances may take
+ // over the keys of this instance, as if it had crashed (this instance
does not, see queryForInt)
+ private void warnOnceWhenLeasesMayHaveExpired() {
+ // now, as a round can fail only after a while (such as the connection
timeout of the pool)
+ long notRefreshedFor = System.currentTimeMillis() - leasesRefreshedAt;
+ int held = processorNameMessageIdSet.size();
+ if (!leasesMayHaveExpired && held > 0 && notRefreshedFor >=
lockMaxAgeMillis) {
+ leasesMayHaveExpired = true;
+ log.warn("The keep-alive of processor {} has not refreshed the
leases of this instance for {} ms, longer than"
+ + " lockMaxAgeMillis ({} ms): another instance may take
over the {} keys this instance holds, as if it"
+ + " had crashed, and process their messages a second
time. This instance still treats them as"
+ + " duplicates. Set lockMaxAgeMillis longer than the
longest database outage to survive.",
+ processorName, notRefreshedFor, lockMaxAgeMillis, held);
}
}
diff --git
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryConfirmTest.java
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryConfirmTest.java
index dc8ace80593b..39b999ad4c76 100644
---
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryConfirmTest.java
+++
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryConfirmTest.java
@@ -16,10 +16,8 @@
*/
package org.apache.camel.processor.idempotent.jdbc;
-import java.lang.reflect.Proxy;
import java.sql.Timestamp;
import java.util.List;
-import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
@@ -32,7 +30,6 @@ import
org.apache.camel.processor.idempotent.jdbc.JdbcOrphanLockAwareIdempotentR
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
-import org.slf4j.Logger;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabase;
@@ -199,9 +196,10 @@ class JdbcOrphanLockAwareIdempotentRepositoryConfirmTest {
@Test
void testCheckAtStartWithAConfirmQueryThatDoesNotStoreNull() {
// as a MySQL TIMESTAMP column with
explicit_defaults_for_timestamp=OFF, which stores the current time for null
- List<String> warnings = new CopyOnWriteArrayList<>();
+ RecordingLogger logs = new RecordingLogger();
+ List<String> warnings = logs.warnings();
JdbcOrphanLockAwareIdempotentRepository node = newRepository();
- node.log = recordingLogger(warnings);
+ node.log = logs.logger();
node.setConfirmQuery(
"UPDATE CAMEL_MESSAGEPROCESSED SET createdAt =
CURRENT_TIMESTAMP WHERE processorName = ? AND messageId = ?");
node.start();
@@ -226,9 +224,10 @@ class JdbcOrphanLockAwareIdempotentRepositoryConfirmTest {
void
testCheckAtStartWithANotNullColumnKeepsTheCallerTransactionCommittable() {
jdbcTemplate.execute("CREATE TABLE CAMEL_MESSAGEPROCESSED_NN
(processorName VARCHAR(255), messageId VARCHAR(100), "
+ "createdAt TIMESTAMP NOT NULL, PRIMARY KEY
(processorName, messageId))");
- List<String> warnings = new CopyOnWriteArrayList<>();
+ RecordingLogger logs = new RecordingLogger();
+ List<String> warnings = logs.warnings();
JdbcOrphanLockAwareIdempotentRepository node = newRepository();
- node.log = recordingLogger(warnings);
+ node.log = logs.logger();
node.setTableName("CAMEL_MESSAGEPROCESSED_NN");
node.start();
try {
@@ -254,8 +253,9 @@ class JdbcOrphanLockAwareIdempotentRepositoryConfirmTest {
@Test
void testConfirmThatFailsLogsAWarnEachTime() {
- List<String> warnings = new CopyOnWriteArrayList<>();
- nodeA.log = recordingLogger(warnings);
+ RecordingLogger logs = new RecordingLogger();
+ List<String> warnings = logs.warnings();
+ nodeA.log = logs.logger();
assertTrue(nodeA.add("FILE_9"));
assertTrue(nodeA.add("FILE_14"));
// the column no longer stores null after the start (a failure the
check at start cannot see)
@@ -384,33 +384,6 @@ class JdbcOrphanLockAwareIdempotentRepositoryConfirmTest {
PROCESSOR_NAME, key);
}
- // records the WARN messages of the repository
- private static Logger recordingLogger(List<String> warnings) {
- return (Logger) Proxy.newProxyInstance(Logger.class.getClassLoader(),
new Class<?>[] { Logger.class },
- (proxy, method, args) -> {
- switch (method.getName()) {
- case "hashCode":
- return System.identityHashCode(proxy);
- case "equals":
- return proxy == args[0];
- case "toString":
- return "recordingLogger";
- default:
- break;
- }
- if ("warn".equals(method.getName()) && args != null &&
args.length > 0) {
- warnings.add(String.valueOf(args[0]));
- }
- if (method.getReturnType() == boolean.class) {
- return false;
- }
- if (method.getReturnType() == String.class) {
- return "test";
- }
- return null;
- });
- }
-
private static ProcessorNameAndMessageId id(String key) {
return new ProcessorNameAndMessageId(PROCESSOR_NAME, key);
}
diff --git
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryOutageTest.java
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryOutageTest.java
new file mode 100644
index 000000000000..ad52c61cd0bc
--- /dev/null
+++
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryOutageTest.java
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.processor.idempotent.jdbc;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+import javax.sql.DataSource;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.jdbc.datasource.DelegatingDataSource;
+import org.springframework.jdbc.datasource.embedded.EmbeddedDatabase;
+import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder;
+import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * A database outage longer than lockMaxAgeMillis: the keep-alive of node A
fails, so the leases of the keys node A is
+ * still processing grow older than lockMaxAgeMillis. Node A must still treat
these keys as duplicates; another node can
+ * take them over, as it cannot tell node A from a node that crashed (see the
Orphan Lock section of the docs). Two
+ * repository instances (two nodes) share one H2 database; only node A loses
it. A statement of node A can be held
+ * before it is sent, so that a copy of a message reaches node A while it
confirms or removes the key.
+ */
+class JdbcOrphanLockAwareIdempotentRepositoryOutageTest {
+
+ private static final String PROCESSOR_NAME = "APP_1";
+ private static final long LOCK_MAX_AGE_MILLIS = 300_000L;
+
+ private EmbeddedDatabase database;
+ private JdbcTemplate jdbcTemplate;
+ private CamelContext context;
+ private Outage outage;
+ private PausingJdbcTemplate statements;
+ private ExecutorService processing;
+ private JdbcOrphanLockAwareIdempotentRepository nodeA;
+ private JdbcOrphanLockAwareIdempotentRepository nodeB;
+
+ @BeforeEach
+ void setup() {
+ database = new EmbeddedDatabaseBuilder()
+ .setType(EmbeddedDatabaseType.H2)
+ .generateUniqueName(true)
+ .build();
+ jdbcTemplate = new JdbcTemplate(database);
+ context = new DefaultCamelContext();
+ outage = new Outage(database);
+ statements = new PausingJdbcTemplate(outage);
+ processing = Executors.newSingleThreadExecutor();
+ nodeA = newRepository(outage, LOCK_MAX_AGE_MILLIS, LOCK_MAX_AGE_MILLIS
/ 2);
+ // no statement is held unless a test says so
+ nodeA.setJdbcTemplate(statements);
+ nodeB = newRepository(database, LOCK_MAX_AGE_MILLIS,
LOCK_MAX_AGE_MILLIS / 2);
+ nodeA.start();
+ nodeB.start();
+ }
+
+ @AfterEach
+ void tearDown() {
+ statements.resume();
+ processing.shutdownNow();
+ nodeA.shutdown();
+ nodeB.shutdown();
+ context.stop();
+ database.shutdown();
+ }
+
+ @Test
+ void
testKeyInFlightIsStillADuplicateOnItsNodeAfterAnOutageLongerThanLockMaxAge() {
+ // node A processes ORDER_1 (the Idempotent Consumer added it, and
confirms it once the message is processed)
+ assertTrue(nodeA.add("ORDER_1"));
+
+ outage.begin();
+ // the keep-alive fails (it logs an ERROR), and the outage lasts
longer than the lock max age
+ nodeA.keepAlive();
+ assertThrows(RuntimeException.class, () -> nodeA.add("ORDER_1"));
+ Timestamp expired = expireLeases();
+ outage.end();
+
+ // a copy of ORDER_1 (a redelivery, or the same file polled again)
reaches node A, which still processes ORDER_1
+ assertTrue(nodeA.contains("ORDER_1"), "A key that node A still holds
must be a duplicate on node A");
+ assertFalse(nodeA.add("ORDER_1"), "A key that node A still holds must
not be taken over by node A");
+
+ // the next keep-alive refreshes the lease, so the key is a lock of
node A again for the other nodes
+ nodeA.keepAlive();
+ assertTrue(createdAt("ORDER_1").after(expired), "The keep-alive did
not refresh the lease after the outage");
+ assertFalse(nodeB.add("ORDER_1"));
+
+ // once processed, the key is confirmed as usual
+ assertTrue(nodeA.confirm("ORDER_1"));
+ assertFalse(nodeA.add("ORDER_1"));
+ }
+
+ @Test
+ void testKeyThatFailedDuringTheOutageIsStillADuplicateOnItsNode() {
+ assertTrue(nodeA.add("ORDER_2"));
+
+ outage.begin();
+ // the processing of ORDER_2 fails, and neither the remove nor the
confirm of the key reaches the database: the
+ // key stays a lock of node A, which keeps it alive once the database
is back
+ assertThrows(RuntimeException.class, () -> nodeA.remove("ORDER_2"));
+ assertFalse(nodeA.confirm("ORDER_2"));
+ nodeA.keepAlive();
+ expireLeases();
+ outage.end();
+
+ assertFalse(nodeA.add("ORDER_2"), "A key that node A still holds must
not be taken over by node A");
+ // the key can be removed once the database is back, and added again
+ assertTrue(nodeA.remove("ORDER_2"));
+ assertTrue(nodeA.add("ORDER_2"));
+ }
+
+ @Test
+ void testKeyBeingConfirmedIsStillADuplicateOnItsNodeAfterAnOutage() throws
Exception {
+ assertTrue(nodeA.add("ORDER_5"));
+
+ outage.begin();
+ nodeA.keepAlive();
+ expireLeases();
+ outage.end();
+
+ // ORDER_5 is processed right after the outage, before the next
keep-alive: its confirm takes the key out of the
+ // keep-alive first, then waits before its UPDATE (as when it waits
for a connection of the pool)
+ statements.pauseBefore(nodeA.getConfirmQuery());
+ Future<Boolean> confirmed = processing.submit(() ->
nodeA.confirm("ORDER_5"));
+ assertTrue(statements.awaitPaused(), "The confirm did not reach its
UPDATE");
+
+ // a copy of ORDER_5 reaches node A meanwhile
+ assertFalse(nodeA.add("ORDER_5"), "A key that node A is confirming
must not be taken over by node A");
+ assertTrue(nodeA.contains("ORDER_5"), "A key that node A is confirming
must be a duplicate on node A");
+
+ statements.resume();
+ assertTrue(confirmed.get(10, TimeUnit.SECONDS));
+ assertNull(createdAt("ORDER_5"), "The key must be confirmed");
+ assertTrue(nodeA.getProcessorNameMessageIdSet().isEmpty(), "A
confirmed key must no longer be kept alive");
+ assertFalse(nodeA.add("ORDER_5"));
+ }
+
+ @Test
+ void testKeyBeingRemovedIsStillADuplicateOnItsNodeUntilItsDelete() throws
Exception {
+ assertTrue(nodeA.add("ORDER_6"));
+
+ outage.begin();
+ nodeA.keepAlive();
+ expireLeases();
+ outage.end();
+
+ // the processing of ORDER_6 fails right after the outage: the remove
takes the key out of the keep-alive only
+ // after its DELETE
+ statements.pauseBefore(nodeA.getDeleteString());
+ Future<Boolean> removed = processing.submit(() ->
nodeA.remove("ORDER_6"));
+ assertTrue(statements.awaitPaused(), "The remove did not reach its
DELETE");
+
+ assertFalse(nodeA.add("ORDER_6"), "A key that node A is removing must
not be taken over by node A");
+
+ statements.resume();
+ assertTrue(removed.get(10, TimeUnit.SECONDS));
+ // once removed, a copy is processed again, as the remove asks for
+ assertTrue(nodeA.add("ORDER_6"));
+ }
+
+ @Test
+ void testAnotherNodeTakesOverAKeyWhoseLeaseExpiredDuringTheOutage() {
+ // the limitation of the lease, as documented: node B, which still
reaches the database, cannot tell node A from
+ // a node that crashed, so it takes over a lease that node A could not
refresh for longer than the lock max age
+ assertTrue(nodeA.add("ORDER_3"));
+
+ outage.begin();
+ nodeA.keepAlive();
+ expireLeases();
+ assertTrue(nodeB.add("ORDER_3"));
+ outage.end();
+
+ // node A, which may still process ORDER_3, does not take it a third
time
+ assertFalse(nodeA.add("ORDER_3"));
+ assertFalse(nodeB.add("ORDER_3"));
+ }
+
+ @Test
+ void testKeepAliveLogsOnceThatTheLeasesMayExpire() {
+ RecordingLogger logs = new RecordingLogger();
+ List<String> warnings = logs.warnings();
+ List<String> errors = logs.errors();
+ List<String> infos = logs.infos();
+ // a short lock max age, so that the scheduled keep-alive fails for
longer than it
+ JdbcOrphanLockAwareIdempotentRepository node = newRepository(outage,
500L, 50L);
+ node.log = logs.logger();
+ node.start();
+ try {
+ assertTrue(node.add("ORDER_4"));
+
+ outage.begin();
+ await().atMost(10, TimeUnit.SECONDS).until(() ->
!warnings.isEmpty());
+ assertTrue(warnings.get(0).contains("lockMaxAgeMillis"), () ->
"Unexpected WARN: " + warnings);
+ int failed = errors.size();
+ await().atMost(10, TimeUnit.SECONDS).until(() -> errors.size() >=
failed + 3);
+ assertEquals(1, warnings.size(), () -> "The keep-alive must log
the WARN once per outage: " + warnings);
+
+ outage.end();
+ await().atMost(10, TimeUnit.SECONDS).until(() ->
infos.stream().anyMatch(info -> info.contains("again")));
+ assertFalse(node.add("ORDER_4"));
+ } finally {
+ node.shutdown();
+ }
+ }
+
+ private JdbcOrphanLockAwareIdempotentRepository newRepository(
+ DataSource dataSource, long lockMaxAgeMillis, long
lockKeepAliveIntervalMillis) {
+ JdbcOrphanLockAwareIdempotentRepository repository
+ = new JdbcOrphanLockAwareIdempotentRepository(dataSource,
PROCESSOR_NAME, context);
+ repository.setLockMaxAgeMillis(lockMaxAgeMillis);
+ // with the default lock max age the test calls keepAlive() itself
+ repository.setLockKeepAliveIntervalMillis(lockKeepAliveIntervalMillis);
+ return repository;
+ }
+
+ // the time passes: every lease is older than the lock max age, as no node
could refresh it
+ private Timestamp expireLeases() {
+ Timestamp expired = new Timestamp(System.currentTimeMillis() -
LOCK_MAX_AGE_MILLIS - 60_000L);
+ jdbcTemplate.update("UPDATE CAMEL_MESSAGEPROCESSED SET createdAt = ?
WHERE createdAt IS NOT NULL", expired);
+ return expired;
+ }
+
+ private Timestamp createdAt(String key) {
+ return jdbcTemplate.queryForObject(
+ "SELECT createdAt FROM CAMEL_MESSAGEPROCESSED WHERE
processorName = ? AND messageId = ?", Timestamp.class,
+ PROCESSOR_NAME, key);
+ }
+
+ /**
+ * Holds the next run of a statement before it is sent, until resumed.
+ */
+ private static final class PausingJdbcTemplate extends JdbcTemplate {
+
+ private final CountDownLatch paused = new CountDownLatch(1);
+ private final CountDownLatch resumed = new CountDownLatch(1);
+ private volatile String pausedSql;
+
+ PausingJdbcTemplate(DataSource dataSource) {
+ super(dataSource);
+ }
+
+ void pauseBefore(String sql) {
+ pausedSql = sql;
+ }
+
+ boolean awaitPaused() throws InterruptedException {
+ return paused.await(10, TimeUnit.SECONDS);
+ }
+
+ void resume() {
+ resumed.countDown();
+ }
+
+ @Override
+ public int update(String sql, Object... args) {
+ if (sql.equals(pausedSql)) {
+ pausedSql = null;
+ paused.countDown();
+ try {
+ if (!resumed.await(10, TimeUnit.SECONDS)) {
+ throw new IllegalStateException("The paused statement
was not resumed");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException(e);
+ }
+ }
+ return super.update(sql, args);
+ }
+ }
+
+ /**
+ * The database as node A sees it: unreachable during an outage, so every
statement fails as it cannot get a
+ * connection.
+ */
+ private static final class Outage extends DelegatingDataSource {
+
+ private volatile boolean down;
+
+ Outage(EmbeddedDatabase database) {
+ super(database);
+ }
+
+ void begin() {
+ down = true;
+ }
+
+ void end() {
+ down = false;
+ }
+
+ @Override
+ public Connection getConnection() throws SQLException {
+ if (down) {
+ throw new SQLException("Connection refused: the database is
unreachable");
+ }
+ return super.getConnection();
+ }
+ }
+}
diff --git
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryRestartTest.java
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryRestartTest.java
index dbb40986998e..9e45bb3201b5 100644
---
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryRestartTest.java
+++
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryRestartTest.java
@@ -33,7 +33,6 @@ import
org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType;
import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
@@ -98,12 +97,13 @@ public class
JdbcOrphanLockAwareIdempotentRepositoryRestartTest {
// the keep-alive still refreshes a lease after the restarts
assertTrue(repository.add("FILE_1"));
Timestamp expired = new Timestamp(System.currentTimeMillis() -
LOCK_MAX_AGE_MILLIS - 60_000L);
- new JdbcTemplate(dataSource).update("UPDATE " + TABLE_NAME + " SET
createdAt = ? WHERE messageId = ?", expired,
- "FILE_1");
- assertFalse(repository.contains("FILE_1"));
+ JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
+ jdbcTemplate.update("UPDATE " + TABLE_NAME + " SET createdAt = ? WHERE
messageId = ?", expired, "FILE_1");
repository.keepAlive();
- assertTrue(repository.contains("FILE_1"));
+ Timestamp refreshed = jdbcTemplate.queryForObject("SELECT createdAt
FROM " + TABLE_NAME + " WHERE messageId = ?",
+ Timestamp.class, "FILE_1");
+ assertTrue(refreshed.after(expired), "The keep-alive did not refresh
the lease");
}
@Test
diff --git
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryTest.java
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryTest.java
index 83794ea8df84..ab53d14b19ae 100644
---
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryTest.java
+++
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/JdbcOrphanLockAwareIdempotentRepositoryTest.java
@@ -17,7 +17,6 @@
package org.apache.camel.processor.idempotent.jdbc;
import java.sql.Timestamp;
-import java.util.concurrent.TimeUnit;
import org.apache.camel.impl.DefaultCamelContext;
import
org.apache.camel.processor.idempotent.jdbc.JdbcOrphanLockAwareIdempotentRepository.ProcessorNameAndMessageId;
@@ -30,7 +29,6 @@ import
org.springframework.jdbc.datasource.embedded.EmbeddedDatabase;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType;
-import static org.awaitility.Awaitility.await;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -72,7 +70,7 @@ public class JdbcOrphanLockAwareIdempotentRepositoryTest {
}
@Test
- public void testLockKeepAliveWorks() {
+ public void testLockKeepAliveWorks() throws Exception {
assertFalse(jdbcMessageIdRepository.contains("FILE_4"));
jdbcMessageIdRepository.insert("FILE_4");
assertTrue(jdbcMessageIdRepository.contains("FILE_4"));
@@ -81,9 +79,18 @@ public class JdbcOrphanLockAwareIdempotentRepositoryTest {
template.update("UPDATE CAMEL_MESSAGEPROCESSED SET createdAT = ? WHERE
processorName = ? AND messageId = ?", timestamp,
APP_NAME, "FILE_4");
- await().atMost(5, TimeUnit.SECONDS).until(() ->
!jdbcMessageIdRepository.contains("FILE_4"));
- jdbcMessageIdRepository.keepAlive();
+ // the lease is older than the lock max age: another instance takes it
for an orphan, while this instance, which
+ // holds the key, does not
+ JdbcOrphanLockAwareIdempotentRepository otherInstance
+ = new JdbcOrphanLockAwareIdempotentRepository(dataSource,
APP_NAME, new DefaultCamelContext());
+ otherInstance.setLockMaxAgeMillis(3000_00L);
+ otherInstance.setLockKeepAliveIntervalMillis(3000L);
+ otherInstance.doInit();
+ assertFalse(otherInstance.contains("FILE_4"));
assertTrue(jdbcMessageIdRepository.contains("FILE_4"));
+
+ jdbcMessageIdRepository.keepAlive();
+ assertTrue(otherInstance.contains("FILE_4"));
}
@Test
diff --git
a/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/RecordingLogger.java
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/RecordingLogger.java
new file mode 100644
index 000000000000..62471fe04cbe
--- /dev/null
+++
b/components/camel-sql/src/test/java/org/apache/camel/processor/idempotent/jdbc/RecordingLogger.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.processor.idempotent.jdbc;
+
+import java.lang.reflect.Proxy;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+
+import org.slf4j.Logger;
+
+/**
+ * A logger for the tests of the repositories: it records the format of the
WARN, ERROR and INFO messages, and drops the
+ * others.
+ */
+final class RecordingLogger {
+
+ private final List<String> warnings = new CopyOnWriteArrayList<>();
+ private final List<String> errors = new CopyOnWriteArrayList<>();
+ private final List<String> infos = new CopyOnWriteArrayList<>();
+ private final Logger logger = (Logger)
Proxy.newProxyInstance(Logger.class.getClassLoader(),
+ new Class<?>[] { Logger.class },
+ (proxy, method, args) -> {
+ switch (method.getName()) {
+ case "hashCode":
+ return System.identityHashCode(proxy);
+ case "equals":
+ return proxy == args[0];
+ case "toString":
+ return "recordingLogger";
+ case "warn":
+ record(warnings, args);
+ return null;
+ case "error":
+ record(errors, args);
+ return null;
+ case "info":
+ record(infos, args);
+ return null;
+ default:
+ break;
+ }
+ if (method.getReturnType() == boolean.class) {
+ return false;
+ }
+ if (method.getReturnType() == String.class) {
+ return "test";
+ }
+ return null;
+ });
+
+ Logger logger() {
+ return logger;
+ }
+
+ List<String> warnings() {
+ return warnings;
+ }
+
+ List<String> errors() {
+ return errors;
+ }
+
+ List<String> infos() {
+ return infos;
+ }
+
+ private static void record(List<String> messages, Object[] args) {
+ if (args != null && args.length > 0) {
+ messages.add(String.valueOf(args[0]));
+ }
+ }
+}