shashank created CAMEL-25614:
--------------------------------
Summary: camel-kubernetes - with
camel.vault.kubernetes.refreshEnabled=true the CamelContext is no longer
reloaded on a secret (config map) update after the watch was closed once (410
Gone); nothing is logged
Key: CAMEL-25614
URL: https://issues.apache.org/jira/browse/CAMEL-25614
Project: Camel
Issue Type: Bug
Components: camel-kubernetes
Reporter: shashank
Assignee: shashank
With {{camel.vault.kubernetes.refreshEnabled=true}}
({{camel.vault.kubernetescm.refreshEnabled=true}} for config maps) camel-main
schedules {{SecretsReloadTriggerTask}} ({{ConfigmapsReloadTriggerTask}}) with
{{PeriodTaskScheduler.scheduledTask}} ({{DefaultConfigurationConfigurer}} 867 /
886 at main a7169d7c6ab6). That method submits the task once:
{{DefaultPeriodTaskScheduler.TaskWrapper.onTimer}} (93-98)
{code:java}
// submit task only once as the task can potentially keep running (until camel
is stopped)
if (running == null) {
running = executorService.submit(task);
}
{code}
{{SecretsReloadTriggerTask.run}} (117-156; {{ConfigmapsReloadTriggerTask.run}}
120-159 is the same) creates one watch, waits for {{onClose(WatcherException)}}
and returns:
{code:java}
@Override
public void onClose(WatcherException e) {
isWatchClosed.countDown();
}
...
// Wait till watch gets closed
isWatchClosed.await();
...
watch.close();
{code}
The Kubernetes client reconnects a watch by itself after transient errors, but
it closes the watch for good and calls {{onClose(WatcherException)}} when the
API server answers 410 Gone (the resource version of the watch is too old). How
often that happens depends on the cluster: the watch asks for bookmarks, which
keep its resource version fresh, so it is an occasional event and not a regular
one. It is the same event that CAMEL-25273 fixed for the consumers of the
component; the two reload tasks were not part of it, and for them one such
close is enough.
After the first such close the task is finished and is never submitted again.
The application keeps running with the secret values it has; a rotated secret
is not followed by a reload, and nothing is logged, neither when the watch is
closed nor later. A watch that cannot be created ({{watch(...)}} throws: API
server not reachable at that moment, request refused) ends the task the same
way; the exception stays in the {{Future}} of the submit, which nobody reads,
so there is no log line either.
h3. Reproduction
New {{SecretsReloadTriggerTaskWatchClosedTest}} and
{{ConfigmapsReloadTriggerTaskWatchClosedTest}} with the Kubernetes mock server
that the module uses for its other tests. The task is scheduled the way
camel-main does it. The first watch request is answered with a watch event
{{ERROR}} / 410; the next one with a {{MODIFIED}} event for the configured
secret (config map). In a second test one more watch request in between is
answered with 403. A {{ContextReloadStrategy}} counts the reloads. On main
these four tests fail after 20 seconds with {{the update of the secret after
the watch was closed did not trigger a reload ==> expected: <true> but was:
<false>}}. A third test per class stops the context while the task has a watch
and expects the thread of the task to end; on main: {{the stopped task still
waits for its watch to be closed ==> expected: <false> but was: <true>}} (on
main a stopped task keeps its thread waiting for the watch).
h3. Proposed fix
{{run()}} of both tasks watches again until the task is stopped: one second
after a closed watch, and with a delay that doubles up to 30 seconds while the
watch cannot be created (one second at start, two seconds for the first failed
attempt after a closed watch; the delays of {{KubernetesHelper.watchAgain}},
which CAMEL-25273 added for the consumers). Both cases are logged with one WARN
line, without a stack trace (the stack trace of a watch that cannot be created
is logged at DEBUG). {{doStop}} of the task ends the wait, so a stopped task
does not keep its thread.
Updates made while nothing watches (between the close and the next watch) are
still not seen; the task does not list the secrets after it watched again.
camel-kubernetes tests with the fix: 179 run, 0 failures, 6 skipped.
How it was found: a sweep of the reconnect / re-watch code of the components
after CAMEL-25273, and a small TLA+ model of the task (submitted once, watch
created or refused, 410, updates). For main TLC reports that the task ends
after one 410 or one refused watch request (invariant "the task never ends
while it is not stopped", traces of 4 and 3 states) and that "in the end the
secrets are watched" does not hold; with the fix both hold for two 410 answers
and two refused requests (144 states).
Affected: main (a7169d7c6ab6; tested). camel-4.23.x, camel-4.22.x, camel-4.18.x
and camel-4.14.x have the same {{run()}} in both tasks and the same
{{TaskWrapper.onTimer}} (read, not run).
Not looked at: the other vault refresh tasks that are scheduled with
{{scheduledTask}}.
Duplicate check (2026-10-13): JIRA text "SecretsReloadTriggerTask"
(CAMEL-24636, CAMEL-21924, CAMEL-21771, CAMEL-21770, CAMEL-21436: none about
the watch), "kubernetes secret refresh watch", "kubernetes vault refresh watch
closed" (CAMEL-24636 only); GitHub pull requests "SecretsReloadTriggerTask"
(none), "kubernetes secret refresh watch" (nothing related).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
