shashank created CAMEL-25857:
--------------------------------
Summary: camel-kubernetes - when the first watch of a watch
consumer cannot be created (API server unreachable, 403), the consumer never
watches, nothing is logged, and the route stays started
Key: CAMEL-25857
URL: https://issues.apache.org/jira/browse/CAMEL-25857
Project: Camel
Issue Type: Bug
Components: camel-kubernetes
Reporter: shashank
Assignee: shashank
The watch consumers of camel-kubernetes ({{kubernetes-pods}}, {{-services}},
{{-deployments}}, {{-config-maps}}, {{-custom-resources}}, {{-events}},
{{-hpa}}, {{-namespaces}}, {{-nodes}}, {{-replication-controllers}} and
{{openshift-deploymentconfigs}}) submit their watch task to their executor
once, in {{doStart}} (main 20cdb309f7d0, for example {{KubernetesPodsConsumer}}
line 63):
{code:java}
podsWatcher = new PodsConsumerTask();
executor.submit(podsWatcher);
{code}
The task calls the fabric8 {{watch(...)}}, which throws a
{{KubernetesClientException}} when the watch cannot be created: the API server
cannot be reached, or answers 403 because the RBAC rules of the service account
are not applied yet (a Role/RoleBinding applied after the Deployment), or 5xx
during an API server upgrade. The exception ends the task inside the {{Future}}
returned by {{submit}}, which nobody reads. So nothing is logged, the consumer
never tries again, and the route stays started without receiving any event,
until the route or the application is restarted.
CAMEL-25273 made the consumers watch again when the client closes a watch with
an error, with a backoff when watching again fails
({{KubernetesHelper.watchAgain}}); its description lists the failed first watch
as not in scope. The first watch is the only path left without a retry.
*Fix*: a new {{KubernetesHelper.watch(consumer, executor, task)}} runs the
first watch on the executor of the consumer; on failure it logs a WARN ("Failed
to watch for <consumer>, trying again in 1000 ms") and tries again with the
backoff of {{watchAgain}} (1 s, doubling up to 30 s) until the watch is created
or the consumer is stopped. The 11 consumers call it instead of
{{executor.submit(task)}}.
*Tests*: new {{KubernetesPodsConsumerFirstWatchFailsTest}} (fabric8 mock
server, as {{KubernetesPodsConsumerWatchClosedTest}}): the first watch request
gets 403, the second one an {{ADDED}} pod. On main it fails with
{{mock://result Received message count. Expected: <1> but was: <0>}}, and the
test log has no line about the 403. With the fix it passes. camel-kubernetes
unit tests with the fix: 180 run, 0 failures, 6 skipped.
*How it was found*: a review of consumers that run their receive loop or watch
on their own thread, looking for an exception that ends it while the route
stays started; then the JUnit test above. Not run against a real cluster.
*Affected*: main and camel-4.23.x (with CAMEL-25273); camel-4.22.x,
camel-4.18.x and camel-4.14.x have the same {{executor.submit}} of the first
watch, and there a watch closed with an error is not retried either
(CAMEL-25273 is fixed in 4.23.0 only) (read, not run).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
