[
https://issues.apache.org/jira/browse/CAMEL-26577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26577 started by shashank.
----------------------------------------
> camel-core - RebalancingCamelClusterService: every rebalancing pass takes a
> reference on the views and never releases it, so the view of a removed
> clustered route keeps the leadership and a released view is started again
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26577
> URL: https://issues.apache.org/jira/browse/CAMEL-26577
> Project: Camel
> Issue Type: Bug
> Components: camel-core, camel-kubernetes
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> {{RebalancingCamelClusterService}} (camel-support) wraps a
> {{CamelPreemptiveClusterService}} (the Kubernetes cluster service) and runs
> {{reconcile()}} on its own scheduled thread every period. Each pass looks at
> every namespace through {{delegate.getView(namespace)}}, in {{members()}},
> {{owned()}} and {{setDisabled()}}, several times per pass, and never calls
> {{releaseView}}.
> {{AbstractCamelClusterService.getView}} is not a plain lookup: it takes a
> reference on the view (and starts it on the first one), and {{releaseView}}
> stops the view when the last reference is released
> ({{CamelClusterService.releaseView}}: "Release a view if it has no
> references"; CAMEL-25089 made sure a released view is not started again by a
> restart of the service). So with the rebalancing service:
> * a namespace whose clustered route is removed keeps its view running: the
> {{ClusteredRoutePolicy}} releases its reference, but the rebalancer holds one
> more per pass. This node stays in the election, and when it is the leader of
> that namespace no other node runs the route (the rebalancer even counts it as
> one of its partitions);
> * a view that was already released (stopped) is started again by the next
> pass, as its {{getView}} takes the first reference again;
> * {{members()}} then also compares the members of views nobody uses.
> Found with a TLA+ model of the reference count of one namespace with three
> actors (the route policy retaining at onInit and releasing at onRemove, the
> rebalancing thread, the service stop/start). Property: once the last route
> using the namespace is removed and the rebalancer is between passes, the view
> is stopped and holds no reference. Short trace on main: route added (count 1,
> view started) -> route removed (count 0, view stopped) -> reconcile pass
> {{getView}} (count 1, view started again) -> pass ends, nothing released ->
> the view keeps running with no user. The same with the pass before the
> removal: the count never drops to 0. The fixed model (look only at views in
> use, release every reference taken) holds; both paths are reachable
> (witnesses).
> Reproduced on the real classes with {{DefaultCamelContext}}, a
> {{ClusteredRoutePolicy}} route and a test {{CamelPreemptiveClusterService}}
> extending {{AbstractCamelClusterService}} (no Kubernetes): after
> {{removeRoute}} the view is still {{Started}}; a view released through the
> API is {{Started}} again after one {{reconcile()}}. Both tests are
> deterministic (the passes are run explicitly, the period is one hour).
> Proposed fix: the rebalancer takes a reference only on the views that are in
> use (a package-private lookup of {{AbstractCamelClusterService}} that does
> not create or start a view) and releases each one in a {{finally}};
> namespaces with no user are not partitions to balance. Delegates that do not
> extend {{AbstractCamelClusterService}} get {{getView}}/{{releaseView}} pairs.
> Not part of this issue (minor): {{doStop}} does not wait for a pass in
> flight, which reschedules itself on the {{serializedExecutor}} field, so
> after a quick stop/start one more chain of passes runs on the new executor.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
