Daniel Iancu created SLING-13259:
------------------------------------
Summary: OakDiscoveryService can block framework shutdown when Oak
CommitQueue is slow during PropertyProvider unbind
Key: SLING-13259
URL: https://issues.apache.org/jira/browse/SLING-13259
Project: Sling
Issue Type: Bug
Components: Discovery
Reporter: Daniel Iancu
h2. Description
h3. What happens
During OSGi framework shutdown, the Felix start-level thread can block for many
minutes — potentially indefinitely — inside
{{OakDiscoveryService.doUpdateProperties()}}, waiting for a JCR commit that
never completes. The stuck thread prevents the framework from stopping any
further bundles, so the JVM only exits when forcibly killed by the surrounding
process manager. During that window the instance is half-dead but still visible
to the rest of the cluster, delaying topology convergence on the surviving
members.
h3. Stack trace
{noformat} FelixStartLevel at jdk.internal.misc.Unsafe.park (Native Method) -
parking to wait for <0x…> (a java.util.concurrent.CountDownLatch
Entry.await (CommitQueue.java:346) at
org.apache.jackrabbit.oak.plugins.document.CommitQueue.waitUntilHeadOfQueue
(CommitQueue.java:285) … at
org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit at
org.apache.sling.resourceresolver.impl.ResourceResolverImpl.commit at
org.apache.sling.discovery.oak.OakDiscoveryService.doUpdateProperties
(OakDiscoveryService.java:550) at
org.apache.sling.discovery.oak.OakDiscoveryService.bindPropertyProviderInteral
at org.apache.sling.discovery.oak.OakDiscoveryService.updatedPropertyProvider
{noformat}
h3. Root cause
{{OakDiscoveryService}} declares {{PropertyProvider}} as a {{MULTIPLE}},
{{DYNAMIC}} reference. Every bind / unbind / update callback triggers a JCR
write via {{doUpdateProperties()}} → {{ResourceResolver.commit()}} in order to
keep the properties node at {{//properties}} in sync with the currently
registered providers.
At shutdown, application bundles (higher start levels) typically stop before
infrastructure bundles like {{discovery.oak}} (low start level). When an
application bundle stops, its {{PropertyProvider}} services are unregistered,
and Declarative Services invokes {{unbindPropertyProvider}} /
{{updatedPropertyProvider}} on {{OakDiscoveryService}} synchronously on the
Felix start-level thread. That callback then issues an Oak commit.
If the Oak {{CommitQueue}} is slow at that moment — a pending entry ahead of
us, repository under load, backlog on the cluster leader — the commit parks on
a {{CountDownLatch}} and does not return. The Felix start-level thread is now
stuck inside a callback for a bundle it hasn't finished stopping. It cannot
advance to stop any further bundles, and in particular it cannot reach
{{discovery.oak}}'s own bundle to trigger {{OakDiscoveryService.deactivate()}}.
Shutdown stalls entirely.
h3. Why the existing {{activated}} guard doesn't help
{{OakDiscoveryService.activated}} is set to {{false}} inside {{deactivate()}},
several lines into the method, under {{viewStateManagerLock}}. In the scenario
above {{deactivate()}} is never reached, so {{activated}} remains {{true}} and
every unbind callback proceeds into {{doUpdateProperties()}} and into
{{commit()}}.
Even in cases where {{deactivate()}} does run, the {{activated}} flag is
written under a different lock ({{viewStateManagerLock}}) from the one guarding
bind/unbind ({{lock}}), so the two are not mutually observable without racing.
h3. Why the property update is unnecessary at shutdown
Once this instance is shutting down, rewriting the properties node has no value
— no other cluster member will meaningfully act on the updated contents before
this instance is removed from the topology anyway. The write is pure overhead,
and during shutdown it is actively harmful because it holds the framework
thread.
h2. Impact
Framework shutdown can hang for the full duration of the surrounding process
manager's grace period before the JVM is forcibly killed.
The stalled instance remains visible in the discovery topology throughout,
delaying leader re-election and topology convergence for the rest of the
cluster.
Any operational tooling that assumes clean JVM exit (health checks, rolling
restarts, orchestrated deployments) is affected.
h2. Steps to reproduce
Start a Sling instance with {{discovery.oak}} active and at least one
{{PropertyProvider}} implementation registered by a bundle at a higher start
level than {{discovery.oak}}.
Induce Oak {{CommitQueue}} slowness — for example by holding an open write
session on another thread, or by loading the repository with concurrent writers
that outpace the commit rate.
Trigger a clean framework shutdown (e.g. {{bin/stop}} or SIGTERM).
Observe: shutdown does not complete. A thread dump shows {{FelixStartLevel}}
parked in {{CommitQueue$Entry.await}} under {{doUpdateProperties}}. The JVM
only exits on forcible termination.
h2. Expected behaviour
Once the framework has begun shutting down, {{OakDiscoveryService}} should skip
all property-update writes and return immediately from
{{doUpdateProperties()}}. Framework shutdown should proceed at its normal pace
regardless of the state of the Oak {{CommitQueue}}.
h2. Proposed fix (summary)
Introduce a {{volatile boolean deactivating}} flag that short-circuits
{{doUpdateProperties()}} once shutdown has begun. The flag must be set before
the first blocking commit can start, which means it cannot rely solely on
{{@Deactivate}} being invoked — because in the observed scenario
{{deactivate()}} is never reached. The trigger should be one of:
A {{FrameworkListener}} on {{FrameworkEvent.STOPPING}}, registered in
{{@Activate}}, that sets the flag as soon as the framework begins shutting
down. or
An inline check of {{bundleContext.getBundle(0).getState() >= Bundle.STOPPING}}
at the top of {{doUpdateProperties()}}.
The flag is also set at the first line of {{deactivate()}} as a belt-and-braces
measure for the case where {{discovery.oak}} is deactivated independently of a
full framework stop.
The fix must not interrupt an already-in-progress commit — that risks
corrupting the Oak session. It only prevents new commits from starting once
shutdown has begun.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
