[
https://issues.apache.org/jira/browse/SLING-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Schneider updated SLING-13395:
----------------------------------------
Description:
On a clustered instance, OnlyOnLeader unregisters its service on every
TOPOLOGY_CHANGING event and registers it again on TOPOLOGY_CHANGED.
DistributionSubscriber has a static reference to OnlyOnLeader, so journal
subscribers are deactivated and activated again even when the local instance
remains leader.
An in-flight package import can continue after deactivation. A replacement
subscriber starts at the last committed offset + 1 and can import the same
package concurrently, causing duplicate processing and conflicting BookKeeper
updates.
Keep OnlyOnLeader registered during TOPOLOGY_CHANGING. On
TOPOLOGY_INIT/TOPOLOGY_CHANGED, register if the local instance is leader and
unregister otherwise. Add regression coverage verifying that an unchanged
leader retains the same service registration.
This change avoids restarts caused by topology changes with unchanged
leadership. It does not prevent duplicate imports during actual subscriber
deactivation or leadership handover.
was:
On a clustered instance, every topology change restarts all journal subscribers
on the leader, even when the leader does not change. If a package import is
still running at that moment, the restarted subscriber imports the same package
again in parallel.
How it happens:
* OnlyOnLeader unregisters its service on every TOPOLOGY_CHANGING event and
registers it again on TOPOLOGY_CHANGED. DistributionSubscriber has a static
reference to OnlyOnLeader (SLING-12639), so it is deactivated and activated
again on every topology change.
* DistributionSubscriber.deactivate() does not interrupt the poller thread
(SLING-9340) and does not wait for it, so an import that is already running
continues.
* The new subscriber instance starts at the last committed offset + 1, which is
the offset still being imported by the old thread. The same package is then
imported twice in parallel.
* Both imports write the BookKeeper state for the same subscriber, so one of
them can fail with an OakState0001 conflict on the subscriber store.
The effect is significant when imports are slow (for example, an import that
takes several minutes because of an expensive package builder) and the topology
changes frequently (for example, instances joining and leaving the cluster).
Several overlapping imports of the same offset were observed, each running for
minutes.
Proposed fix:
* OnlyOnLeader: don't unregister on TOPOLOGY_CHANGING; only update the
registration on TOPOLOGY_INIT/TOPOLOGY_CHANGED, and unregister only when the
local instance is no longer the leader.
* DistributionSubscriber.deactivate(): wake up a pending retry back-off and
wait, bounded (30 s), for an import that is still running before returning. No
locking between subscriber instances, to avoid hanging processing; a duplicate
import remains possible when an import takes longer than the wait or when
leadership really changes.
Summary: Keep journal subscribers running when topology changes without
a leadership change (was: DistributionSubscriber imports the same package
twice when it is restarted during a topology change)
> Keep journal subscribers running when topology changes without a leadership
> change
> ----------------------------------------------------------------------------------
>
> Key: SLING-13395
> URL: https://issues.apache.org/jira/browse/SLING-13395
> Project: Sling
> Issue Type: Bug
> Components: Content Distribution
> Reporter: Christian Schneider
> Assignee: Christian Schneider
> Priority: Major
> Fix For: Content Distribution Journal Core 0.6.0
>
>
> On a clustered instance, OnlyOnLeader unregisters its service on every
> TOPOLOGY_CHANGING event and registers it again on TOPOLOGY_CHANGED.
> DistributionSubscriber has a static reference to OnlyOnLeader, so journal
> subscribers are deactivated and activated again even when the local instance
> remains leader.
> An in-flight package import can continue after deactivation. A replacement
> subscriber starts at the last committed offset + 1 and can import the same
> package concurrently, causing duplicate processing and conflicting BookKeeper
> updates.
> Keep OnlyOnLeader registered during TOPOLOGY_CHANGING. On
> TOPOLOGY_INIT/TOPOLOGY_CHANGED, register if the local instance is leader and
> unregister otherwise. Add regression coverage verifying that an unchanged
> leader retains the same service registration.
> This change avoids restarts caused by topology changes with unchanged
> leadership. It does not prevent duplicate imports during actual subscriber
> deactivation or leadership handover.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
