void-ptr974 opened a new pull request, #25793:
URL: https://github.com/apache/pulsar/pull/25793
### Motivation
Automatic managed ledger offload can be triggered repeatedly while a
previous offload is still running, for example around ledger rollover or topic
load.
Before this change, every automatic trigger could independently enter the
offload path:
1. read offload policies
2. scan the managed ledger's ledger list
3. try to acquire the offload mutex
4. fail because another offload is already running
5. schedule another retry after 100ms
When offload is slow, or when a managed ledger has many ledgers, repeated
automatic triggers can build up unnecessary scheduler/executor work. These
retries do not improve the final offload
result because only one offload can run at a time.
### Modifications
This change coalesces automatic offload triggers in `ManagedLedgerImpl`.
After this change:
- there is at most one in-flight automatic offload
- repeated automatic triggers during an in-flight offload are merged into
one pending rerun
- after the current automatic offload completes, one follow-up pass runs
if any trigger arrived meanwhile
- explicit/manual offload requests keep their existing
`CompletableFuture<Position>` behavior
- the automatic offload sentinel is renamed to make it clear that its
`Position` value is not consumed
- duplicate `getOffloadPolicies()` calls are avoided in the appendable
offloader policy lookup path
### Impact
The final automatic offload progression is preserved: if new ledgers
become eligible while an offload is running, the follow-up pass still picks
them up.
The main benefit is reducing unnecessary work under slow offload or
large-ledger workloads:
- avoids repeated 100ms automatic offload retry loops
- reduces redundant offload policy reads
- reduces redundant ledger-list scans
- lowers scheduler and executor pressure while offload is already in
progress
- keeps explicit/manual offload behavior unchanged
In practice, repeated automatic triggers while one offload is running are
reduced from many independent retry loops to one active run plus one pending
rerun.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Added a test that repeated automatic triggers during an in-flight
offload do not create independent retry loops.
- Added a test that a coalesced automatic trigger causes one follow-up
offload pass.
- Added a test that automatic offload state is released when offload
thresholds are disabled, so later valid triggers can still run.
Local verification:
```bash
git diff --check
./gradlew :managed-ledger:test --tests
org.apache.bookkeeper.mledger.impl.OffloadPrefixTest
./gradlew :managed-ledger:test --tests
org.apache.bookkeeper.mledger.impl.OffloadLedgerDeleteTest
### Does this pull request potentially affect one of the following parts:
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
This only changes the internal scheduling behavior of automatic managed
ledger offload. Public APIs, configs, metrics, explicit/manual offload
behavior, and offload metadata semantics are
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
