joewitt commented on PR #11164:
URL: https://github.com/apache/nifi/pull/11164#issuecomment-5648885075
Saved at `/tmp/NIFI-15862-1000-pair-fairness.md`. Copy below:
```markdown
# Virtual-thread scheduling: 1,000-pair fairness run
Follow-on to the earlier NIFI-15862 notes. Same branch
(`nifi-15862-virtual-threads`), same machine, same “do not cheat provenance”
rule. This run is about whether the **global permit cap** still gives every
component a fair turn when the canvas is far larger than the cap — including
mixed scheduling periods.
## Setup
- **NiFi:** 2.12.0-SNAPSHOT, `nifi.scheduling.strategy=AUTO` → virtual
threads (Java 25)
- **JDK:** Azul Zulu 25, macOS, 16 cores, heap 16g
- **Repos:** `VolatileFlowFileRepository` + `VolatileProvenanceRepository`
(CPU/scheduling, not disk)
- **Controller cap:** Maximum Timer Driven Thread Count = **128** (global
`DynamicSemaphore` permits, not a platform pool)
- **Flow:** **1,000 independent pairs** = **2,000 processors**, **1,000
connections**
- GenerateFlowFile → UpdateAttribute
- 0-byte FlowFiles, Generate **batch 50**
- **1 concurrent task** on every processor
- UpdateAttribute `benchmark=true`, run duration 25 ms, `success`
auto-terminated
- connection backpressure 15,000 / 1 GB
- layout: 5 pairs across, Generate above UpdateAttribute, same spacing as
the original three pairs
Built up in stages (3 → 20 → 220 → 1,000) with the same pair shape so we
could see fairness at each scale.
## Scheduling mix (all 1,000 pairs)
Generate and UpdateAttribute in a pair share the same period. Assignment is
`n % 10` so cadences are interleaved across the canvas, not parked in one
corner.
| Period | Pairs | Share |
|---|---|---|
| 0 sec (as fast as possible) | 100 | 10% |
| 50 ms | 100 | 10% |
| 100 ms | 100 | 10% |
| 500 ms | 100 | 10% |
| 750 ms | 100 | 10% |
| 1 sec | 500 | 50% |
**2,000** scheduled tasks vs **128** permits (~15× oversubscribed).
## Findings
### Uniform pairs (before mixing periods)
At 1+1 concurrent tasks, batch 50, all 0 sec:
| Scale | Combined Generate FF/s | Per-pair | max/min | Notes |
|---|---|---|---|---|
| 20 pairs | ~549k | ~27.2k–27.8k | 1.02 | queues often at 15k; Generate
yielded; **not** permit-bound (64 cap unused) |
| 220 pairs | ~630k | 2,858–2,865 | **1.002** | cap **pegged** (~127/128);
1–20 vs 21–220 same mean |
| 1,000 pairs | ~610k | 609.3–610.5 | **1.002** | stdev **0.19 FF/s**;
slowest **99.9%** of equal share |
Raising the cap **64 → 128** did **not** double throughput (still ~600k). It
let more tasks sit in `onTrigger` at once. Limit is **16 cores + engine work**,
not a few pairs hogging the semaphore.
30s deltas of NiFi’s 5-minute counters will go negative when that window is
sliding. Compare pairs with **5-minute count / 300**, or treat a uniform delta
as “everyone moved together.” Generate **out** vs UpdateAttribute **in** also
looks “starved vs busy” in the UI; that is processor type, not unfairness.
### Mixed periods (1,000 pairs)
5-minute averages, ~6 minutes after applying the mix. All 2,000 processors
Running. Combined **~600k FF/s**. Generate FF/s ≈ tasks/s × 50 (batch size is
real).
| Period | Pairs | Expected Generate Hz | Actual Hz | Of expected | Actual
FF/s / pair | Expected FF/s |
|---|---|---|---|---|---|---|
| 0 sec | 100 | max | **86.6** | — | **4,329** | — |
| 50 ms | 100 | 20 | **16.3** | **81%** | 813 | 1,000 |
| 100 ms | 100 | 10 | **9.0** | **90%** | 449 | 500 |
| 500 ms | 100 | 2 | **1.96** | **98%** | 98 | 100 |
| 750 ms | 100 | 1.33 | **1.32** | **99%** | 66 | 67 |
| 1 sec | 500 | 1 | **0.99** | **99%** | 50 | 50 |
Within every bucket, max/min ≈ **1.00**. **Zero** of 500 one-second
Generates were below 0.5 Hz or above 1.5 Hz.
Interpretation:
- **Slow pairs are not starved off the board.** 1 s / 750 ms / 500 ms hit
their deadlines.
- **50 ms / 100 ms miss some deadlines** because they compete with 100
always-runnable 0-sec pairs.
- Those 0-sec pairs are 10% of pairs (~10% of processors) but held **~97 of
~133** active tasks and **~72%** of FlowFiles/s (433k of 600k). That is
expected: the fair permit semaphore gives turns to whoever is runnable; 0-sec
is always runnable.
- UpdateAttribute **task** counts on slow pairs are **not** a schedule
check. Empty queue uses **bored yield (~10 ms)**, not the 1 s period, so UA
tasks include no-work polls. **FlowFiles in** still tracks Generate.
## What this says about VT readiness
1. **The scheduler scales past the cap without collapsing into winners and
losers.** 2,000 looping virtual threads vs 128 permits stayed even to three
decimals on FlowFiles/s.
2. **Max Timer Driven Thread Count still means something** — it is a global
concurrency throttle, not “create 128 platform threads.” Oversubscribe it and
combined FF/s stays in the same ~600k band; each pair just gets a thinner slice.
3. **Timer-driven periods still work** under that oversubscription.
One-second components kept ~1 Hz. The miss rate on 50–100 ms is permit
competition with 0-sec work, the same class of effect you would get from a
small platform pool — not VT-specific unfairness.
4. **Yield / no-work is a sleep**, not cancel-and-resubmit. Mixed cadences
did not require future juggling. Stop/start of 2,000 processors was routine.
5. **We did not see pinning** on this path in earlier JFR; this 1,000-pair
run did not re-profile, but behavior is consistent with “park on permit, run
invoke, release.”
6. Combined throughput did **not** grow with pair count (20 → 220 → 1,000).
That is **cores + engine** (queues, maps, provenance), not evidence the VT
agent failed. Two stacked pairs earlier **did** add; that was the VT win. This
run is the fairness/oversubscription win.
## Caveats (same as before)
- One Mac, 16 cores, volatile repos, 0-byte UpdateAttribute loop.
- Not a WAL / persistent provenance / real content soak.
- Not a head-to-head TIMER_DRIVEN vs AUTO A/B at 1,000 pairs.
- Default `AUTO` remains the right upgrade default (VT on Java 25+, platform
pool on 21–24). `STANDARD` stays the escape hatch.
**Bottom line:** with 1,000 pairs, mixed periods, and a 128-permit cap,
virtual-thread scheduling looked **ready** for the property it was meant to
have — many concurrent tasks, a simple global bound, and a fair shake for slow
timers rather than a few 0-sec processors owning the engine.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
