Vivek1106-04 commented on issue #12339:
URL: https://github.com/apache/seatunnel/issues/12339#issuecomment-5760116122
Benchmark results, as promised. The headline is that the shared scheduler is
better where it was designed to help and indistinguishable elsewhere, but the
run also turned up a cliff I had not looked for, and it corrects something I
said earlier in this thread.
## How this was measured
`CheckpointSchedulingBenchmark` from #12418, with a second arm added locally
for
the #12165 scheduler. Both arms ran in the **same JVM invocation on the same
branch**, so this is a paired comparison rather than two runs stitched
together.
The per-pipeline arm reproduces `dev`'s construction directly -
`newScheduledThreadPool(2, ...)` per `(jobId, pipelineId)`, same thread
names,
same `setRemoveOnCancelPolicy(true)` - so it does not depend on branch code
and
is valid on either base.
`Mode.SampleTime`, 3 forks, 3 warmup and 5 measurement iterations, JDK 17,
`-Xms4g -Xmx4g -XX:ActiveProcessorCount=4`, `checkpointIntervalMillis=1000`.
All figures us/op.
## Pipeline count sweep
| P | body us | per-pipeline p50 | shared p50 | per-pipeline p99 | shared
p99 | mean ratio |
|---|---|---|---|---|---|---|
| 1 | 0 | 7.12 | 10.74 | 11.82 | 15.33 | 0.66x |
| 1 | 500 | 7.16 | 10.50 | 10.45 | 15.41 | 0.67x |
| 10 | 0 | 7.42 | 9.90 | 10.78 | 13.49 | 0.75x |
| 10 | 500 | 7.78 | 9.90 | 11.87 | 13.66 | 0.80x |
| 100 | 0 | 9.62 | 9.95 | 15.74 | 14.42 | 0.97x |
| 100 | 500 | 9.66 | 9.90 | 16.61 | 13.41 | 1.00x |
| 500 | 0 | 45.70 | 9.95 | 81.15 | 14.83 | 4.61x |
| 500 | 500 | 45.50 | 9.90 | 78.85 | 13.82 | 4.63x |
Ratio above 1 means the shared scheduler is faster. Errors on all means are
between 0.002 and 0.061 us, so the differences are far outside the noise.
Three things this says:
1. **The shared curve is flat.** About 10 us p50 from 1 pipeline to 500. The
per-pipeline curve has a knee between 100 and 500 and reaches 45.7 us.
2. **Crossover is at P=100**, where the two are level. Below that the shared
scheduler is *worse* by roughly 3 us, which is the extra
timer-to-dispatcher
hop. Against the default `checkpoint.interval` of 300000 ms that is three
microseconds added to a five minute timer. Real, measurable, and of no
operational consequence - but it is a regression and I would rather state
it
than round it away.
3. **Thread count behaves as the design claims.** 509 live scheduler threads
at
P=500 for the per-pipeline model, against at most 10 by construction for
the
shared one (2 timer plus a dispatch bound of `max(8, cores * 2)` = 8
here),
independent of P.
The `max` column is omitted deliberately: it is a single sample out of
several
million and is noisy in both arms (per-pipeline maxima ranged 1.2-13 ms, and
one
shared sample at P=1 hit 63 ms). It is not a comparable statistic here.
## A correction, and the cliff
In my first pass I concluded that `triggerBodyMicros` moved nothing in either
arm and read that as evidence the dispatch pool is never the constraint.
**That
conclusion was wrong, and I am glad I checked it before arguing from it.**
Every configuration above sits at about 3% dispatch utilization:
```
P=500, interval=1000ms, body=500us
-> 500 triggers/s x 500us = 0.25 thread-seconds/s against 8 threads = 3%
```
Body occupancy could not have mattered at that load. Saturating 8 dispatch
threads at P=500 needs `body >= 16 ms`. So I ran the regime that actually
matters:
| body us | dispatch util | arm | mean | p50 | p99 |
|---|---|---|---|---|---|
| 500 | 3% | per-pipeline | 46.37 | 45.50 | 78.85 |
| 500 | 3% | shared | 10.02 | 9.90 | 13.82 |
| 5000 | 31% | per-pipeline | 50.84 | 51.07 | 83.84 |
| 5000 | 31% | shared | 9.45 | 9.33 | 13.28 |
| 20000 | 125% | per-pipeline | 49.39 | 49.98 | 83.20 |
| 20000 | 125% | **shared** | **477564** | **460849** | **656188** |
At 31% utilization the shared scheduler still wins by 5.4x. Past saturation
it
collapses: p50 goes from 9.3 us to 461 ms, a factor of about 49000. The
per-pipeline arm is flat across all three, because a dedicated two-thread
pool
per pipeline never contends.
461 ms is not a bound. Past saturation the unbounded queue grows for as long
as
the overload lasts; the 10 second iteration simply truncates the backlog.
This
is the "delay rather than drop" tradeoff that was already argued explicitly
on
#12165 when the pool was bounded - the benchmark now puts a number on where
it
starts.
## Is that cliff reachable? I traced the dispatch body
The cliff only matters if a real checkpoint trigger can occupy a dispatch
thread
for tens of milliseconds. The body is `tryTriggerPendingCheckpoint`, and on
the
#12165 branch everything expensive in it is already delegated:
- `createPendingCheckpoint` allocates the checkpoint id through
`supplyAsync(..., executorService)`, with a comment stating it must happen
off
the lock because it talks to external services in HA mode and may block
- `triggerPendingCheckpoint` chains `thenApplyAsync(..., executorService)`
- `startTriggerPendingCheckpoint` is `thenAcceptAsync(..., executorService)`
since `31a454f35`, and the barrier `allOf(...).get()` runs inside that
async
body
- `completePendingCheckpoint` is `public synchronized`, so it takes the
`this`
monitor rather than `lock`; its `checkpointStorage.storeCheckPoint` IO
does not
contend with the trigger
What is left on the dispatch thread is atomics, timestamp comparisons,
logging
and a map put. Microseconds. **At the production default interval of 300000
ms,
utilization is effectively zero and the cliff is not reachable in steady
state.**
There is one mechanism I found that can hold a dispatch thread far longer.
`startSavepoint` holds the coordinator lock across a sleep-poll:
```java
synchronized (lock) {
while (pendingCounter.get() > 0 && !shutdown) {
Thread.sleep(500);
}
savepoint = createPendingCheckpoint(...);
```
`tryTriggerPendingCheckpoint` takes that same `lock`, so a periodic trigger
that
fires during a savepoint window blocks on a dispatch thread until the
in-flight
checkpoint completes - seconds under backpressure. `CheckpointManager
.triggerSavePoints()` fans out over every coordinator in a `parallelStream`,
so
one job suspend puts all of that job's pipelines into this state at once.
I want to be careful about how far that goes. The mechanism is verified in
the
code. **Reaching eight simultaneous occupancies is not demonstrated** - it
needs
periodic triggers to coincide with savepoint windows, which at a 300 s
interval
is a narrow coincidence, though the 500 ms re-arm on a pending checkpoint
widens
it. I have not reproduced it and I am not reporting it as a bug.
The part I am confident about is the structural one: this is a **new
cross-pipeline coupling**. On `dev` a savepointing pipeline can block only
its
own two threads, and no other pipeline can be affected by it. With one shared
bounded pool, enough concurrently blocked pipelines can delay unrelated
pipelines' triggers. That is the same blast-radius concern as the
`thenAccept`
path, relocated rather than eliminated.
## Where this leaves the option
My earlier position was that with the blocking call moved off the dispatch
thread there was no case I could point to where the auto-sized pool is wrong.
That was too strong, and the saturation run is why.
I still would not add `checkpoint.scheduler-dispatch-thread-num` in this PR.
Nothing here shows the auto-sized pool being wrong in steady state, and the
one
path that could saturate it is a coincidence I have not been able to
demonstrate.
What the data does change is the priority of the metric. I had it as a
follow-up
behind the option. It should come first, because scheduling delay is exactly
the
detector for approaching this cliff: it is flat at about 10 us at 31%
utilization and four orders of magnitude worse past 100%, so it separates the
safe regime from the unsafe one cleanly, and it is the evidence that would
justify the option if a real deployment ever needs it.
## Caveats
One machine, JDK 17 local, while the CI benchmark matrix is Java 8 and 11.
`ActiveProcessorCount=4` is pinned by the benchmark while the host has more
cores, so the dispatch bound under test is 8. These numbers establish the
shape
and the crossover, not absolutes for any particular deployment.
Raw JSON for all three runs is available if it is useful. If you would rather
see this confirmed on CI hardware across the Java 8 and 11 matrix, #12418
adds
`CheckpointSchedulingBenchmark` to the workflow dispatch list, so it can be
run
there against both branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
