junaiddshaukat opened a new pull request, #39752:
URL: https://github.com/apache/beam/pull/39752
## Summary
Part of #18479. An application for measuring what the runner does when
instances come and go.
This is an application rather than a test, following the review on #39745. A
test asserts something and passes; the question here is *how long* a handover
takes and what throughput does across it, and that is a thing you run against a
real Kafka with several instances and watch. It is not wired into any build
task.
## What it runs
An unbounded source, keyed over a configurable number of keys, into fixed
windows and a `GroupByKey`. Each instance prints once a second:
```
<timestamp> <instance> groups_per_second=<n> groups_total=<n>
elements_read=<n>
```
Two counters rather than one, because they answer different questions.
`elements_read` says whether the source is being read by this instance, and
`groups_total` says whether the grouping downstream of the shuffle is
producing. A stall before the shuffle and a stall after it look identical if
you only count output.
## Running it
One broker is enough — what gets run several times is the runner instance,
not the broker.
```
docker compose -f runners/kafka-streams/measurement/docker-compose.yml up -d
./gradlew :runners:kafka-streams:measurement:installDist
BIN=runners/kafka-streams/measurement/build/install/measurement/bin/measurement
$BIN --applicationId=demo --instanceName=one --stateDir=/tmp/ks-one \
--internalParallelism=3 --numKeys=2000 --maxBundleSize=1000
--readMaxElementsPerPoll=200 &
$BIN --applicationId=demo --instanceName=two --stateDir=/tmp/ks-two \
--internalParallelism=3 --numKeys=2000 --maxBundleSize=1000
--readMaxElementsPerPoll=200 &
```
The instances share an application id, so Kafka's consumer group divides the
work between them. Each needs its own `--stateDir`; sharing one fails with a
`LockException`.
## What it measures
Killing the instance that is reading, with `kill -9` so there is no graceful
`LeaveGroup`, and timing how long before the survivor's `elements_read` starts
climbing:
| `--sessionTimeoutMs` | handover | beyond detection |
| --- | --- | --- |
| 6000 | 8.6s, 8.7s, 9.1s, 10.0s | 2.6–4.0s |
| 45000 (default) | 55.7s, 55.8s | 10.7–10.8s |
Handover is dominated by how long the consumer group takes to notice, which
is `session.timeout.ms` — the option added in #39748. The survivor's read rate
returns to the same ~40k elements/sec the killed instance had, and in the 6s
runs its grouping continued across the handover (485 → 412 groups/sec).
Reading resumes as a step rather than a ramp: 0 to full rate within one
one-second sample.
The remainder beyond detection is not constant — about 3s when the group
notices after 6s, and about 10.7s when it notices after 45s. A longer blind
period leaving more to catch up on would explain that, but this does not show
it: the two are varied together here, so the measurement cannot separate the
cost of a longer wait from the cost of the backlog it accumulates. Both 45s
runs landed within 0.1s of each other, so it is reproducible rather than noise,
whatever the cause.
These are numbers from a laptop with one broker and two instances, and are
meant to show the shape of the thing — detection dominating, the rest following
— rather than to be quoted as the runner's performance.
## Notes
Reading concentrates on one instance rather than spreading, so a handover
measurement has to kill the instance that is actually reading — killing the
other measures the loss of a process that was not doing the work.
Picking a read rate matters. At `--readMaxElementsPerPoll=20000` this
pipeline read tens of millions of elements and emitted no groups at all; at 200
it emits steadily. Output stops altogether rather than degrading gradually,
which is worth knowing when choosing a rate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
