Aetherance commented on PR #3541:
URL: https://github.com/apache/kvrocks/pull/3541#issuecomment-5119394839
I ran multiple rounds of local performance benchmarks using
`redis-benchmark`, covering several scenarios. I also included Redis as a
reference to verify that this PR does not introduce a performance regression.
For example, the SET benchmark was run using the following command:
```bash
redis-benchmark -h 127.0.0.1 -p 6670 --threads 4 -c 64 -P 16 \
-n 1000000 -r 1000000000 --seed 2002 --csv \
SET bench:set:__rand_int__
0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
```
The benchmarks covered the Kvrocks `unstable` branch, the current PR branch,
and Redis with notifications both disabled and enabled, allowing comparison
with the keyspace notification implementation in this PR.
### Kvrocks
| Version / configuration | Test | RPS | Avg latency (ms) | Min (ms) | P50
(ms) | P95 (ms) | P99 (ms) | Max (ms) |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| unstable | DEL | 399,042.28 | 2.441 | 0.312 | 2.287 | 3.911 | 4.239 |
13.383 |
| unstable | SET | 498,753.09 | 1.798 | 0.248 | 1.831 | 2.287 | 2.479 |
9.895 |
| unstable | PING | 1,992,032.00 | 0.275 | 0.048 | 0.239 | 0.455 | 0.471 |
2.671 |
| PR, notifications disabled | DEL | 398,724.09 | 2.348 | 0.480 | 2.351 |
2.727 | 3.015 | 11.791 |
| PR, notifications disabled | SET | 570,450.62 | 1.757 | 0.096 | 1.839 |
2.367 | 2.551 | 11.679 |
| PR, notifications disabled | PING | 1,992,032.00 | 0.249 | 0.024 | 0.255 |
0.343 | 0.383 | 2.087 |
| PR, `KE$g` | DEL | 332,225.91 | 2.914 | 0.864 | 2.735 | 3.983 | 5.199 |
12.815 |
| PR, `KE$g` | SET | 399,201.59 | 2.327 | 0.352 | 2.423 | 3.351 | 4.151 |
13.863 |
### Redis reference
| Configuration | Test | RPS | Avg latency (ms) | Min (ms) | P50 (ms) | P95
(ms) | P99 (ms) | Max (ms) |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| Notifications disabled | DEL | 1,329,787.25 | 0.636 | 0.120 | 0.639 |
0.911 | 1.063 | 3.271 |
| Notifications disabled | SET | 996,016.00 | 0.801 | 0.120 | 0.823 | 1.111
| 1.615 | 3.423 |
| `KE$g` | DEL | 664,451.81 | 1.411 | 0.168 | 1.439 | 1.871 | 2.247 | 7.111 |
| `KE$g` | SET | 569,800.56 | 1.601 | 0.192 | 1.615 | 2.175 | 2.655 | 4.311 |
I ran multiple rounds of these benchmarks and observed no significant
variation between runs.
The results show no meaningful performance difference between the current PR
and the original `unstable` branch when notifications are disabled or when
unrelated commands are executed. This indicates that the PR has no measurable
performance impact in these scenarios.
With `KE$g` enabled and an active subscriber connected, SET and DEL
performance decreased. In this benchmark, the notification overhead in Kvrocks,
measured as throughput loss, was approximately 16.68% for DEL and 30.02% for
SET. Under the same conditions, the corresponding overhead in Redis was
approximately 50.03% for DEL and 42.79% for SET.
Because Redis and Kvrocks use different storage engines, their absolute RPS
values are not directly comparable. Instead, the relevant comparison is the
relative performance change before and after notifications are enabled. The
official Redis documentation also notes that keyspace notifications are
disabled by default partly because the feature consumes additional CPU
resources. [Redis Keyspace
Notify](https://redis.io/docs/latest/develop/pubsub/keyspace-notifications/)
To further identify the source of the performance overhead, I profiled the
PR branch using Linux `perf` and generated pprof-style differential call graphs
from the samples. The graphs compare two scenarios with `KE$g` enabled: one
without subscribers and one with an active subscriber. Red nodes indicate an
increase in CPU usage after the subscriber was connected.
SET
<img width="1724" height="1308" alt="set-notify pprof-style"
src="https://github.com/user-attachments/assets/547ce790-0162-4fed-a43a-1df080215b2e";
/>
DEL
<img width="1397" height="1259" alt="del-notify pprof-style"
src="https://github.com/user-attachments/assets/d5d3e8ac-8e6f-44c6-a181-43ff08bf7c76";
/>
The call graphs show that the performance bottleneck is in
`Server::PublishMessage`. The underlying Pub/Sub delivery path is part of
Kvrocks’ existing implementation, and this PR invokes it only when
notifications are enabled and a notification event is actually generated. These
results therefore indicate that the PR does not affect performance when
notifications are disabled or when unrelated commands are executed. For the hot
path that actually generates notifications, the observed performance cost has a
clear and expected source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
