ChenSammi commented on PR #11744:
URL: https://github.com/apache/ozone/pull/11744#issuecomment-6974973053
Master branch benchmark data
Random access: 1,009 rows/sec
Vector search: 73.8 QPS
nprobes= 1 QPS= 82.5 avg_latency=12.1ms
nprobes= 4 QPS= 88.8 avg_latency=11.3ms
nprobes= 8 QPS= 95.8 avg_latency=10.4ms
nprobes= 16 QPS= 90.2 avg_latency=11.1ms
nprobes= 32 QPS= 95.4 avg_latency=10.5ms
Current patch benchmark data
Random access: 1,198 rows/sec
Vector search: 80.5 QPS
nprobes= 1 QPS= 88.4 avg_latency=11.3ms
nprobes= 4 QPS= 95.5 avg_latency=10.5ms
nprobes= 8 QPS= 94.6 avg_latency=10.6ms
nprobes= 16 QPS= 95.2 avg_latency=10.5ms
nprobes= 32 QPS= 96.6 avg_latency=10.4ms
@smengcl, so the performance of current patch is better than the master. I
run it locally with my laptop, so consider there is some fluctuations from run
to run.
I reviewed the code. Can we keep inFlights and remove responseSlots, since
if both are applied, it's `responseSlots` do the actural cap for short-circuit
to reuse the executor, and it's default value is kind of small to me. In order
to make sure a slower/dead short-circuit client will not hold all executor
threads, the responses queue can achieve it already. And it's better to reduce
the inFlights max, 72k or 288k seems too big, DN will run out of FD definitely
in edge cases. How about a fixed value, because it's a per connection limit,
has no logic relationship with executor size or DN disk count, we can make it a
cofigurable paramerter if needed.
| Datanode | N | `inFlight` only (5N × N) | With `responseSlots` (2N) |
| ----------|---- |-------------------------|----------------------------|
| 1 disk | 10 | 500 | 20
|
| 12 disks. | 120 | 72,000 | 240
|
| 24 disks | 240 | 288,000 | 480
|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]