ChenSammi commented on PR #11744:
URL: https://github.com/apache/ozone/pull/11744#issuecomment-6974973053

   Master branch benchmark data
   Random access: 1,009 rows/sec
   Vector search: 73.8 QPS
   nprobes=  1  QPS=  82.5  avg_latency=12.1ms
   nprobes=  4  QPS=  88.8  avg_latency=11.3ms
   nprobes=  8  QPS=  95.8  avg_latency=10.4ms
   nprobes= 16  QPS=  90.2  avg_latency=11.1ms
   nprobes= 32  QPS=  95.4  avg_latency=10.5ms
   
   Current patch benchmark data
   Random access: 1,198 rows/sec
   Vector search: 80.5 QPS
   nprobes=  1  QPS=  88.4  avg_latency=11.3ms
   nprobes=  4  QPS=  95.5  avg_latency=10.5ms
   nprobes=  8  QPS=  94.6  avg_latency=10.6ms
   nprobes= 16  QPS=  95.2  avg_latency=10.5ms
   nprobes= 32  QPS=  96.6  avg_latency=10.4ms
   
   @smengcl, so the performance of current patch is better than the master. I 
run it locally with my laptop, so consider there is some fluctuations from run 
to run. 
   
   I reviewed the code. Can we keep inFlights and remove responseSlots, since 
if both are applied, it's `responseSlots` do the actural cap for short-circuit 
to reuse the executor, and it's default value is kind of small to me. In order 
to make sure a slower/dead short-circuit client will not hold all executor 
threads, the responses queue can achieve it already. And it's better to reduce 
the inFlights max, 72k or 288k seems too big, DN will run out of FD definitely 
in edge cases. How about a fixed value, because it's a per connection limit, 
has no logic relationship with executor size or DN disk count, we can make it a 
cofigurable paramerter if needed. 
   
   | Datanode | N    | `inFlight` only (5N × N) | With `responseSlots` (2N) |
   | ----------|---- |-------------------------|----------------------------|
   | 1 disk       | 10   | 500                                 | 20             
                            |
   | 12 disks.  | 120 | 72,000                            | 240                 
                      |
   | 24 disks  | 240 | 288,000                         | 480                    
                   |
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to