Hi, thanks for the data. I suspect the increase of latencies will be caused by other things than contention in the completion registry. Basically the completion registry is a dictionary of scheduled "operation completions" (where a completion is one of the primitive operations defined by the C-core native layer - e.g. start a call, send a message, receive a message etc.). Access to this dictionary is protected by a SpinLock (as you've pointed out). The thing is, it seems weird that your system would be able to generate so much traffic that it would lead to contention on the completion registry dictionary. Updating the dictionary is hardly any work compared to the overhead of e.g. sending or receiving a message so I'd expect the throughput of completion registry is much higher than the actual throughput of your system. I suspect that since the time spent in SpinLock is extremely short AND most of the work actually happens in the native C-core layer, the profiler might be giving incorrect data?
The spin lock could be of course replaced by other synchronization primitive (e.g. concurrent dictionary or a normal lock), but in my experience I've never seen completion registry being the main reason for Grpc.Core being slow. The change is doable, but it would require careful benchmarking first. Since Grpc.Core is in maintenace mode (see https://grpc.io/blog/grpc-csharp-future/), this is likely now worth the work. On Monday, January 17, 2022 at 8:32:14 AM UTC+1 [email protected] wrote: > Hi, > > We are running a client-server load test with 500 concurrent users, and we > are seeing a high CPU on a SpinLock called > by Grpc.Core.Internal.CompletionRegistry.Register. > At average, this SpinLock is seen on 15% of the CPU frames under a > profiler. But during few seconds where we're seeing a significant increase > in latencies, the profiler shows that this SpinLock cost had an average > cost of ~80% of the CPU frames, and even 95% of the CPU frames: > [image: GrpcCore_SpinLock5.JPG] > > Is there any way to avoid this penalty? > I see that this SpinLock is protecting a Dictionary > of IOpCompletionCallbacks. Are those callbacks affected by anything that we > control in our code? > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1b605ddb-8731-47a1-98ea-113808cb11e3n%40googlegroups.com.
