c-taylor opened a new issue #7373:
URL: https://github.com/apache/trafficserver/issues/7373
When performing a TLS handshake microbenchmark: The TLS session cache lock
causes TLS handshake rate to drop as more ET_NET threads are added. This is a
problem as crypto operations on a system scale horizontally by core, and yet
ATS scales inversely in this aspect.
Benchmarking tools: tls-perf or a 0 byte reply from a standard HTTP
benchmarking tool.
perf tracing shows that ET_NET threads spend most of there time spinning on
a lock:
```
0.89% [ET_NET 17] [kernel.kallsyms] [k]
queued_spin_lock_slowpath
|
---queued_spin_lock_slowpath
```
Digging a little deeper into the stack trace, this is always:
```
0.03% [ET_NET 8] traffic_server [.] SSLSessionBucket::insertSession
0.03% [ET_NET 30] traffic_server [.] SSLSessionBucket::insertSession
0.03% [ET_NET 49] traffic_server [.] SSLSessionBucket::insertSession
```
### Existing workarounds
Whilst you can configure to skip the lock on a cache insertion, you can't on
an eviction.
Busy systems tend to be 1:1 on insertions to evictions, so the lock is still
heavily contended either way.
Disabling the TLS session cache really helps, and it is also largely
superfluous to some workloads.
However it is also used for the TLSv1.3 replay protection, and doesn't seem
to be separable.
## Replication
* Higher numbers of ET_NET threads amplify this issue.
* Pinning threads to logical CPUs also seems to amplify the issue.
## Issues
1. Session cache lock causes inverse scaling on high CPU count systems
It does look as though some refactoring could be achieved to hold the lock
for less time. Although any lockless approach to what is an optimistic
optimisation would clearly be preferable.
2. Disabling TLS session cache disables TLSv1.3 replay protection.
This seems like the session cache has been 'overloaded' with this use case:
Replay protection is reducing the overall TLS handshake capability, and should
be replaced with a lighter, faster structure.
It should be possible to configure this independently of the session cache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
