Hi,

A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 connections 
in parallel. The threads of one process get about the throughput of a single 
thread, and adding threads makes it worse. The same client as separate 
processes scales as expected. I ran into it with FrankenPHP (PHP's ZTS build 
served from the threads of one process, libpq 17.12, OpenSSL 3.5.7, Debian 13): 
8 threads doing one connect per request reach ~600 connects/s against ~1,700 
for 8 processes, and ~2,900 for the same 8 threads once the role uses md5 or 
trust.

The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one of 
the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, 
EVP_sha256(), NULL) does an implicit fetch of the digest, which takes 
process-wide locks inside libcrypto (openssl/openssl#19612 describes the same 
cost for EVP_DigestInit_ex). With one thread that is a per-iteration overhead; 
with several threads it is a lock convoy. The EVP_MAC conversion in 1f3b9bb109 
keeps the per-iteration EVP_MAC_init() with the digest parameter and has the 
same behaviour.

A 60-line C program with no libpq in it shows the effect. It runs the 
4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 CPUs, 
loops per second:

  pattern per iteration                               1 thr   8 thr  16 thr   8 
proc
  HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL)      519     547     252    
3,397   (libpq <= 17)
  EVP_MAC_init(ctx, key, len, {digest=SHA256})         600   1,191     547    
4,582   (master)
  HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150   5,684   9,832    
6,366
  HMAC_Init_ex(ctx, NULL, 0, NULL, NULL)             1,997   9,272  12,095   
18,557   (reuse key and digest)
  EVP_MAC_init(ctx, NULL, 0, NULL)                   1,085   8,045 13,785 13,613

And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL 
16, scram role:

  role            1 thr   8 thr  16 thr   1 proc  8 proc  16 proc
  scram-sha-256     237     592     268      222   1,641    2,386
  trust             595   1,882   3,814      559   2,640    3,594

This was discussed in February 2025 ("Optimize scram_SaltedPassword 
performance", 
https://www.postgresql.org/message-id/Z6B7QXlPwPXewaQZ%40paquier.xyz). That 
proposal was about single-thread latency and was dropped, partly because a 
faster PBKDF2 does not help the server much and partly because of the static 
state in the patch. The multithreaded case is different: it is not that the 
client is 4 times slower than it could be, it is that N threads are no faster 
than one, which for a threaded application server means the database connect 
rate of the whole process is capped around a few hundred per second. The fix 
does not need any static state: inside scram_SaltedPassword() the key is the 
same for every iteration, so the OpenSSL implementation can keep it in the 
context after the first pg_hmac_init() and re-init with a NULL key 
(HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) on 
the EVP_MAC path). The fallback src/common/hmac.c can keep its current 
behaviour, or skip re-deriving the pads when the key pointer and length are 
unchanged.

I can send a patch for the OpenSSL path if there is interest. The reproduction 
(Docker, PHP and C) is at 
https://github.com/mattleads/frankenphp-pgconnect-repro.

Regards,
Matt Mochalkin



Reply via email to