Yes, same thing. The contention your perf report showed inside HMAC_Init_ex() 
is what the patch removes.

I ran pgbench --connect through pgbouncer as you suggested. Same pgbench binary 
(master), base and patched libpq swapped with LD_LIBRARY_PATH. pgbouncer 
1.24.1, auth_type = scram-sha-256, the role's SCRAM verifier in auth_file, 
PostgreSQL 16 behind it, OpenSSL 3.5.7, 8 CPUs for the client.

pgbench --connect -n -c N -j N -T 20 -f select1.sql, connections per second:

  -c = -j    base    patched
  1           403        451
  8         1,170      5,380
  16          734      8,555

With the base libpq pgbench used 6-8 cores for those ~1,000 connections/s, the 
"1000% CPU" you saw. With the patch it uses about the same CPU for 5-8x the 
connections.

If you’d like to repeat it. Put the SCRAM verifier in pgbouncer's auth_file, 
not the plain password: with the plain password pgbouncer does the PBKDF2 
itself and caps the test at ~380 logins/s no matter what libpq does. And at 16 
threads the patched client runs out of ephemeral ports in 20 s unless 
tcp_tw_reuse is on.

Regards,
Matt Mochalkin






> On 17 Nov 2026, at 5:24 PM, Jakub Wartak <[email protected]> 
> wrote:
> 
> On Mon, Nov 16, 2026 at 11:08 PM Matthew Mochalkin
> <[email protected] <mailto:[email protected]>> wrote:
>> 
>> Hi,
>> 
>> A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 
>> connections in parallel. The threads of one process get about the throughput 
>> of a single thread, and adding threads makes it worse. The same client as 
>> separate processes scales as expected. I ran into it with FrankenPHP (PHP's 
>> ZTS build served from the threads of one process, libpq 17.12, OpenSSL 
>> 3.5.7, Debian 13): 8 threads doing one connect per request reach ~600 
>> connects/s against ~1,700 for 8 processes, and ~2,900 for the same 8 threads 
>> once the role uses md5 or trust.
>> 
>> The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one 
>> of the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, 
>> EVP_sha256(), NULL) does an implicit fetch of the digest, which takes 
>> process-wide locks inside libcrypto (openssl/openssl#19612 describes the 
>> same cost for EVP_DigestInit_ex). With one thread that is a per-iteration 
>> overhead; with several threads it is a lock convoy. The EVP_MAC conversion 
>> in 1f3b9bb109 keeps the per-iteration EVP_MAC_init() with the digest 
>> parameter and has the same behaviour.
>> 
>> A 60-line C program with no libpq in it shows the effect. It runs the 
>> 4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 
>> CPUs, loops per second:
>> 
>>  pattern per iteration                               1 thr   8 thr  16 thr   
>> 8 proc
>>  HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL)      519     547     252    
>> 3,397   (libpq <= 17)
>>  EVP_MAC_init(ctx, key, len, {digest=SHA256})         600   1,191     547    
>> 4,582   (master)
>>  HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150   5,684   9,832    
>> 6,366
>>  HMAC_Init_ex(ctx, NULL, 0, NULL, NULL)             1,997   9,272  12,095   
>> 18,557   (reuse key and digest)
>>  EVP_MAC_init(ctx, NULL, 0, NULL)                   1,085   8,045 13,785 
>> 13,613
>> 
>> And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL 
>> 16, scram role:
>> 
>>  role            1 thr   8 thr  16 thr   1 proc  8 proc  16 proc
>>  scram-sha-256     237     592     268      222   1,641    2,386
>>  trust             595   1,882   3,814      559   2,640    3,594
>> 
>> This was discussed in February 2025 ("Optimize scram_SaltedPassword 
>> performance", 
>> https://www.postgresql.org/message-id/Z6B7QXlPwPXewaQZ%40paquier.xyz). That 
>> proposal was about single-thread latency and was dropped, partly because a 
>> faster PBKDF2 does not help the server much and partly because of the static 
>> state in the patch. The multithreaded case is different: it is not that the 
>> client is 4 times slower than it could be, it is that N threads are no 
>> faster than one, which for a threaded application server means the database 
>> connect rate of the whole process is capped around a few hundred per second. 
>> The fix does not need any static state: inside scram_SaltedPassword() the 
>> key is the same for every iteration, so the OpenSSL implementation can keep 
>> it in the context after the first pg_hmac_init() and re-init with a NULL key 
>> (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) 
>> on the EVP_MAC path). The fallback src/common/hmac.c can keep its current 
>> behaviour, or skip re-deriving the pads when the key pointer and length are 
>> unchanged.
>> 
>> I can send a patch for the OpenSSL path if there is interest. The 
>> reproduction (Docker, PHP and C) is at 
>> https://github.com/mattleads/frankenphp-pgconnect-repro.
> 
> To me it looks like another instance of [1], isn't it? If that would be true
> then pgbench --connect along with pgbouncer (quite to setup) could be enough
> to test/verify your patch idea too.
> 
> [1] - 
> https://www.postgresql.org/message-id/CAKZiRmw6jbShG%2BLJe0pY5G4w0ktoGwzPozsA%3DLhH_mN9wC%2Bbhg%40mail.gmail.com

Reply via email to