Yes, same thing. The contention your perf report showed inside HMAC_Init_ex() is what the patch removes.
I ran pgbench --connect through pgbouncer as you suggested. Same pgbench binary (master), base and patched libpq swapped with LD_LIBRARY_PATH. pgbouncer 1.24.1, auth_type = scram-sha-256, the role's SCRAM verifier in auth_file, PostgreSQL 16 behind it, OpenSSL 3.5.7, 8 CPUs for the client. pgbench --connect -n -c N -j N -T 20 -f select1.sql, connections per second: -c = -j base patched 1 403 451 8 1,170 5,380 16 734 8,555 With the base libpq pgbench used 6-8 cores for those ~1,000 connections/s, the "1000% CPU" you saw. With the patch it uses about the same CPU for 5-8x the connections. If you’d like to repeat it. Put the SCRAM verifier in pgbouncer's auth_file, not the plain password: with the plain password pgbouncer does the PBKDF2 itself and caps the test at ~380 logins/s no matter what libpq does. And at 16 threads the patched client runs out of ephemeral ports in 20 s unless tcp_tw_reuse is on. Regards, Matt Mochalkin > On 17 Nov 2026, at 5:24 PM, Jakub Wartak <[email protected]> > wrote: > > On Mon, Nov 16, 2026 at 11:08 PM Matthew Mochalkin > <[email protected] <mailto:[email protected]>> wrote: >> >> Hi, >> >> A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 >> connections in parallel. The threads of one process get about the throughput >> of a single thread, and adding threads makes it worse. The same client as >> separate processes scales as expected. I ran into it with FrankenPHP (PHP's >> ZTS build served from the threads of one process, libpq 17.12, OpenSSL >> 3.5.7, Debian 13): 8 threads doing one connect per request reach ~600 >> connects/s against ~1,700 for 8 processes, and ~2,900 for the same 8 threads >> once the role uses md5 or trust. >> >> The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one >> of the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, >> EVP_sha256(), NULL) does an implicit fetch of the digest, which takes >> process-wide locks inside libcrypto (openssl/openssl#19612 describes the >> same cost for EVP_DigestInit_ex). With one thread that is a per-iteration >> overhead; with several threads it is a lock convoy. The EVP_MAC conversion >> in 1f3b9bb109 keeps the per-iteration EVP_MAC_init() with the digest >> parameter and has the same behaviour. >> >> A 60-line C program with no libpq in it shows the effect. It runs the >> 4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 >> CPUs, loops per second: >> >> pattern per iteration 1 thr 8 thr 16 thr >> 8 proc >> HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) 519 547 252 >> 3,397 (libpq <= 17) >> EVP_MAC_init(ctx, key, len, {digest=SHA256}) 600 1,191 547 >> 4,582 (master) >> HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150 5,684 9,832 >> 6,366 >> HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) 1,997 9,272 12,095 >> 18,557 (reuse key and digest) >> EVP_MAC_init(ctx, NULL, 0, NULL) 1,085 8,045 13,785 >> 13,613 >> >> And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL >> 16, scram role: >> >> role 1 thr 8 thr 16 thr 1 proc 8 proc 16 proc >> scram-sha-256 237 592 268 222 1,641 2,386 >> trust 595 1,882 3,814 559 2,640 3,594 >> >> This was discussed in February 2025 ("Optimize scram_SaltedPassword >> performance", >> https://www.postgresql.org/message-id/Z6B7QXlPwPXewaQZ%40paquier.xyz). That >> proposal was about single-thread latency and was dropped, partly because a >> faster PBKDF2 does not help the server much and partly because of the static >> state in the patch. The multithreaded case is different: it is not that the >> client is 4 times slower than it could be, it is that N threads are no >> faster than one, which for a threaded application server means the database >> connect rate of the whole process is capped around a few hundred per second. >> The fix does not need any static state: inside scram_SaltedPassword() the >> key is the same for every iteration, so the OpenSSL implementation can keep >> it in the context after the first pg_hmac_init() and re-init with a NULL key >> (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) >> on the EVP_MAC path). The fallback src/common/hmac.c can keep its current >> behaviour, or skip re-deriving the pads when the key pointer and length are >> unchanged. >> >> I can send a patch for the OpenSSL path if there is interest. The >> reproduction (Docker, PHP and C) is at >> https://github.com/mattleads/frankenphp-pgconnect-repro. > > To me it looks like another instance of [1], isn't it? If that would be true > then pgbench --connect along with pgbouncer (quite to setup) could be enough > to test/verify your patch idea too. > > [1] - > https://www.postgresql.org/message-id/CAKZiRmw6jbShG%2BLJe0pY5G4w0ktoGwzPozsA%3DLhH_mN9wC%2Bbhg%40mail.gmail.com
