On Mon, Oct 26, 2026 at 11:06 PM Bryan Green <[email protected]> wrote: > > Greetings, > > When vm_extend() grows the visibility map it sends an smgr > invalidation. Every backend that has the relation open then closes > all of its files, and reopens each segment the next time it touches > the table. VACUUM extends the map about once per 255MB of heap. On a > multi-TB table with a few hundred backends inserting, that is a lot of > opens, and as far as I can tell none of them are needed. The other > backends only have to learn that the map got bigger. > > This assumes open() is cheap, and it isn't always. I ran into it on > 17 with a ~4TB partition, on a system where open() goes through a > filesystem encryption layer. Each time VACUUM extended the map, every > inserting backend reopened about 4,000 segments, and inserts stalled > for seconds at a time. The same invalidation is in every branch from > 14 to master, and on master on-access pruning can extend the map as > well, so it isn't only VACUUM. > > The attached patch adds an invalidation message that just resets the > cached fork sizes, and vm_extend() sends that instead. Readers see > new map pages when they do now. I also tried dropping the > invalidation entirely, but then a backend that doesn't prune the > pages itself keeps a stale map size until VACUUM finishes. > > On eCryptfs, with about 2,700 segments and 300 inserting clients, > VACUUM went from about 10 million opens to under 1 million, the extra > system CPU it caused mostly went away, and p99 insert latency during > VACUUM dropped from ~350ms to under 20ms. check-world passes. > > Is there a reason the full smgr invalidation is needed here that I'm > missing?
I've digged a little into histoy and got this context: It appers that years earlier (see b4e074172768) smgrinvals were sent only for truncation earlier. It also seems that code before than that ended up doing lots of lseek?/open() without VM and this commit (optimization) assumed (somewhat correctly) that open()/close() is cheap, but it's only has low latency on standard stuff (ext4/XFS), but there might some (encrypted) filesystems where that is not that cheap. I've tracked down this commit down to this thread [1] which contains that this was an optmization to a regression due to lack of VM in the first place. Anyway, we can repro extreme high 50%+ sys% (kernel) with with encFS(LUKS) just with lots of close() /open() stress test, so avoiding this would help users of such encrypted filesystems. E.g. pure open/close syscall benchmark shows: ext4 throughput 1736656 opens/s @ 8 processes encfs throughput 87017 opens/s @ 8 processes (of course it's FUSE/userland)but that's like ~20x slower and when hundreths of such backends do receive such sinval (by VACCUUM -> vm_extend()) almost all at once, it's almost DoS to the whole system shortly, the bigger the impact the more segments (bigger relation/more files are being opened()/ closed()) multipled by backends handling sinvals. I think we can say about that for three such filesystems already on Linux (two via FUSE, and at least one implemented via LKM). So question is: should we optimize further for such high-latency and costly open()/close() filesystems? My thinking is that we could help vm_extend() here avoiding doing that, but then there are other sinvals (e.g. due to DDLs) that will also cause such "kernel stress" (hiccups). BTW: @Bryan: I've created commitfest entry [2] -J. [1] - https://www.postgresql.org/message-id/flat/CAEZATCVufYwtULWuL5Um1YvnW1N8jMyBy%2BqkPU9Q1xOqCcMa-Q%40mail.gmail.com [2] - https://commitfest.postgresql.org/patch/7528/
