On Fri, Jul 23, 2021 at 5:46 PM Bossart, Nathan <bossa...@amazon.com> wrote: > My apologies for chiming in so late to this thread, but a similar idea > crossed my mind while working on a bug where .ready files get created > too early [0]. Specifically, instead of maintaining a status file per > WAL segment, I was thinking we could narrow it down to a couple of > files to keep track of the boundaries we care about: > > 1. earliest_done: the oldest segment that has been archived and > can be recycled/removed > 2. latest_done: the newest segment that has been archived > 3. latest_ready: the newest segment that is ready for archival > > This might complicate matters for backup utilities that currently > modify the .ready/.done files, but it would simplify this archive > status stuff quite a bit and eliminate the need to worry about the > directory scans in the first place.
In terms of immediate next steps, I think we should focus on eliminating the O(n^2) problem and not get sucked into a bigger redesign. The patch on the table aims to do just that much and I think that's a good thing. But in the longer term I agree that we want to redesign the signalling somehow. I am not convinced that using a file is the right way to go. If we had to rewrite that file for every change, and especially if we had to fsync it, it would be almost as bad as what we're doing right now in terms of the amount of traffic to the filesystem. Atomicity is a problem too, because if we simply create a file then after a crash it will either exist or not, but a file might end up garbled with a mix of old and new contents unless we always write a temporary file and automatically rename that over the existing one. As I said in my original post, I'm kind of wondering about keeping the information in shared memory instead of using the filesystem. I think we would still need to persist it to disk at least occasionally but perhaps there is a way to avoid having to do that as frequently as what we do now. I haven't thought too deeply about what the requirements are here. -- Robert Haas EDB: http://www.enterprisedb.com
