On Wed, Feb 8, 2023 at 7:24 PM Nathan Bossart <nathandboss...@gmail.com> wrote: > On Thu, Feb 09, 2023 at 08:56:24AM +0900, Michael Paquier wrote: > > On Wed, Feb 08, 2023 at 02:25:54PM -0800, Nathan Bossart wrote: > >> These are all good points. Perhaps there could be a base archiver > >> implementation that shell_archive uses (and that other modules could use if > >> desired, which might be important for backward compatibility with the > >> existing callbacks). But if you want to do something fancier than > >> archiving sequentially, you could write your own. > > > > Which is basically the kind of things you can already achieve with a > > background worker and a module of your own? > > IMO one of the big pieces that's missing is a way to get the next N files > to archive. Right now, you'd have to trawl through archive_status on your > own if you wanted to batch/parallelize. I think one advantage of what > Robert is suggesting is that we could easily provide a supported way to get > the next set of files to archive, and we can asynchronously mark them > "done". Otherwise, each module has to implement this.
Right. I think that we could certainly, as Michael suggests, have people provide their own background worker rather than having the archiver invoke the user-supplied code directly. As long as the functions that you need in order to get the necessary information can be called from some other process, that's fine. The only difficulty I see is that if the archiving is happening from a separate background worker rather than from the archiver, then what is the archiver doing? We could somehow arrange to not run the archiver process at all, or I guess to just sit there and have it do nothing. Or, we can decide not to have a separate background worker and just have the archiver call the user-supplied core directly. I kind of like that approach at the moment; it seems more elegant to me. -- Robert Haas EDB: http://www.enterprisedb.com