On Wed, Jun 11, 2014 at 12:25 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Jun 11, 2014 at 7:59 AM, Gurjeet Singh <gurj...@singh.im> wrote: >> On Sun, Jun 8, 2014 at 3:24 AM, Amit Kapila <amit.kapil...@gmail.com> >> wrote: >> > Yeap, but if it crashes before writing checkpoint record, it will lead >> > to >> > recovery which is what we are considering. >> >> Good point. >> >> In case of such recovery, the recovery process will read in the blocks >> that were recently modified, and were possibly still in shared-buffers >> when Checkpointer crashed. So after recovery finishes, the >> BlockReaders will be invoked (because save-files were successfully >> written before the crash), and they would request the same blocks to >> be restored. Most likely, those blocks would already be in >> shared-buffers, hence no cause of concern regarding BlockReaders >> evicting buffers populated by recovery. > > Not necessarily because after crash, recovery has to start from > previous checkpoint, so it might not perform operations on same > pages as are saved by buffer saver.
Granted, the recovery may not start that way (that is, reading in blocks that were in shared-buffers when shutdown was initiated), but it sure would end that way. Towards the end of recovery, the blocks it'd read back in are highly likely to be the ones that were present in shared-buffers at the time of shutdown. By the end of recovery, either (a) blocks read in at the beginning of recovery are evicted by later operations of recovery, or (b) they are still present in shared-buffers. So the blocks requested by the BlockReaders are highly likely to be already in shared-buffers at the end of recovery, because these are the same blocks that were dirty (and hence recorded in WAL) just before shutdown time. I guess what I am trying to say is that the blocks read in by the BlockReaders will be a superset of those read in by the reocvery process. At the time of shutdown/saving-buffers, the shared-buffers may have contained dirty and clean buffers. WAL contains the info of which blocks were dirtied. Recovery will read back the blocks that were dirty, to replay the WAL, and since the BlockReaders are started _after_ recovery finishes, the BlockReaders will effectively read in only those blocks that are not already read-in by the recovery. I am not yet convinced, at least in this case, that Postgres Hibernator would restore blocks that can cause eviction of buffers restored by recovery. I don't have intimate knowledge of recovery but I think the above assessment of recovery's operations holds true. If you still think this is a concern, can you please provide a bit firm example using which I can visualize the problem you're talking about. > Also as the file saved by > buffer saver can be a file which contains only partial list of > buffers which were in shared buffer's, it becomes more likely that > in such cases it can override the buffers populated by recovery. I beg to differ. As described above, the blocks read-in by the BlockReader will not evict the recovery-restored blocks. The save-files being written partially does not change that. > Now as pg_hibernator doesn't give any preference to usage_count while > saving buffer's, it can also evict the buffers populated by recovery > with some lower used pages of previous run. The case we're discussing (checkpointer/BufferSaver/some-other-process crash during a smart/fast shutdown) should occur rarely in practice. Although Postgres Hibernator is not yet proven to do the wrong thing in this case, I hope you'd agree that BlockReaders evicting buffers populated by recovery process is not catastrophic at all, merely inconvenient from performance perspective. Also, the impact is only on the initial performance immediately after startup, since application queries will re-prime the shared-buffers with whatever buffers they need. > I think Block Readers will remove file only after reading and populating > buffers from it Correct. > and that's the reason I mentioned that it can lead to doing > both recovery as well as load buffers based on file saved by buffer > saver. I am not sure I completely understand the implication here, but I think the above description of case where recovery-followed-by-BlockReaders not causing a concern may cover it. Best regards, -- Gurjeet Singh http://gurjeet.singh.im/ EDB www.EnterpriseDB.com -- Sent via pgsql-hackers mailing list (firstname.lastname@example.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers