Re: [HACKERS] Production block comparison facility

2014-08-12 Thread Michael Paquier
On Wed, Jul 23, 2014 at 11:14 PM, Michael Paquier michael.paqu...@gmail.com wrote: Things could be refactored and improved for sure, but this patch is already useful as-is so I am going to add it to the next commit fest. After some more investigation, I am going to mark this patch as Returned

Re: [HACKERS] Production block comparison facility

2014-07-31 Thread Simon Riggs
On 29 July 2014 11:30, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't understand how this works. A full-page image contains the new page contents *after* the WAL-logged operation. For example, in a heap insert, the full-page image contains the new tuple. How can you compare that

Re: [HACKERS] Production block comparison facility

2014-07-31 Thread Michael Paquier
On Thu, Jul 31, 2014 at 2:59 PM, Simon Riggs si...@2ndquadrant.com wrote: On 29 July 2014 11:30, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't understand how this works. A full-page image contains the new page contents *after* the WAL-logged operation. For example, in a heap

Re: [HACKERS] Production block comparison facility

2014-07-31 Thread Simon Riggs
On 31 July 2014 07:45, Michael Paquier michael.paqu...@gmail.com wrote: So I don't see the need for two full page images. By doing so you definitely need an additional mode for full-page writes: one certifying that process does not apply this FPW because it wants to compare it to current

Re: [HACKERS] Production block comparison facility

2014-07-31 Thread Michael Paquier
On Thu, Jul 31, 2014 at 4:07 PM, Simon Riggs si...@2ndquadrant.com wrote: Yeh, it looks like you need to do CheckBackupBlock() exactly as many times as you do RestoreBackupBlock(), with the sequence of actions being RestoreBackupBlock(), apply WAL then CheckBackupBlock(). That will work

Re: [HACKERS] Production block comparison facility

2014-07-29 Thread Heikki Linnakangas
On 07/23/2014 05:14 PM, Michael Paquier wrote: On Tue, Jul 22, 2014 at 4:49 PM, Michael Paquier michael.paqu...@gmail.com wrote: Then, looking at the code, we would need to tweak XLogInsert for the WAL record construction to always do a FPW and to update XLogCheckBufferNeedsBackup. Then for

Re: [HACKERS] Production block comparison facility

2014-07-29 Thread Michael Paquier
On Tue, Jul 29, 2014 at 7:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't understand how this works. A full-page image contains the new page contents *after* the WAL-logged operation. For example, in a heap insert, the full-page image contains the new tuple. How can you compare

Re: [HACKERS] Production block comparison facility

2014-07-24 Thread Michael Paquier
On Thu, Jul 24, 2014 at 12:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On 23 July 2014 15:14, Michael Paquier michael.paqu...@gmail.com wrote: I have spent some time digging more into this idea and finished with the patch attached Thank you for investigating the idea. I'll review by

Re: [HACKERS] Production block comparison facility

2014-07-24 Thread Andres Freund
On 2014-07-24 20:35:04 +0900, Michael Paquier wrote: - FPW/page consistency check is done after converting them to hex. This is done only this way to facilitate viewing the page diffs with a debugger. A best method would be to perform the checks using MASK_MARKER (which should be moved to

Re: [HACKERS] Production block comparison facility

2014-07-24 Thread Michael Paquier
On Thu, Jul 24, 2014 at 8:36 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-07-24 20:35:04 +0900, Michael Paquier wrote: - FPW/page consistency check is done after converting them to hex. This is done only this way to facilitate viewing the page diffs with a debugger. A best method

Re: [HACKERS] Production block comparison facility

2014-07-23 Thread Michael Paquier
On Tue, Jul 22, 2014 at 4:49 PM, Michael Paquier michael.paqu...@gmail.com wrote: Then, looking at the code, we would need to tweak XLogInsert for the WAL record construction to always do a FPW and to update XLogCheckBufferNeedsBackup. Then for the redo part, we would need to do some extra

Re: [HACKERS] Production block comparison facility

2014-07-23 Thread Simon Riggs
On 23 July 2014 15:14, Michael Paquier michael.paqu...@gmail.com wrote: I have spent some time digging more into this idea and finished with the patch attached Thank you for investigating the idea. I'll review by Monday. -- Simon Riggs http://www.2ndQuadrant.com/

Re: [HACKERS] Production block comparison facility

2014-07-22 Thread Michael Paquier
On Sun, Jul 20, 2014 at 5:31 PM, Simon Riggs si...@2ndquadrant.com wrote: The block comparison facility presented earlier by Heikki would not be able to be used in production systems. ISTM that it would be desirable to have something that could be used in that way. ISTM easy to make these

Re: [HACKERS] Production block comparison facility

2014-07-22 Thread Simon Riggs
On 22 July 2014 08:49, Michael Paquier michael.paqu...@gmail.com wrote: On Sun, Jul 20, 2014 at 5:31 PM, Simon Riggs si...@2ndquadrant.com wrote: The block comparison facility presented earlier by Heikki would not be able to be used in production systems. ISTM that it would be desirable to

Re: [HACKERS] Production block comparison facility

2014-07-22 Thread Greg Stark
If you're always going FPW then there's no point in the rest of the record. The point here was to find problems so that users could run normally with confidence. The cases you might want to run in the mode you describe are the build farm or integration testing. When treating your application on

Re: [HACKERS] Production block comparison facility

2014-07-22 Thread Simon Riggs
On 22 July 2014 12:54, Greg Stark st...@mit.edu wrote: If you're always going FPW then there's no point in the rest of the record. I think its a simple matter to mark them XLP_BKP_REMOVABLE and to skip any optimization of remainder of WAL records. The point here was to find problems so that