Re: [HACKERS] WAL replay bugs

2015-06-17 Thread Alvaro Herrera
Michael Paquier wrote: From 077d675795b4907904fa4e85abed8c4528f4666f Mon Sep 17 00:00:00 2001 From: Michael Paquier mich...@otacoo.com Date: Sat, 19 Jul 2014 10:40:20 +0900 Subject: [PATCH 3/3] Buffer capture facility: check WAL replay consistency Is there a newer version of this tech? --

Re: [HACKERS] WAL replay bugs

2015-06-17 Thread Michael Paquier
On Thu, Jun 18, 2015 at 3:39 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Michael Paquier wrote: From 077d675795b4907904fa4e85abed8c4528f4666f Mon Sep 17 00:00:00 2001 From: Michael Paquier mich...@otacoo.com Date: Sat, 19 Jul 2014 10:40:20 +0900 Subject: [PATCH 3/3] Buffer capture

Re: [HACKERS] WAL replay bugs

2014-11-05 Thread Alvaro Herrera
Michael Paquier wrote: Now, do we really want this feature in-core? That's somewhat a duplicate of what is mentioned here: http://www.postgresql.org/message-id/CAB7nPqQMq=4ejak317mxz4has0i+1rslbqu29zx18jwlb2j...@mail.gmail.com Of course both things do not have the same coverage as the former

Re: [HACKERS] WAL replay bugs

2014-11-05 Thread Peter Eisentraut
On 11/4/14 10:50 PM, Michael Paquier wrote: Except pg_upgrade, are there other tests using bash? There are a few obscure things under src/test/. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] WAL replay bugs

2014-11-05 Thread Michael Paquier
On Thu, Nov 6, 2014 at 5:41 AM, Peter Eisentraut pete...@gmx.net wrote: On 11/4/14 10:50 PM, Michael Paquier wrote: Except pg_upgrade, are there other tests using bash? There are a few obscure things under src/test/. Oh, I see. There is quite a number here, and each script is doing quite

Re: [HACKERS] WAL replay bugs

2014-11-04 Thread Alvaro Herrera
Michael Paquier wrote: On Mon, Jul 14, 2014 at 6:14 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Although I doubt necessity of the flexibility seeing the current testing framework, I don't have so strong objection about that. Nevertheless, perhaps you are appreciated to put

Re: [HACKERS] WAL replay bugs

2014-11-04 Thread Alvaro Herrera
Michael Paquier wrote: On Mon, Jul 14, 2014 at 6:14 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Although I doubt necessity of the flexibility seeing the current testing framework, I don't have so strong objection about that. Nevertheless, perhaps you are appreciated to put

Re: [HACKERS] WAL replay bugs

2014-11-04 Thread Peter Eisentraut
On 11/4/14 3:21 PM, Alvaro Herrera wrote: FWIW I gave this a trial run and found I needed some tweaks to test.sh and the Makefile in order to make it work on VPATH; mainly replace ./ with `dirname $0` in a couple test.sh in a couple of places, and something similar in the Makefile. Also you

Re: [HACKERS] WAL replay bugs

2014-11-04 Thread Michael Paquier
Thanks for the tests. On Wed, Nov 5, 2014 at 5:21 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Michael Paquier wrote: On Mon, Jul 14, 2014 at 6:14 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Although I doubt necessity of the flexibility seeing the current

Re: [HACKERS] WAL replay bugs

2014-11-04 Thread Michael Paquier
On Wed, Nov 5, 2014 at 6:29 AM, Peter Eisentraut pete...@gmx.net wrote: On 11/4/14 3:21 PM, Alvaro Herrera wrote: FWIW I gave this a trial run and found I needed some tweaks to test.sh and the Makefile in order to make it work on VPATH; mainly replace ./ with `dirname $0` in a couple

Re: [HACKERS] WAL replay bugs

2014-07-25 Thread Kyotaro HORIGUCHI
Hi, I'm not so confident whether it's the time to do this... I mark this as 'Ready for Committer' since no additional comment or objection was put by others on this patch. After all, I have no more comment about this patch. I will mark this as 'Ready for committer' unless no comment comes

Re: [HACKERS] WAL replay bugs

2014-07-22 Thread Kyotaro HORIGUCHI
Hello, Although I doubt necessity of the flexibility seeing the current testing framework, I don't have so strong objection about that. Nevertheless, perhaps you are appreciated to put a notice on.. README or somewhere. Hm, well... Fine, I added it in this updated series. Thank you for

Re: [HACKERS] WAL replay bugs

2014-07-14 Thread Kyotaro HORIGUCHI
Hello, Let me apologize for continuing the discussion even though the deadline is approaching. At Fri, 11 Jul 2014 09:49:55 +0900, Michael Paquier michael.paqu...@gmail.com wrote in cab7npqtjezoz-fotibsejyg0eabngx2plqywodchyfxzfqy...@mail.gmail.com Updated patches attached. On Thu, Jul 10,

Re: [HACKERS] WAL replay bugs

2014-07-10 Thread Kyotaro HORIGUCHI
Hello, The new patch looks good for me. The usage of this is a bit irregular as a (extension) module but it is the nature of this 'contrib'. The rearranged page type detection logic seems neater and keeps to work as intended. This logic will be changed after the new page type detection scheme

Re: [HACKERS] WAL replay bugs

2014-07-04 Thread Kyotaro HORIGUCHI
Hello, thank you for considering my comments, including somewhat impractical ones. I'll have a look at the latest patch sooner. At Fri, 4 Jul 2014 15:29:51 +0900, Michael Paquier michael.paqu...@gmail.com wrote in cab7npqt_fs_3jlnhywc6nzej4sbl6dgslfvcg0jbukgjep9...@mail.gmail.com OK, I have

Re: [HACKERS] WAL replay bugs

2014-07-04 Thread Kyotaro HORIGUCHI
Hello, At Thu, 3 Jul 2014 14:48:50 +0900, Michael Paquier michael.paqu...@gmail.com wrote in cab7npqq2y3qkapasac6oxxqtwbvkkxcrftua0w+dx-j3i-l...@mail.gmail.com TODO ... Each type of page can be confirmed by the following way *if* its type is previously *hinted* except for gin.

Re: [HACKERS] WAL replay bugs

2014-07-03 Thread Tom Lane
Michael Paquier michael.paqu...@gmail.com writes: On Wed, Jul 2, 2014 at 5:32 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Pehaps this is showing that no tidy or generalized way to tell what a page is used for. Many of the modules which have their own page format has a magic

Re: [HACKERS] WAL replay bugs

2014-07-03 Thread Michael Paquier
On Thu, Jul 3, 2014 at 3:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Michael Paquier michael.paqu...@gmail.com writes: On Wed, Jul 2, 2014 at 5:32 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Pehaps this is showing that no tidy or generalized way to tell what a page is used for.

Re: [HACKERS] WAL replay bugs

2014-07-03 Thread Kyotaro HORIGUCHI
Hello, This is the additional comments for other part. I haven't see significant defect in the code so far. = bufcapt.c: - buffer_capture_remember() or so. Pages for unlogged tables are avoided to be written taking advantage that the lsn for such pages stays 0/0. I'd like to see a

Re: [HACKERS] WAL replay bugs

2014-07-02 Thread Kyotaro HORIGUCHI
Hello, Thanks for your comments. Looking forward to seeing some more input. Thank you. bufcapt.c was a poser. bufcapt.c: 326 memcpy(tail, page[BLCKSZ - 2], 2); This seems duzzling.. Isn't *(uint16*)(page[BLCKSZ - 2]) applicable? bufcapt.c: 331 else if (PageGetSpecial Generally

Re: [HACKERS] WAL replay bugs

2014-07-02 Thread Michael Paquier
TODO On Wed, Jul 2, 2014 at 5:32 PM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: bufcapt.c: 326 memcpy(tail, page[BLCKSZ - 2], 2); This seems duzzling.. Isn't *(uint16*)(page[BLCKSZ - 2]) applicable? Well yes it is. Pehaps this is showing that no tidy or generalized way to

Re: [HACKERS] WAL replay bugs

2014-07-01 Thread Kyotaro HORIGUCHI
Hello, I had a look on this patch. Let me show you some comments about the README, Makefile and buffer_capture_cmp of the second part for the present. A continuation of this comment would be seen later.. - contrib/buffer_capture_cmp/README - 'contains' seems duplicate in the first paragraph.

Re: [HACKERS] WAL replay bugs

2014-06-27 Thread Michael Paquier
On Fri, Jun 27, 2014 at 2:57 PM, Michael Paquier michael.paqu...@gmail.com wrote: Here are rebased patches, their was a conflict with a recent commit in contrib/pg_upgrade. I am resending patch 2 as it contained a rebase conflict not correctly resolved (Thanks Alvaro). Regards, -- Michael

Re: [HACKERS] WAL replay bugs

2014-06-18 Thread Robert Haas
On Tue, Jun 17, 2014 at 5:40 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Jun 18, 2014 at 1:40 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jun 2, 2014 at 8:55 AM, Michael Paquier michael.paqu...@gmail.com wrote: I'm not sure if this is reasonably possible, but one

Re: [HACKERS] WAL replay bugs

2014-06-17 Thread Robert Haas
On Mon, Jun 2, 2014 at 8:55 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: And here is the tool itself. It consists of two parts: 1. Modifications to the backend to write the page images 2. A

Re: [HACKERS] WAL replay bugs

2014-06-17 Thread Michael Paquier
On Wed, Jun 18, 2014 at 1:40 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jun 2, 2014 at 8:55 AM, Michael Paquier michael.paqu...@gmail.com wrote: I'm not sure if this is reasonably possible, but one thing that would make this tool a whole lot easier to use would be if you could make

Re: [HACKERS] WAL replay bugs

2014-06-13 Thread Heikki Linnakangas
On 06/13/2014 10:14 AM, Michael Paquier wrote: On Mon, Jun 2, 2014 at 9:55 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Perhaps there are parts of what is proposed here that could be made more

Re: [HACKERS] WAL replay bugs

2014-06-13 Thread Michael Paquier
On Fri, Jun 13, 2014 at 4:48 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 06/13/2014 10:14 AM, Michael Paquier wrote: On Mon, Jun 2, 2014 at 9:55 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas hlinnakan...@vmware.com

Re: [HACKERS] WAL replay bugs

2014-06-13 Thread Michael Paquier
On Fri, Jun 13, 2014 at 4:50 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Fri, Jun 13, 2014 at 4:48 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 06/13/2014 10:14 AM, Michael Paquier wrote: On Mon, Jun 2, 2014 at 9:55 PM, Michael Paquier michael.paqu...@gmail.com wrote:

Re: [HACKERS] WAL replay bugs

2014-06-02 Thread Michael Paquier
On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: And here is the tool itself. It consists of two parts: 1. Modifications to the backend to write the page images 2. A post-processing tool to compare the logged images between master and standby. Having that

Re: [HACKERS] WAL replay bugs

2014-04-23 Thread Heikki Linnakangas
On 04/17/2014 07:59 PM, Heikki Linnakangas wrote: On 04/08/2014 06:41 AM, Michael Paquier wrote: On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I've been playing with a little hack that records a before and after image of every page modification that is

Re: [HACKERS] WAL replay bugs

2014-04-18 Thread Peter Geoghegan
On Thu, Apr 17, 2014 at 10:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: Any objections to changing those two? Not here. I've always suspected #2 was going to bite us someday anyway. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] WAL replay bugs

2014-04-17 Thread Heikki Linnakangas
On 04/08/2014 06:41 AM, Michael Paquier wrote: On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I've been playing with a little hack that records a before and after image of every page modification that is WAL-logged, and writes the images to a file along with

Re: [HACKERS] WAL replay bugs

2014-04-17 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: Two things that are not bugs, but I'd like to change just to make this tool easier to maintain, and to generally clean things up: 1. When creating a sequence, we first use simple_heap_insert() to insert the sequence tuple, which creates a

Re: [HACKERS] WAL replay bugs

2014-04-10 Thread sachin kotwal
I executed given steps many times to produce this bug. But still I unable to hit this bug. I used attached scripts to produce this bug. Can I get scripts to produce this bug? wal_replay_bug.sh http://postgresql.1045698.n5.nabble.com/file/n5799512/wal_replay_bug.sh - Thanks and

Re: [HACKERS] WAL replay bugs

2014-04-10 Thread Heikki Linnakangas
On 04/10/2014 10:52 AM, sachin kotwal wrote: I executed given steps many times to produce this bug. But still I unable to hit this bug. I used attached scripts to produce this bug. Can I get scripts to produce this bug? wal_replay_bug.sh

Re: [HACKERS] WAL replay bugs

2014-04-10 Thread Sachin D. Kotwal
On Thu, Apr 10, 2014 at 6:21 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 04/10/2014 10:52 AM, sachin kotwal wrote: I executed given steps many times to produce this bug. But still I unable to hit this bug. I used attached scripts to produce this bug. Can I get scripts to

Re: [HACKERS] WAL replay bugs

2014-04-07 Thread Josh Berkus
On 04/07/2014 02:16 PM, Heikki Linnakangas wrote: I've been playing with a little hack that records a before and after image of every page modification that is WAL-logged, and writes the images to a file along with the LSN of the corresponding WAL record. I set up a master-standby replication

Re: [HACKERS] WAL replay bugs

2014-04-07 Thread Michael Paquier
On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I've been playing with a little hack that records a before and after image of every page modification that is WAL-logged, and writes the images to a file along with the LSN of the corresponding WAL record. I set