Re: [HACKERS] WAL format changes

2012-06-25 Thread Amit Kapila
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Tom Lane Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: So I think we should change pg_resetxlog -l option to take a WAL file name as argument, and fix pg_upgrade accordingly.

Re: [HACKERS] WAL format changes

2012-06-24 Thread Heikki Linnakangas
Ok, committed all the WAL format changes now. On 19.06.2012 18:57, Robert Haas wrote: Should we keep the old representation in the replication protocol messages? That would make it simpler to write a client that works with different server versions (like pg_receivexlog). Or, while we're at it,

Re: [HACKERS] WAL format changes

2012-06-24 Thread Simon Riggs
On 24 June 2012 17:24, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok, committed all the WAL format changes now. Nice! --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] WAL format changes

2012-06-20 Thread Magnus Hagander
On Tue, Jun 19, 2012 at 5:57 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Well, that was easier than I thought. Attached is a patch to make XLogRecPtr a uint64, on top of my other WAL format patches.

Re: [HACKERS] WAL format changes

2012-06-20 Thread Fujii Masao
On Wed, Jun 20, 2012 at 8:19 PM, Magnus Hagander mag...@hagander.net wrote: On Tue, Jun 19, 2012 at 5:57 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Well, that was easier than I thought. Attached is

Re: [HACKERS] WAL format changes

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 20:43, Fujii Masao wrote: On Wed, Jun 20, 2012 at 8:19 PM, Magnus Hagandermag...@hagander.net wrote: On Tue, Jun 19, 2012 at 5:57 PM, Robert Haasrobertmh...@gmail.com wrote: On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Well,

Re: [HACKERS] WAL format changes

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote: On 18.06.2012 21:08, Heikki Linnakangas wrote: On 18.06.2012 21:00, Robert Haas wrote: On Thu, Jun 14, 2012 at 5:58 PM, Andres Freundand...@2ndquadrant.com wrote: 1. Use a 64-bit segment number, instead of the log/seg

Re: [HACKERS] WAL format changes

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Well, that was easier than I thought. Attached is a patch to make XLogRecPtr a uint64, on top of my other WAL format patches. I think we should go ahead with this. +1. The LSNs on pages are still

Re: [HACKERS] WAL format changes

2012-06-19 Thread Heikki Linnakangas
On 19.06.2012 18:46, Andres Freund wrote: On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote: Well, that was easier than I thought. Attached is a patch to make XLogRecPtr a uint64, on top of my other WAL format patches. I think we should go ahead with this. Cool. You plan to merge

Re: [HACKERS] WAL format changes

2012-06-19 Thread Andres Freund
Hi, On Wednesday, June 20, 2012 12:24:54 AM Heikki Linnakangas wrote: On 19.06.2012 18:46, Andres Freund wrote: On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote: Well, that was easier than I thought. Attached is a patch to make XLogRecPtr a uint64, on top of my other WAL

Re: [HACKERS] WAL format changes

2012-06-18 Thread Robert Haas
On Thu, Jun 14, 2012 at 5:58 PM, Andres Freund and...@2ndquadrant.com wrote: 1. Use a 64-bit segment number, instead of the log/seg combination. And don't waste the last segment on each logical 4 GB log file. The concept of a logical log file is now completely gone. XLogRecPtr is unchanged,

Re: [HACKERS] WAL format changes

2012-06-18 Thread Heikki Linnakangas
On 18.06.2012 21:00, Robert Haas wrote: On Thu, Jun 14, 2012 at 5:58 PM, Andres Freundand...@2ndquadrant.com wrote: 1. Use a 64-bit segment number, instead of the log/seg combination. And don't waste the last segment on each logical 4 GB log file. The concept of a logical log file is now

Re: [HACKERS] WAL format changes

2012-06-18 Thread Andres Freund
On Monday, June 18, 2012 08:08:14 PM Heikki Linnakangas wrote: On 18.06.2012 21:00, Robert Haas wrote: On Thu, Jun 14, 2012 at 5:58 PM, Andres Freundand...@2ndquadrant.com wrote: 1. Use a 64-bit segment number, instead of the log/seg combination. And don't waste the last segment on each

Re: [HACKERS] WAL format changes

2012-06-18 Thread Robert Haas
On Mon, Jun 18, 2012 at 2:08 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 18.06.2012 21:00, Robert Haas wrote: On Thu, Jun 14, 2012 at 5:58 PM, Andres Freundand...@2ndquadrant.com  wrote: 1. Use a 64-bit segment number, instead of the log/seg combination. And don't

Re: [HACKERS] WAL format changes

2012-06-18 Thread Heikki Linnakangas
On 18.06.2012 21:13, Andres Freund wrote: On Monday, June 18, 2012 08:08:14 PM Heikki Linnakangas wrote: The page header contains an XLogRecPtr (LSN), so if we change it we'll have to deal with pg_upgrade. I guess we could still keep XLogRecPtr around as the on-disk representation, and convert

Re: [HACKERS] WAL format changes

2012-06-18 Thread Andres Freund
On Monday, June 18, 2012 08:32:54 PM Heikki Linnakangas wrote: On 18.06.2012 21:13, Andres Freund wrote: On Monday, June 18, 2012 08:08:14 PM Heikki Linnakangas wrote: The page header contains an XLogRecPtr (LSN), so if we change it we'll have to deal with pg_upgrade. I guess we could still

Re: [HACKERS] WAL format changes

2012-06-18 Thread Heikki Linnakangas
On 18.06.2012 21:45, Andres Freund wrote: On Monday, June 18, 2012 08:32:54 PM Heikki Linnakangas wrote: On 18.06.2012 21:13, Andres Freund wrote: On Monday, June 18, 2012 08:08:14 PM Heikki Linnakangas wrote: The page header contains an XLogRecPtr (LSN), so if we change it we'll have to deal

Re: [HACKERS] WAL format changes

2012-06-18 Thread Andres Freund
On Monday, June 18, 2012 09:19:48 PM Heikki Linnakangas wrote: On 18.06.2012 21:45, Andres Freund wrote: On Monday, June 18, 2012 08:32:54 PM Heikki Linnakangas wrote: On 18.06.2012 21:13, Andres Freund wrote: On Monday, June 18, 2012 08:08:14 PM Heikki Linnakangas wrote: The page header

Re: [HACKERS] WAL format changes

2012-06-14 Thread Andres Freund
On Thursday, June 14, 2012 11:01:42 PM Heikki Linnakangas wrote: As I threatened earlier (http://archives.postgresql.org/message-id/4fd0b1ab.3090...@enterprisedb.co m), here are three patches that change the WAL format. The goal is to change the format so that when you're inserting a WAL

Re: [HACKERS] WAL format changes

2012-06-14 Thread Andres Freund
On Thursday, June 14, 2012 11:01:42 PM Heikki Linnakangas wrote: As I threatened earlier (http://archives.postgresql.org/message-id/4fd0b1ab.3090...@enterprisedb.co m), here are three patches that change the WAL format. The goal is to change the format so that when you're inserting a WAL

Re: [HACKERS] WAL format

2009-12-09 Thread Greg Stark
On Mon, Dec 7, 2009 at 8:48 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@commandprompt.com writes: Heikki Linnakangas wrote: - at the end of WAL segment, when there's not enough space to write the next WAL record, always write an XLOG SWITCH record to fill the rest of the

Re: [HACKERS] WAL format

2009-12-07 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: In particular I wonder why we bother with the page headers. Since we re-use the file for a new segment, without overwriting the old contents, it seems like we would need to do *something* to reliably determine when we've hit the

Re: [HACKERS] WAL format

2009-12-07 Thread Alvaro Herrera
Heikki Linnakangas wrote: - at the end of WAL segment, when there's not enough space to write the next WAL record, always write an XLOG SWITCH record to fill the rest of the segment. What happens if a record is larger than a WAL segment? For example, what if I insert a 16 MB+ datum into a

Re: [HACKERS] WAL format

2009-12-07 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: In particular I wonder why we bother with the page headers. A much simpler format would be: - get rid of page headers, except for the header at the beginning of each WAL segment - get rid of continuation records - at the end of

Re: [HACKERS] WAL format

2009-12-07 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: In particular I wonder why we bother with the page headers. Since we re-use the file for a new segment, without overwriting the old contents, it seems like we would need to do

Re: [HACKERS] WAL format

2009-12-07 Thread Andres Freund
On Monday 07 December 2009 21:44:37 Tom Lane wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: In particular I wonder why we bother with the page headers. Since we re-use the file for a new segment, without overwriting

Re: [HACKERS] WAL format

2009-12-07 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Heikki Linnakangas wrote: - at the end of WAL segment, when there's not enough space to write the next WAL record, always write an XLOG SWITCH record to fill the rest of the segment. What happens if a record is larger than a WAL segment? For

Re: [HACKERS] WAL format

2009-12-07 Thread Simon Riggs
On Mon, 2009-12-07 at 21:28 +0200, Heikki Linnakangas wrote: The changes to ReadRecord in the streaming replication patch feel a bit awkward, because it has to work around the fact that WAL is streamed as a stream of bytes, but ReadRecord works one page at a time. I'd like to replace

Re: [HACKERS] WAL format

2009-12-07 Thread Heikki Linnakangas
Tom Lane wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: In particular I wonder why we bother with the page headers. Since we re-use the file for a new segment, without overwriting the old contents, it seems like we

Re: [HACKERS] WAL format

2009-12-07 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: In particular I wonder why we bother with the page headers. A much simpler format would be: - get rid of page headers, except for the header at the beginning of each WAL segment - get rid of continuation records

Re: [HACKERS] WAL format

2009-12-07 Thread Fujii Masao
On Tue, Dec 8, 2009 at 10:28 AM, Simon Riggs si...@2ndquadrant.com wrote: If this was earlier in the release cycle, I'd feel happier. 2.5 months before beta is the wrong time to re-design the crash recovery data format, especially because its only a bit awkward. We're bound to break something

<    1   2