Re: [HACKERS] increasing the default WAL segment size

2017-09-19 Thread Andres Freund
Hi, On 2017-09-14 11:31:33 +0530, Beena Emerson wrote: > The change looks good and is working as expected. > PFA the updated patch after running pgindent. I've pushed this version. Yay! Thanks for the work Beena, everyone! The only change I made is to run the pg_upgrade tests with a 1 MB

Re: [HACKERS] increasing the default WAL segment size

2017-09-13 Thread Andres Freund
Hi, On 2017-09-06 20:24:16 +0530, Beena Emerson wrote: > > - pg_standby's RetrieveWALSegSize() does too much for it's name. It > > seems quite weird that a function named that way has the section below > > "/* check if clean up is necessary */" > > we set 2 cleanup related variables once

Re: [HACKERS] increasing the default WAL segment size

2017-09-05 Thread Andres Freund
Hi, I was looking to commit this, but the changes I made ended up being pretty large. Here's what I changed in the attached: - split GUC_UNIT_BYTE into a separate commit, squashed rest - renamed GUC_UNIT_BYT to GUC_UNIT_BYTE, don't see why we'd have such a weird abbreviation? - bumped control

Re: [HACKERS] increasing the default WAL segment size

2017-08-29 Thread Andres Freund
Hi, On 2017-08-23 12:13:15 +0530, Beena Emerson wrote: > >> + /* > >> + * The calculation of XLOGbuffers requires the run-time > >> parameter > >> + * XLogSegSize which is set from the control file. This > >> value is > >> + * required to

Re: [HACKERS] increasing the default WAL segment size

2017-08-15 Thread Andres Freund
Hi, Personally I find the split between 03 and 04 and their naming a bit confusing. I'd rather just merge them. These patches need a rebase, they don't apply anymore. On 2017-07-06 12:04:12 +0530, Beena Emerson wrote: > @@ -4813,6 +4836,18 @@ XLOGShmemSize(void) > { > char

Re: [HACKERS] increasing the default WAL segment size

2017-07-06 Thread Beena Emerson
On Thu, Jul 6, 2017 at 3:21 PM, tushar wrote: > On 07/06/2017 12:04 PM, Beena Emerson wrote: >> >> The 04-initdb-walsegsize_v2.patch has the following improvements: >> - Rebased over new 03 patch >> - Pass the wal-segsize intidb option as command-line option rathern

Re: [HACKERS] increasing the default WAL segment size

2017-07-06 Thread tushar
On 07/06/2017 12:04 PM, Beena Emerson wrote: The 04-initdb-walsegsize_v2.patch has the following improvements: - Rebased over new 03 patch - Pass the wal-segsize intidb option as command-line option rathern than in an environment variable. - Since new function check_wal_size had only had two

Re: [HACKERS] increasing the default WAL segment size

2017-07-06 Thread Beena Emerson
Hello, On Tue, Mar 28, 2017 at 1:06 AM, Beena Emerson wrote: > Hello, > > On Sat, Mar 25, 2017 at 10:32 PM, Peter Eisentraut > wrote: >> >> At this point, I suggest splitting this patch up into several >> potentially less

Re: [HACKERS] increasing the default WAL segment size

2017-04-08 Thread David Steele
On 4/7/17 2:59 AM, Beena Emerson wrote: > I ran tests and following are the details: > > Machine details: > Architecture: ppc64le > Byte Order:Little Endian > CPU(s):192 > On-line CPU(s) list: 0-191 > Thread(s) per core:8 > Core(s) per socket:1 >

Re: [HACKERS] increasing the default WAL segment size

2017-04-07 Thread Beena Emerson
I ran tests and following are the details: Machine details: Architecture: ppc64le Byte Order:Little Endian CPU(s):192 On-line CPU(s) list: 0-191 Thread(s) per core:8 Core(s) per socket:1 Socket(s): 24 NUMA node(s): 4 Model:

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Beena Emerson
On Fri, Apr 7, 2017 at 2:35 AM, Tomas Vondra wrote: > On 04/06/2017 08:33 PM, David Steele wrote: >> >> >> I'm in favor of 16,64,256,1024. >> >> > I don't see a particular reason for this, TBH. The sweet spots will be > likely dependent hardware / OS configuration

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Beena Emerson
Hello, On Wed, Apr 5, 2017 at 6:06 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > > (Roughly speaking, to get started, this would mean compiling with > --with-wal-segsize 16, 32, 64, 128, 256, run make check-world both > sequentially and in parallel, and take note of a)

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread David Steele
On 4/6/17 6:52 PM, Tomas Vondra wrote: > On 04/06/2017 11:45 PM, David Steele wrote: >> >> How many people in the field are running custom builds of >> Postgres? And of those, how many have changed the WAL segment size? >> I've never encountered a non-standard segment size or talked to anyone >>

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Tomas Vondra
On 04/06/2017 11:45 PM, David Steele wrote: On 4/6/17 5:05 PM, Tomas Vondra wrote: On 04/06/2017 08:33 PM, David Steele wrote: On 4/5/17 7:29 AM, Simon Riggs wrote: 2. It's not clear to me the advantage of being able to pick varying filesizes. I see great disadvantage in having too many

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread David Steele
On 4/6/17 5:05 PM, Tomas Vondra wrote: > On 04/06/2017 08:33 PM, David Steele wrote: >> On 4/5/17 7:29 AM, Simon Riggs wrote: >> >>> 2. It's not clear to me the advantage of being able to pick varying >>> filesizes. I see great disadvantage in having too many options, which >>> greatly increases

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Tomas Vondra
On 04/06/2017 08:33 PM, David Steele wrote: On 4/5/17 7:29 AM, Simon Riggs wrote: On 5 April 2017 at 06:04, Beena Emerson wrote: The WALfilename - LSN mapping disruption for higher values you mean? Is there anything else I have missed? I see various issues raised

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread David Steele
On 4/5/17 7:29 AM, Simon Riggs wrote: > On 5 April 2017 at 06:04, Beena Emerson wrote: >> >> The WALfilename - LSN mapping disruption for higher values you mean? Is >> there anything else I have missed? > > I see various issues raised but not properly addressed > > 1.

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Peter Eisentraut
On 4/6/17 07:13, Beena Emerson wrote: > Does the options 16, 64 and 1024 seem good. > We can remove sizes below 16 as most have agreed and as per the > discussion, 64MB and 1GB seems favoured. We could probably allow 32MB > since it was an already allowed size? I don't see the need to remove

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Beena Emerson
Hello, On Wed, Apr 5, 2017 at 6:06 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote:e format and expand the range? > > > I don't think me saying it felt a bit slow around 256 MB is a proper > technical analysis that should lead to the conclusion that that upper > limit should be 128

Re: [HACKERS] increasing the default WAL segment size

2017-04-06 Thread Beena Emerson
Hello, On Wed, Apr 5, 2017 at 4:59 PM, Simon Riggs wrote: > On 5 April 2017 at 06:04, Beena Emerson wrote: > > I see various issues raised but not properly addressed > > 1. we would need to drop support for segment sizes < 16MB unless we > adopt

Re: [HACKERS] increasing the default WAL segment size

2017-04-05 Thread Simon Riggs
On 5 April 2017 at 08:36, Peter Eisentraut wrote: > On 4/5/17 06:04, Beena Emerson wrote: >> I suggest the next step is to dial up the allowed segment size in >> configure and run some tests about what a reasonable maximum value could >> be. I did a

Re: [HACKERS] increasing the default WAL segment size

2017-04-05 Thread Peter Eisentraut
On 4/5/17 06:04, Beena Emerson wrote: > I suggest the next step is to dial up the allowed segment size in > configure and run some tests about what a reasonable maximum value could > be. I did a little bit of that, but somewhere around 256 MB, things got > really slow. > > >

Re: [HACKERS] increasing the default WAL segment size

2017-04-05 Thread Simon Riggs
On 5 April 2017 at 06:04, Beena Emerson wrote: >> >> No commitment yet to increasing wal-segsize in the way this patch has >> >> it. >> >> >> > >> > What part of patch you don't like and do you have any suggestions to >> > improve the same? >> >> I think there are still

Re: [HACKERS] increasing the default WAL segment size

2017-04-05 Thread Simon Riggs
On 4 April 2017 at 22:47, Amit Kapila wrote: > On Wed, Apr 5, 2017 at 3:36 AM, Simon Riggs wrote: >> On 27 March 2017 at 15:36, Beena Emerson wrote: >> >>> 02-increase-max-wal-segsize.patch - Increases the wal-segsize and

Re: [HACKERS] increasing the default WAL segment size

2017-04-05 Thread Beena Emerson
Hello, On Wed, Apr 5, 2017 at 9:29 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 4/4/17 22:47, Amit Kapila wrote: > >> Committed first part to allow internal representation change (only). > >> > >> No commitment yet to increasing wal-segsize in the way this patch has > it.

Re: [HACKERS] increasing the default WAL segment size

2017-04-04 Thread Peter Eisentraut
On 4/4/17 22:47, Amit Kapila wrote: >> Committed first part to allow internal representation change (only). >> >> No commitment yet to increasing wal-segsize in the way this patch has it. >> > > What part of patch you don't like and do you have any suggestions to > improve the same? I think

Re: [HACKERS] increasing the default WAL segment size

2017-04-04 Thread Amit Kapila
On Wed, Apr 5, 2017 at 3:36 AM, Simon Riggs wrote: > On 27 March 2017 at 15:36, Beena Emerson wrote: > >> 02-increase-max-wal-segsize.patch - Increases the wal-segsize and changes >> the internal representation of max_wal_size and min_wal_size to

Re: [HACKERS] increasing the default WAL segment size

2017-04-04 Thread Simon Riggs
On 27 March 2017 at 15:36, Beena Emerson wrote: > 02-increase-max-wal-segsize.patch - Increases the wal-segsize and changes > the internal representation of max_wal_size and min_wal_size to mb. Committed first part to allow internal representation change (only). No

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Beena Emerson
Hello, On Fri, Mar 31, 2017 at 11:20 AM, Kuntal Ghosh wrote: > On Fri, Mar 31, 2017 at 10:40 AM, Beena Emerson > wrote: > > On 30 Mar 2017 15:10, "Kuntal Ghosh" wrote: > > > I do not think a generalised function

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 10:40 AM, Beena Emerson wrote: > On 30 Mar 2017 15:10, "Kuntal Ghosh" wrote: >> 03-modify-tools.patch - Makes XLogSegSize into a variable, currently set >> as >> XLOG_SEG_SIZE and modifies the tools to fetch the size

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Beena Emerson
Hello, Thanks for testing my patch. On 30 Mar 2017 15:10, "Kuntal Ghosh" wrote: On Tue, Mar 28, 2017 at 1:06 AM, Beena Emerson wrote: > On Sat, Mar 25, 2017 at 10:32 PM, Peter Eisentraut > wrote: >> >> At

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Kuntal Ghosh
On Tue, Mar 28, 2017 at 1:06 AM, Beena Emerson wrote: > On Sat, Mar 25, 2017 at 10:32 PM, Peter Eisentraut > wrote: >> >> At this point, I suggest splitting this patch up into several >> potentially less controversial pieces. >> >> One

Re: [HACKERS] increasing the default WAL segment size

2017-03-27 Thread Beena Emerson
Hello, On Sat, Mar 25, 2017 at 10:32 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > At this point, I suggest splitting this patch up into several > potentially less controversial pieces. > > One big piece is that we currently don't support segment sizes larger > than 64 GB,

Re: [HACKERS] increasing the default WAL segment size

2017-03-27 Thread Simon Riggs
On 25 March 2017 at 17:02, Peter Eisentraut wrote: > At this point, I suggest splitting this patch up into several > potentially less controversial pieces. > > One big piece is that we currently don't support segment sizes larger > than 64 GB, for various

Re: [HACKERS] increasing the default WAL segment size

2017-03-25 Thread Peter Eisentraut
At this point, I suggest splitting this patch up into several potentially less controversial pieces. One big piece is that we currently don't support segment sizes larger than 64 GB, for various internal arithmetic reasons. Your patch appears to address that. So I suggest isolating that.

Re: [HACKERS] increasing the default WAL segment size

2017-03-25 Thread Stephen Frost
Peter, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 3/24/17 08:18, Stephen Frost wrote: > > Beyond that, this also bakes in an assumption that we would then require > > access to a database > > That is a good point, but then any change to the naming whatsoever will > create

Re: [HACKERS] increasing the default WAL segment size

2017-03-25 Thread Stephen Frost
Peter, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 3/24/17 19:13, David Steele wrote: > > Behavior for the current default of 16MB is unchanged, and all other > > sizes go through a logical progression. > > Just at a glance, without analyzing the math behind it, this

Re: [HACKERS] increasing the default WAL segment size

2017-03-25 Thread Peter Eisentraut
On 3/24/17 08:18, Stephen Frost wrote: > Peter, > > * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: >> There is a function for that. > [...] >> There is not a function for that, but there could be one. > > I'm not sure you've really considered what you're suggesting here. Create a

Re: [HACKERS] increasing the default WAL segment size

2017-03-25 Thread Peter Eisentraut
On 3/24/17 19:13, David Steele wrote: > Behavior for the current default of 16MB is unchanged, and all other > sizes go through a logical progression. Just at a glance, without analyzing the math behind it, this scheme seems super confusing. > > 1GB: > 00010040 >

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread David Steele
Hi Robert, On 3/24/17 3:00 PM, Robert Haas wrote: On Wed, Mar 22, 2017 at 6:05 PM, David Steele wrote: Wait, really? I thought you abandoned this approach because there's then no principled way to handle WAL segments of less than the default size. I did say that, but I

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread Robert Haas
On Wed, Mar 22, 2017 at 6:05 PM, David Steele wrote: >> Wait, really? I thought you abandoned this approach because there's >> then no principled way to handle WAL segments of less than the default >> size. > > I did say that, but I thought I had hit on a compromise. > >

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread Beena Emerson
Hello, On Wed, Mar 22, 2017 at 9:41 PM, Kuntal Ghosh wrote: > On Wed, Mar 22, 2017 at 3:14 PM, Beena Emerson > wrote: > > PFA an updated patch which fixes a minor bug I found. It only increases > the > > string size in pretty_wal_size

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread David Steele
On 3/23/17 4:45 PM, Peter Eisentraut wrote: On 3/22/17 17:33, David Steele wrote: I think if we don't change the default size it's very unlikely I would use alternate WAL segment sizes or recommend that anyone else does, at least in v10. I simply don't think it would get the level of testing

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread Stephen Frost
Jeff, * Jeff Janes (jeff.ja...@gmail.com) wrote: > On Thu, Mar 23, 2017 at 1:45 PM, Peter Eisentraut < > peter.eisentr...@2ndquadrant.com> wrote: > > On 3/22/17 17:33, David Steele wrote: > > > and I doubt that most tool writers would be quick to > > > add support for a feature that very few

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread David Steele
On 3/24/17 12:27 AM, Peter Eisentraut wrote: On 3/23/17 16:58, Stephen Frost wrote: The backup tools need to also get the LSN from the pg_stop_backup and verify that they have the WAL file associated with that LSN. There is a function for that. They also need to make sure that they have all

Re: [HACKERS] increasing the default WAL segment size

2017-03-24 Thread Stephen Frost
Peter, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > There is a function for that. [...] > There is not a function for that, but there could be one. I'm not sure you've really considered what you're suggesting here. We need to to make sure we have every file between two LSNs.

Re: [HACKERS] increasing the default WAL segment size

2017-03-23 Thread Peter Eisentraut
On 3/23/17 21:47, Jeff Janes wrote: > I have a pg_restore which predicts the file 5 files ahead of the one it > was asked for, and initiates a pre-fetch and decompression of it. Then > it delivers the file it was asked for, either by pulling it out of the > pre-staging area set up by the N-5th

Re: [HACKERS] increasing the default WAL segment size

2017-03-23 Thread Peter Eisentraut
On 3/23/17 16:58, Stephen Frost wrote: > The backup tools need to also get the LSN from the pg_stop_backup and > verify that they have the WAL file associated with that LSN. There is a function for that. > They also > need to make sure that they have all of the WAL files between the > starting

Re: [HACKERS] increasing the default WAL segment size

2017-03-23 Thread Jeff Janes
On Thu, Mar 23, 2017 at 1:45 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 3/22/17 17:33, David Steele wrote: > > > and I doubt that most tool writers would be quick to > > add support for a feature that very few people (if any) use. > > I'm not one of those tool writers,

Re: [HACKERS] increasing the default WAL segment size

2017-03-23 Thread Stephen Frost
Peter, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 3/22/17 17:33, David Steele wrote: > > and I doubt that most tool writers would be quick to > > add support for a feature that very few people (if any) use. > > I'm not one of those tool writers, although I have written my

Re: [HACKERS] increasing the default WAL segment size

2017-03-23 Thread Peter Eisentraut
On 3/22/17 17:33, David Steele wrote: > I think if we don't change the default size it's very unlikely I would > use alternate WAL segment sizes or recommend that anyone else does, at > least in v10. > > I simply don't think it would get the level of testing required to be > production worthy

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
David, * David Steele (da...@pgmasters.net) wrote: > On 3/22/17 3:45 PM, Robert Haas wrote: > >On Wed, Mar 22, 2017 at 3:24 PM, David Steele wrote: > >>>One of the reasons to go with the LSN is that we would actually be > >>>maintaining what happens when the WAL files are

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread David Steele
Hi Robert, On 3/22/17 3:45 PM, Robert Haas wrote: On Wed, Mar 22, 2017 at 3:24 PM, David Steele wrote: One of the reasons to go with the LSN is that we would actually be maintaining what happens when the WAL files are 16MB in size. David's initial expectation was this

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread David Steele
On 3/22/17 3:39 PM, Peter Eisentraut wrote: On 3/22/17 15:37, Peter Eisentraut wrote: If changing WAL sizes catches on, I do think we should keep thinking about a new format for a future release, I think that means that I'm skeptical about changing the default size right now. I think if we

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
* Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > The question is, which property is more useful to preserve: matching > LSN, or having a mostly consecutive numbering. > > Actually, I would really really like to have both, but if I had to pick > one, I'd lean 55% toward consecutive

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Robert Haas
On Wed, Mar 22, 2017 at 3:24 PM, David Steele wrote: >> One of the reasons to go with the LSN is that we would actually be >> maintaining what happens when the WAL files are 16MB in size. >> >> David's initial expectation was this for 64MB WAL files: >> >>

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Peter Eisentraut
On 3/22/17 15:37, Peter Eisentraut wrote: > If changing WAL sizes catches on, I do think we should keep thinking > about a new format for a future release, I think that means that I'm skeptical about changing the default size right now. -- Peter Eisentraut

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Peter Eisentraut
On 3/22/17 15:09, Stephen Frost wrote: > David's initial expectation was this for 64MB WAL files: > > 00010040 > 00010080 > 000100CO > 00010001 > > Which both matches the LSN *and* keeps the file names the same when > they're 16MB.

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
* David Steele (da...@pgmasters.net) wrote: > On 3/22/17 3:09 PM, Stephen Frost wrote: > >* Robert Haas (robertmh...@gmail.com) wrote: > >>On Wed, Mar 22, 2017 at 1:49 PM, Stephen Frost wrote: > >>>Then perhaps we do need to be thinking of moving this to PG11 instead of >

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread David Steele
On 3/22/17 3:09 PM, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Wed, Mar 22, 2017 at 1:49 PM, Stephen Frost wrote: Then perhaps we do need to be thinking of moving this to PG11 instead of exposing an option that users will start to use which will

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Mar 22, 2017 at 1:49 PM, Stephen Frost wrote: > > Then perhaps we do need to be thinking of moving this to PG11 instead of > > exposing an option that users will start to use which will result in WAL > > naming

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Robert Haas
On Wed, Mar 22, 2017 at 1:49 PM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> On Wed, Mar 22, 2017 at 1:22 PM, Stephen Frost wrote: >> > To put this in another light, had this issue been brought up post >> > feature-freeze, your

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread David G. Johnston
On Wed, Mar 22, 2017 at 9:51 AM, Stephen Frost wrote: > Robert, > > * Robert Haas (robertmh...@gmail.com) wrote: > > On Wed, Mar 22, 2017 at 12:22 PM, Stephen Frost > wrote: > > > While I understand that you'd like to separate the concerns between > > >

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Mar 22, 2017 at 1:22 PM, Stephen Frost wrote: > > To put this in another light, had this issue been brought up post > > feature-freeze, your definition would mean that we would only have the > > option to either

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Robert Haas
On Wed, Mar 22, 2017 at 1:22 PM, Stephen Frost wrote: > To put this in another light, had this issue been brought up post > feature-freeze, your definition would mean that we would only have the > option to either revert the patch entirely or to live with the poor > naming

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Mar 22, 2017 at 12:51 PM, Stephen Frost wrote: > > This would clearly be an adjustment to the submitted patch, which > > happens regularly during the review and commit process and is part of > > the commitfest

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Robert Haas
On Wed, Mar 22, 2017 at 12:51 PM, Stephen Frost wrote: > This would clearly be an adjustment to the submitted patch, which > happens regularly during the review and commit process and is part of > the commitfest process, so I don't agree that holding it to new-feature > level

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Mar 22, 2017 at 12:22 PM, Stephen Frost wrote: > > While I understand that you'd like to separate the concerns between > > changing the renaming scheme and changing the default and enabling this > > option, I

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Robert Haas
On Wed, Mar 22, 2017 at 12:22 PM, Stephen Frost wrote: > While I understand that you'd like to separate the concerns between > changing the renaming scheme and changing the default and enabling this > option, I don't agree that they can or should be independently > considered.

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On the topic of whether to also change the default, I'm not sure what > is best and will defer to others. On the topic of whether to whack > around the file naming scheme, -1 from me. This patch was posted > three months ago and nobody

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Robert Haas
On Wed, Mar 22, 2017 at 8:46 AM, Stephen Frost wrote: >> I was definitely initially in favor of >> raising the value, but I got cold feet, a bit, when Alvaro pointed out >> that going to 64MB would require a substantial increase in >> min_wal_size. > > The performance concern

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Kuntal Ghosh
On Wed, Mar 22, 2017 at 3:14 PM, Beena Emerson wrote: > PFA an updated patch which fixes a minor bug I found. It only increases the > string size in pretty_wal_size function. > The 01-add-XLogSegmentOffset-macro.patch has also been rebased. Thanks for the updated

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Peter, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 3/22/17 08:46, Stephen Frost wrote: > > It's not my intent to 'torpedo' this patch but I'm pretty disappointed > > that we're introducing yet another initdb-time option with, as far as I > > can tell, no option to change it

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Peter Eisentraut
On 3/22/17 08:46, Stephen Frost wrote: > It's not my intent to 'torpedo' this patch but I'm pretty disappointed > that we're introducing yet another initdb-time option with, as far as I > can tell, no option to change it after the cluster has started (without > some serious hackery), and

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Bruce Momjian
On Tue, Mar 21, 2017 at 11:49:30PM -0400, Robert Haas wrote: > To be honest, I'd sort of forgotten about the change which is the > nominal subject of this thread - I was more focused on the patch, > which makes it configurable. I was definitely initially in favor of > raising the value, but I got

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Peter Eisentraut
On 3/22/17 05:44, Beena Emerson wrote: > As stated above, the default 16MB has not changed and so we can take > this separately and not as part of this patch. It's good to have that discussion separately, but if we're planning to do it for PG10 (not saying we should), then we should have that

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, Mar 21, 2017 at 8:10 PM, Stephen Frost wrote: > >> We've already > >> created quite a few incompatibilities in this release, and I'm not > >> entirely eager to just keep cranking them out at top speed. > > > >

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Beena Emerson
Hello, On Wed, Mar 22, 2017 at 9:19 AM, Robert Haas wrote: > > I'm a little worried that this whole question of changing the file > naming scheme is a diversion which will result in torpedoing any > chance of getting some kind of improvement here for v11. I don't >

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 11:49 PM, Robert Haas wrote: > I'm a little worried that this whole question of changing the file > naming scheme is a diversion which will result in torpedoing any > chance of getting some kind of improvement here for v11. I don't > think the patch

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 8:10 PM, Stephen Frost wrote: >> We've already >> created quite a few incompatibilities in this release, and I'm not >> entirely eager to just keep cranking them out at top speed. > > That position would seem to imply that you're in favor of keeping the

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, Mar 21, 2017 at 6:02 PM, David Steele wrote: > > The biggest downside I can see is that this would change the naming scheme > > for the default of 16MB compared to previous versions of Postgres. However, > > for

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 6:02 PM, David Steele wrote: > The biggest downside I can see is that this would change the naming scheme > for the default of 16MB compared to previous versions of Postgres. However, > for all other wal-seg-size values changes would need to be made

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Peter Eisentraut
On 3/21/17 15:22, Robert Haas wrote: > If you take the approach that Beena did, then you lose the > correspondence with LSNs, which is admittedly not great but there are > already helper functions available to deal with LSN -> filename > mappings and I assume those will continue to work. If you

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread David Steele
On 3/21/17 3:22 PM, Robert Haas wrote: On Tue, Mar 21, 2017 at 9:04 AM, Stephen Frost wrote: In short, I'm also concerned about this change to make WAL file names no longer match up with LSNs and also about the odd stepping that you get as a result of this change when it

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 9:04 AM, Stephen Frost wrote: > In short, I'm also concerned about this change to make WAL file names no > longer match up with LSNs and also about the odd stepping that you get > as a result of this change when it comes to WAL file names. OK, that's a

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread David Steele
On 3/21/17 9:04 AM, Stephen Frost wrote: Robert, * Robert Haas (robertmh...@gmail.com) wrote: On Mon, Mar 20, 2017 at 7:23 PM, David Steele wrote: With 16MB WAL segments the filename neatly aligns with the LSN. For example: WAL FILE 0001000100FE = LSN

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Mar 20, 2017 at 7:23 PM, David Steele wrote: > > With 16MB WAL segments the filename neatly aligns with the LSN. For > > example: > > > > WAL FILE 0001000100FE = LSN 1/FE00 > > > > This no longer

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Beena Emerson
PFA an updated patch. This fixes an issue reported by Tushar internally. Since the patch changes the way min and max wal_size is stored internally from segment count to size in kb, it limited the maximum size of min and max_wal_size to 2GB in 32 bit systems. The minimum required segment is 2 and

Re: [HACKERS] increasing the default WAL segment size

2017-03-20 Thread Robert Haas
On Mon, Mar 20, 2017 at 7:23 PM, David Steele wrote: > With 16MB WAL segments the filename neatly aligns with the LSN. For > example: > > WAL FILE 0001000100FE = LSN 1/FE00 > > This no longer holds true with this patch. It is already possible to change the

Re: [HACKERS] increasing the default WAL segment size

2017-03-20 Thread David Steele
Hi Beena, On 3/20/17 2:07 PM, Beena Emerson wrote: Added check for the version, the SHOW command will be run only in v10 and above. Previous versions do not need this. I've just had the chance to have a look at this patch. This is not a complete review, just a test of something I've been

Re: [HACKERS] increasing the default WAL segment size

2017-03-20 Thread Beena Emerson
Hello, PFA the updated patch. On Fri, Mar 17, 2017 at 6:40 AM, Robert Haas wrote: > On Tue, Mar 14, 2017 at 1:44 AM, Beena Emerson > wrote: > > Attached is the updated patch. It fixes the issues and also updates few > code > > comments. > > I

Re: [HACKERS] increasing the default WAL segment size

2017-03-19 Thread David Steele
On 3/17/17 4:56 PM, Tom Lane wrote: Peter Eisentraut writes: On 3/17/17 16:20, Peter Eisentraut wrote: I think we would have to extend restore_command with an additional placeholder that communicates the segment size, and add a new pg_standby option to

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Robert Haas
On Fri, Mar 17, 2017 at 6:11 PM, Peter Eisentraut wrote: > On 3/17/17 16:56, Tom Lane wrote: >> Tools could get the segment size out of >> XLogLongPageHeaderData.xlp_seg_size in the first page of the segment. > > OK, then pg_standby would have to wait until the

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Peter Eisentraut
On 3/17/17 16:56, Tom Lane wrote: > Tools could get the segment size out of > XLogLongPageHeaderData.xlp_seg_size in the first page of the segment. OK, then pg_standby would have to wait until the file is at least XLOG_BLCKSZ, then look inside and get the expected final size. A bit more

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Tom Lane
Peter Eisentraut writes: > On 3/17/17 16:20, Peter Eisentraut wrote: >> I think we would have to extend restore_command with an additional >> placeholder that communicates the segment size, and add a new pg_standby >> option to accept that size somehow. And

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Peter Eisentraut
On 3/17/17 16:20, Peter Eisentraut wrote: > On 3/16/17 21:10, Robert Haas wrote: >> The changes to pg_standby seem to completely break the logic to wait >> until the file has attained the correct size. I don't know how to >> salvage that logic off-hand, but just breaking it isn't acceptable. > >

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Peter Eisentraut
On 3/16/17 21:10, Robert Haas wrote: > The changes to pg_standby seem to completely break the logic to wait > until the file has attained the correct size. I don't know how to > salvage that logic off-hand, but just breaking it isn't acceptable. I think we would have to extend restore_command

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Robert Haas
On Fri, Mar 17, 2017 at 2:08 AM, Beena Emerson wrote: > The option was intended to only accept values in MB as the original config > --with-wal-segsize option, unfortunately, the patch does not throw error as > in the config option when the units are specified. Yeah,

Re: [HACKERS] increasing the default WAL segment size

2017-03-17 Thread Beena Emerson
Hello, Thank you for your comments, I will post an updated patch soon. On Fri, Mar 17, 2017 at 6:40 AM, Robert Haas wrote: > > > +assign_wal_segment_size(int newval, void *extra) > > Why does a PGC_INTERNAL GUC need an assign hook? I think the GUC > should only be there

Re: [HACKERS] increasing the default WAL segment size

2017-03-16 Thread Robert Haas
On Tue, Mar 14, 2017 at 1:44 AM, Beena Emerson wrote: > Attached is the updated patch. It fixes the issues and also updates few code > comments. I did an initial readthrough of this patch tonight just to get a feeling for what's going on. Based on that, here are a few

  1   2   3   >