Re: pg_upgrade: Pass -j down to vacuumdb

2019-04-04 Thread Tom Lane
Peter Eisentraut writes: > I think the real fix is to have pg_upgrade copy over the statistics. > They are right there after all, just take them. Well, it's not "just take them", we need to develop some infrastructure that will permit coping with cross-version differences in the stats storage.

Re: pg_upgrade: Pass -j down to vacuumdb

2019-04-04 Thread Peter Eisentraut
On 2019-04-03 23:24, Justin Pryzby wrote: > I just did a test on one of our large-but-not-huge customers. With > stats_target=1, analyzing a 145GB partitioned table looks like it'll take > perhaps an hour; they have ~1TB data, so delaying services during ANALYZE > would > nullify the utility of

Re: pg_upgrade: Pass -j down to vacuumdb

2019-04-03 Thread Justin Pryzby
On Wed, Apr 03, 2019 at 04:42:14PM -0400, Jeff Janes wrote: > So maybe the first stage could > be run by pg_upgrade itself, while the new server is still running on a > linux socket in a private directory. I think that would take too long. It would be less of an issue if there was

Re: pg_upgrade: Pass -j down to vacuumdb

2019-04-03 Thread Jeff Janes
On Fri, Mar 29, 2019 at 5:58 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2019-03-28 02:43, Jeff Janes wrote: > > At first blush I thought it was obvious that you would not want to run > > analyze-in-stages in parallel. But after thinking about it some more > > and

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-29 Thread Peter Eisentraut
On 2019-03-28 02:43, Jeff Janes wrote: > At first blush I thought it was obvious that you would not want to run > analyze-in-stages in parallel.  But after thinking about it some more > and reflecting on experience doing some troublesome upgrades, I would > reverse that and say it is now obvious

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-27 Thread Jeff Janes
On Tue, Mar 26, 2019 at 7:28 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2019-03-25 22:57, Tom Lane wrote: > > + fprintf(script, "echo %sYou may wish to add --jobs=N for parallel > analyzing.%s\n", > > + ECHO_QUOTE, ECHO_QUOTE); > > But then you get

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-26 Thread Tom Lane
Peter Eisentraut writes: > On 2019-03-25 22:57, Tom Lane wrote: >> +fprintf(script, "echo %sYou may wish to add --jobs=N for parallel >> analyzing.%s\n", >> +ECHO_QUOTE, ECHO_QUOTE); > But then you get that information after you have already started the script. Yes, but

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-26 Thread Peter Eisentraut
On 2019-03-25 22:57, Tom Lane wrote: > + fprintf(script, "echo %sYou may wish to add --jobs=N for parallel > analyzing.%s\n", > + ECHO_QUOTE, ECHO_QUOTE); But then you get that information after you have already started the script. I don't find any information about this

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-26 Thread Daniel Gustafsson
On Tuesday, March 26, 2019 3:20 AM, Michael Paquier wrote: > On Mon, Mar 25, 2019 at 05:57:50PM -0400, Tom Lane wrote: > > > In short, I'm not convinced that most of this patch is an improvement > > on the status quo. I think we'd be best off to just take the idea > > of explicitly mentioning

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-25 Thread Michael Paquier
On Mon, Mar 25, 2019 at 05:57:50PM -0400, Tom Lane wrote: > In short, I'm not convinced that most of this patch is an improvement > on the status quo. I think we'd be best off to just take the idea > of explicitly mentioning adding --jobs to a manual run, ie roughly Yeah, no objections from here

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-25 Thread Tom Lane
Jesper Pedersen writes: > [ v9_0001-Highlight-that-the-jobs-option-isn-t-passed-down-to-.patch ] Now that I've looked at this, it seems like it's fairly confused about what the proposed VACUUMDB_OPTS variable would actually do for you. If you're going to run vacuumdb directly, it hardly seems

RE: pg_upgrade: Pass -j down to vacuumdb

2019-03-12 Thread Jamison, Kirk
Hi Jesper, > Thanks Kirk ! > > > On 3/12/19 2:20 PM, Robert Haas wrote: > > The words 'by default' should be removed here, because there is also > > no non-default way to get that behavior, either. > > > > Here is v9 based on Kirk's and your input. Thanks! Although there were trailing

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-12 Thread Jesper Pedersen
Hi, Thanks Kirk ! On 3/12/19 2:20 PM, Robert Haas wrote: The words 'by default' should be removed here, because there is also no non-default way to get that behavior, either. Here is v9 based on Kirk's and your input. Best regards, Jesper >From 5b879f79300412638705e32aa3ed51aff3cbe75c

Re: pg_upgrade: Pass -j down to vacuumdb

2019-03-12 Thread Robert Haas
On Mon, Mar 11, 2019 at 10:05 PM Jamison, Kirk wrote: > I was thinking of something like the attached, + machine. Note that this option isn't passed to the + vacuumdb application by default. The words 'by default' should be removed here, because there is also no non-default way to get

RE: pg_upgrade: Pass -j down to vacuumdb

2019-03-11 Thread Jamison, Kirk
Hi Jesper, Sorry I almost completely forgot to get back to you on this. Actually your patch works when I tested it before, and I understand the intention. . Although a point was raised by other developers by making --jobs optional in the suggested line by using the env variable instead. > >

RE: pg_upgrade: Pass -j down to vacuumdb

2019-02-03 Thread Jamison, Kirk
Hi, On February 1, 2019 8:14 PM +, Jesper Pedersen wrote: > On 2/1/19 4:58 AM, Alvaro Herrera wrote: >> On 2019-Feb-01, Jamison, Kirk wrote: >>> I'm not sure if misunderstood the purpose of $VACUUMDB_OPTS. I >>> thought what the other developers suggested is to utilize it so that >>>

Re: pg_upgrade: Pass -j down to vacuumdb

2019-02-03 Thread Andres Freund
Hi, On 2019-02-01 14:14:27 -0500, Jesper Pedersen wrote: > From 4b5856725af5d971a26a2ba150c1067ee21bb242 Mon Sep 17 00:00:00 2001 > From: jesperpedersen > Date: Fri, 21 Dec 2018 05:05:31 -0500 > Subject: [PATCH] Highlight that the --jobs option isn't passed down to > vacuumdb by default. This

Re: pg_upgrade: Pass -j down to vacuumdb

2019-02-01 Thread Jesper Pedersen
Hi, On 2/1/19 4:58 AM, Alvaro Herrera wrote: On 2019-Feb-01, Jamison, Kirk wrote: I'm not sure if misunderstood the purpose of $VACUUMDB_OPTS. I thought what the other developers suggested is to utilize it so that --jobs for vacuumdb would be optional and just based from user-specified option

Re: pg_upgrade: Pass -j down to vacuumdb

2019-02-01 Thread Alvaro Herrera
On 2019-Feb-01, Jamison, Kirk wrote: > I'm not sure if misunderstood the purpose of $VACUUMDB_OPTS. I thought what > the other developers suggested is to utilize it so that --jobs for vacuumdb > would be optional and just based from user-specified option $VACUUMDB_OPTS. > By which it would not

RE: pg_upgrade: Pass -j down to vacuumdb

2019-02-01 Thread Jamison, Kirk
On January 31, 2019, 9:29PM +, Jesper Pedersen wrote: >>> I added most of the documentation back, as requested by Kirk. >> >> Actually, I find it useful to be documented. However, major contributors >> have higher opinions in terms of experience, so I think it's alright with me >> not to

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-31 Thread Jesper Pedersen
Hi, On 1/30/19 7:59 PM, Jamison, Kirk wrote: I added most of the documentation back, as requested by Kirk. Actually, I find it useful to be documented. However, major contributors have higher opinions in terms of experience, so I think it's alright with me not to include the doc part if

RE: pg_upgrade: Pass -j down to vacuumdb

2019-01-30 Thread Jamison, Kirk
On January 29, 2019 8:19 PM +, Jesper Pedersen wrote: >On 1/29/19 12:08 AM, Michael Paquier wrote: >> I would document the optional VACUUM_OPTS on the page of pg_upgrade. >> If Peter thinks it is fine to not do so, that's fine for me as well. >> .. >I added most of the documentation back, as

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-29 Thread Jesper Pedersen
Hi, On 1/29/19 12:08 AM, Michael Paquier wrote: On Tue, Jan 29, 2019 at 12:35:30AM +, Jamison, Kirk wrote: I just checked the patch. As per advice, you removed the versioning and specified --jobs. The patch still applies, builds and passed the tests successfully. I would document the

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Michael Paquier
On Tue, Jan 29, 2019 at 12:35:30AM +, Jamison, Kirk wrote: > I just checked the patch. > As per advice, you removed the versioning and specified --jobs. > The patch still applies, builds and passed the tests successfully. I would document the optional VACUUM_OPTS on the page of pg_upgrade.

RE: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Jamison, Kirk
Hi Jesper, Thanks for updating your patches quickly. >On 1/28/19 3:50 PM, Peter Eisentraut wrote: >>> Done, and v4 attached. >> >> I would drop the changes in pgupgrade.sgml. We don't need to explain >> what doesn't happen, when nobody before now ever thought that it would >> happen. >> >>

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Jesper Pedersen
On 1/28/19 3:50 PM, Peter Eisentraut wrote: Done, and v4 attached. I would drop the changes in pgupgrade.sgml. We don't need to explain what doesn't happen, when nobody before now ever thought that it would happen. Also, we don't need the versioning stuff. The new cluster in pg_upgrade is

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Peter Eisentraut
On 28/01/2019 17:47, Jesper Pedersen wrote: > On 1/28/19 11:30 AM, Fabrízio de Royes Mello wrote: >> IMHO you should use long option name '--jobs' like others. >> > > Thanks for your feedback ! > > Done, and v4 attached. I would drop the changes in pgupgrade.sgml. We don't need to explain what

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Jesper Pedersen
On 1/28/19 11:30 AM, Fabrízio de Royes Mello wrote: IMHO you should use long option name '--jobs' like others. Thanks for your feedback ! Done, and v4 attached. Best regards, Jesper >From 142b4723f433f39d0eed2ced4beccc3721451103 Mon Sep 17 00:00:00 2001 From: jesperpedersen Date: Fri, 21

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Fabrízio de Royes Mello
On Mon, Jan 28, 2019 at 1:15 PM Jesper Pedersen wrote: > > Done. > > Attached is v3. > IMHO you should use long option name '--jobs' like others. Regards, -- Fabrízio de Royes Mello Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-28 Thread Jesper Pedersen
Hi, On 1/27/19 7:50 PM, Jamison, Kirk wrote: There were also helpful comments from the developers above, pointing it to the right direction. In addition to Peter's comment, quoting "...if you want to do this as fast as possible, don't use this script. That comment could be enhanced to suggest

RE: pg_upgrade: Pass -j down to vacuumdb

2019-01-27 Thread Jamison, Kirk
Hi Jesper, On Friday, January 25, 2019, Jesper Pedersen > Thanks for your feedback ! > > As per Peter's comments I have changed the patch (v2) to not pass down the -j > option to vacuumdb. > > Only an update to the documentation and console output is made in order to > make it more clear.

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-27 Thread Michael Paquier
On Fri, Jan 25, 2019 at 12:16:49PM -0500, Tom Lane wrote: > Alvaro Herrera writes: >> So let's have it write with a $VACUUMDB_OPTS variable, which is by >> default defined as empty but with a comment suggesting that maybe the >> user wants to add the -j option. This way, if they have to edit it,

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-25 Thread Tom Lane
Alvaro Herrera writes: > So let's have it write with a $VACUUMDB_OPTS variable, which is by > default defined as empty but with a comment suggesting that maybe the > user wants to add the -j option. This way, if they have to edit it, > they only have to edit the VACUUMDB_OPTS line instead of

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-25 Thread Alvaro Herrera
On 2019-Jan-25, Peter Eisentraut wrote: > On 25/01/2019 11:28, Michael Paquier wrote: > > based on that linking the value used by pg_upgrade and vacuumdb is a > > bad concept in my opinion, and the patch should be rejected. More > > documentation on pg_upgrade side to explain that a bit better

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-25 Thread Jesper Pedersen
Hi Kirk, On 1/24/19 9:31 PM, Jamison, Kirk wrote: According to CF app, this patch needs review so I took a look at it. Currently, your patch applies and builds cleanly, and all tests are also successful based from the CF bot patch tester. I'm not sure if I have understood the argument raised

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-25 Thread Peter Eisentraut
On 25/01/2019 11:28, Michael Paquier wrote: > based on that linking the value used by pg_upgrade and vacuumdb is a > bad concept in my opinion, and the patch should be rejected. More > documentation on pg_upgrade side to explain that a bit better could be > a good idea though, as it is perfectly

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-25 Thread Michael Paquier
On Fri, Jan 25, 2019 at 02:31:57AM +, Jamison, Kirk wrote: > I'm not sure if I have understood the argument raised by Peter > correctly. Quoting Peter, "it's not clear that pg_upgrade and > vacuumdb are bound the same way, so it's not a given that the same > -j number should be used." I

RE: pg_upgrade: Pass -j down to vacuumdb

2019-01-24 Thread Jamison, Kirk
Hi, According to CF app, this patch needs review so I took a look at it. Currently, your patch applies and builds cleanly, and all tests are also successful based from the CF bot patch tester. I'm not sure if I have understood the argument raised by Peter correctly. Quoting Peter, "it's not

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-07 Thread Jesper Pedersen
Hi Peter, On 1/3/19 7:03 AM, Peter Eisentraut wrote: Perhaps more documentation would be useful here. Here is v2 that just notes that the -j option isn't passed down. Best regards, Jesper >From fe0be6c1f5cbcaeb2981ff4dcfceae2e2cb286b7 Mon Sep 17 00:00:00 2001 From: jesperpedersen Date:

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-03 Thread Jerry Sievers
Peter Eisentraut writes: > On 02/01/2019 20:47, Jesper Pedersen wrote: > >> Well, that really depends. The user passed -j to pg_upgrade in order for >> the upgrade to happen faster, so maybe they would expect, as I would, >> that the ANALYZE phase would happen in parallel too. > > pg_upgrade

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-03 Thread Peter Eisentraut
On 02/01/2019 20:47, Jesper Pedersen wrote: > Well, that really depends. The user passed -j to pg_upgrade in order for > the upgrade to happen faster, so maybe they would expect, as I would, > that the ANALYZE phase would happen in parallel too. pg_upgrade -j reduces the *downtime* caused by

Re: pg_upgrade: Pass -j down to vacuumdb

2019-01-02 Thread Jesper Pedersen
Hi, On 12/29/18 10:03 AM, Peter Eisentraut wrote: On 21/12/2018 11:12, Jesper Pedersen wrote: Here is a patch that passes the -j option from pg_upgrade down to vacuumdb if supported. It's not clear to me that that is what is usually wanted. The point of the post-upgrade analyze script is

Re: pg_upgrade: Pass -j down to vacuumdb

2018-12-29 Thread Peter Eisentraut
On 21/12/2018 11:12, Jesper Pedersen wrote: > Here is a patch that passes the -j option from pg_upgrade down to > vacuumdb if supported. It's not clear to me that that is what is usually wanted. The point of the post-upgrade analyze script is specifically to do the analyze in a gentle fashion.