Re: [HACKERS] pg_restore --multi-thread

2009-03-19 Thread Peter Eisentraut
On Thursday 12 February 2009 17:41:01 Peter Eisentraut wrote: I know we've already had a discussion on the naming of the pg_restore -m option, but in any case this description in pg_restore --help is confusing: -m, --multi-thread=NUM use this many parallel connections to restore Either it

Re: [HACKERS] pg_restore --multi-thread

2009-03-19 Thread Andrew Dunstan
Peter Eisentraut wrote: While I think jobs isn't a totally accurate description, I would still propose to use -j/--jobs for the option name, because it is neutral about the implementation and has a strong precedent as being used to increase the parallelization to get the work done faster.

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Peter Eisentraut
Andrew Dunstan wrote: Cédric Villemain wrote: -j [jobs], --jobs[=jobs] Specifies the number of jobs (pg_restore) to run simultaneously. If the -j option is given without an argument, pg_restore will not limit the number of jobs that can run simultaneously. Quite apart from anything

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Andrew Dunstan
Peter Eisentraut wrote: Andrew Dunstan wrote: Cédric Villemain wrote: -j [jobs], --jobs[=jobs] Specifies the number of jobs (pg_restore) to run simultaneously. If the -j option is given without an argument, pg_restore will not limit the number of jobs that can run simultaneously.

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Joshua D. Drake
On Fri, 2009-02-20 at 09:33 -0500, Andrew Dunstan wrote: The short answer is that we don't know yet. There is anecdotal evidence that the number of CPUs on the server is a good place to start, but we should be honest enough to say that this is a new feature and we are still gathering

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Andrew Dunstan
Joshua D. Drake wrote: On Fri, 2009-02-20 at 09:33 -0500, Andrew Dunstan wrote: The short answer is that we don't know yet. There is anecdotal evidence that the number of CPUs on the server is a good place to start, but we should be honest enough to say that this is a new feature and we

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Kenneth Marshall
On Fri, Feb 20, 2009 at 09:22:58AM -0800, Joshua D. Drake wrote: On Fri, 2009-02-20 at 09:33 -0500, Andrew Dunstan wrote: The short answer is that we don't know yet. There is anecdotal evidence that the number of CPUs on the server is a good place to start, but we should be honest

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Kevin Grittner
Andrew Dunstan and...@dunslane.net wrote: Joshua D. Drake wrote: the fastest restore time for 220G was performed with 24 threads with an 8 core box. It is important to point out that this was a machine with 50 spindles. Which is where your bottleneck is going to be immediately after

Re: [HACKERS] pg_restore --multi-thread

2009-02-20 Thread Joshua D. Drake
On Fri, 2009-02-20 at 11:57 -0600, Kevin Grittner wrote: But you are right that there isn't a simple formula. Perhaps the greater of the number of CPUs or effective spindles? (24 sounds suspiciously close to effective spindles on a 50 spindle box with RAID 10.) It does except that

Re: [HACKERS] pg_restore --multi-thread

2009-02-16 Thread Cédric Villemain
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Cédric Villemain a écrit : Joshua D. Drake a écrit : On Thu, 2009-02-12 at 11:47 -0500, Andrew Dunstan wrote: Joshua D. Drake wrote: On Thu, 2009-02-12 at 11:32 -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: The

Re: [HACKERS] pg_restore --multi-thread

2009-02-16 Thread Laurent Laborde
On Mon, Feb 16, 2009 at 12:10 PM, Cédric Villemain cedric.villem...@dalibo.com wrote: is -j already affected ? else (like make): -j [jobs], --jobs[=jobs] Specifies the number of jobs (pg_restore) to run simultaneously. If the -j option is given without an argument, pg_restore will not

Re: [HACKERS] pg_restore --multi-thread

2009-02-16 Thread Andrew Dunstan
Cédric Villemain wrote: -j [jobs], --jobs[=jobs] Specifies the number of jobs (pg_restore) to run simultaneously. If the -j option is given without an argument, pg_restore will not limit the number of jobs that can run simultaneously. Quite apart from anything else, this

Re: [HACKERS] pg_restore --multi-thread

2009-02-14 Thread Robert Treat
On Thursday 12 February 2009 11:50:26 Joshua D. Drake wrote: On Thu, 2009-02-12 at 11:47 -0500, Andrew Dunstan wrote: Joshua D. Drake wrote: On Thu, 2009-02-12 at 11:32 -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: The implementation is actually different across

Re: [HACKERS] pg_restore --multi-thread

2009-02-13 Thread Peter Eisentraut
Andrew Dunstan wrote: I also don't really understand what is confusing about the description. Where does the benefit of using it come from? When would one want to use it? Is it because the parallelization happens on the client or on the server? Does it happen because to CPU

[HACKERS] pg_restore --multi-thread

2009-02-12 Thread Peter Eisentraut
I know we've already had a discussion on the naming of the pg_restore -m option, but in any case this description in pg_restore --help is confusing: -m, --multi-thread=NUM use this many parallel connections to restore Either it is using that many threads in the client, or it is using that

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Andrew Dunstan
Peter Eisentraut wrote: I know we've already had a discussion on the naming of the pg_restore -m option, but in any case this description in pg_restore --help is confusing: -m, --multi-thread=NUM use this many parallel connections to restore Either it is using that many threads in the

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: The implementation is actually different across platforms: on Windows the workers are genuine threads, while elsewhere they are forked children in the same fashion as the backend (non-EXEC_BACKEND case). In either case, the program will use up to

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Joshua D. Drake
On Thu, 2009-02-12 at 11:32 -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: The implementation is actually different across platforms: on Windows the workers are genuine threads, while elsewhere they are forked children in the same fashion as the backend

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Andrew Dunstan
Joshua D. Drake wrote: On Thu, 2009-02-12 at 11:32 -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: The implementation is actually different across platforms: on Windows the workers are genuine threads, while elsewhere they are forked children in the same fashion as

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Joshua D. Drake
On Thu, 2009-02-12 at 11:47 -0500, Andrew Dunstan wrote: Joshua D. Drake wrote: On Thu, 2009-02-12 at 11:32 -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: The implementation is actually different across platforms: on Windows the workers are genuine

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Cédric Villemain
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joshua D. Drake a écrit : On Thu, 2009-02-12 at 11:47 -0500, Andrew Dunstan wrote: Joshua D. Drake wrote: On Thu, 2009-02-12 at 11:32 -0500, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: The implementation is actually

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Jonah H. Harris
On Thu, Feb 12, 2009 at 11:37 AM, Joshua D. Drake j...@commandprompt.comwrote: --num-workers or --num-connections would both work. --num-parallel? -- Jonah H. Harris, Senior DBA myYearbook.com

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread Michael Glaesemann
On 2009-02-12, at 14:15 , Jonah H. Harris wrote: On Thu, Feb 12, 2009 at 11:37 AM, Joshua D. Drake j...@commandprompt.com wrote: --num-workers or --num-connections would both work. --num-parallel? --num-concurrent? Michael Glaesemann michael.glaesem...@myyearbook.com -- Sent via

Re: [HACKERS] pg_restore --multi-thread

2009-02-12 Thread David Fetter
On Thu, Feb 12, 2009 at 02:16:39PM -0500, Michael Glaesemann wrote: On 2009-02-12, at 14:15 , Jonah H. Harris wrote: On Thu, Feb 12, 2009 at 11:37 AM, Joshua D. Drake j...@commandprompt.com wrote: --num-workers or --num-connections would both work. --num-parallel? --num-concurrent?