Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Stephen Frost
* Joshua D. Drake ([EMAIL PROTECTED]) wrote: > Custom format rocks for partial set restores from a whole dump. See the > TOC option :) I imagine it does, but that's very rarely what I need. Most of the time we're dumping out a schema to load it into a seperate schema (usually on another host).

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Joshua D. Drake
Stephen Frost wrote: * Tom Lane ([EMAIL PROTECTED]) wrote: Stephen Frost <[EMAIL PROTECTED]> writes: I dislike, and doubt that I'd use, this approach. At the end of the day, it ends up processing the same (very large amount of data) multiple times. This would depend on the dump being in the

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Stephen Frost
* Tom Lane ([EMAIL PROTECTED]) wrote: > Right, but the parallelization is going to happen sometime, and it is > going to happen in the context of pg_restore. So I think it's pretty > silly to argue that no one will ever want this feature to work in > pg_restore. I think you've about convinced me

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Stephen Frost
* Tom Lane ([EMAIL PROTECTED]) wrote: > Stephen Frost <[EMAIL PROTECTED]> writes: > > I dislike, and doubt that I'd use, this approach. At the end of the > > day, it ends up processing the same (very large amount of data) multiple > > times. > > Well, that's easily avoided: just replace the third

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Joshua D. Drake
Andrew Dunstan wrote: Joshua D. Drake wrote: Agreed but that is a problem I understand with a solution I don't. I am all eyes on a way to fix that. One thought I had and please, be gentle in response was some sort of async transaction capability. I know that libpq has the ability to send a

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Andrew Dunstan
Joshua D. Drake wrote: Agreed but that is a problem I understand with a solution I don't. I am all eyes on a way to fix that. One thought I had and please, be gentle in response was some sort of async transaction capability. I know that libpq has the ability to send async queries. Is it pos

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Simon Riggs
On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote: > 2. We have no concurrency which means, anyone with any database over 50G > has unacceptable restore times. Agreed. Also the core reason for wanting -w > 3. We have to continue develop hacks to define custom utilization. Why > am I pas

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Simon Riggs
On Sat, 2008-07-26 at 13:56 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > I want to dump tables separately for performance reasons. There are > > documented tests showing 100% gains using this method. There is no gain > > adding this to pg_restore. There is a gain to be had

Re: [PATCHES] pg_dump additional options for performance

2008-07-27 Thread Joshua D. Drake
Simon Riggs wrote: On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote: 2. We have no concurrency which means, anyone with any database over 50G has unacceptable restore times. Agreed. Sounds good. Doesn't help with the main element of dump time: one table at a time to one output fil