Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-10 Thread Robert Haas
On Tue, Feb 8, 2011 at 10:54 PM, Joachim Wieland j...@mcknight.de wrote: On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Tue, Feb 8, 2011 at 13:34, Robert Haas robertmh...@gmail.com wrote: So how close are we to having a committable version of this?  

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Itagaki Takahiro
On Tue, Feb 8, 2011 at 13:34, Robert Haas robertmh...@gmail.com wrote: So how close are we to having a committable version of this?  Should we push this out to 9.2? I think so. The feature is pretty attractive, but more works are required: * Re-base on synchronized snapshots patch * Consider

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Joachim Wieland
On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Tue, Feb 8, 2011 at 13:34, Robert Haas robertmh...@gmail.com wrote: So how close are we to having a committable version of this?  Should we push this out to 9.2? I think so. The feature is pretty attractive,

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Joachim Wieland
Hi Jaime, thanks for your review! On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova ja...@2ndquadrant.com wrote: code review: something i found, and is a very simple one, is this warning (there's a similar issue in _StartMasterParallel with the buf variable) pg_backup_directory.c: In

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Robert Haas
On Mon, Feb 7, 2011 at 10:42 PM, Joachim Wieland j...@mcknight.de wrote: i guess the huge amount of info is showing the patch is just for debugging and will be removed before commit, right? That's right. So how close are we to having a committable version of this? Should we push this out to

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-06 Thread Jaime Casanova
On Tue, Feb 1, 2011 at 11:32 PM, Joachim Wieland j...@mcknight.de wrote: On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas robertmh...@gmail.com wrote: The parallel pg_dump portion of this patch (i.e. the still-uncommitted part) no longer applies.  Please rebase. Here is a rebased version with

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-06 Thread Jaime Casanova
On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Tue, Feb 1, 2011 at 11:32 PM, Joachim Wieland j...@mcknight.de wrote: On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas robertmh...@gmail.com wrote: The parallel pg_dump portion of this patch (i.e. the still-uncommitted

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Joachim Wieland
On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: I think we have 2 important technical issues here:  * The consistency is not perfect. Each transaction is started   with small delays in step 1, but we cannot guarantee no other   transaction between them.

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Magnus Hagander
On Sat, Feb 5, 2011 at 04:50, Joachim Wieland j...@mcknight.de wrote: On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: It might be better to remove Windows-specific codes from the first try. I doubt Windows message queue is the best API in such console-based

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-03 Thread Itagaki Takahiro
On Wed, Feb 2, 2011 at 13:32, Joachim Wieland j...@mcknight.de wrote: Here is a rebased version with some minor changes as well. I read the patch works as below. Am I understanding correctly? 1. Open all connections in a parent process. 2. Start transactions for each connection in the

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-01 Thread Joachim Wieland
On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas robertmh...@gmail.com wrote: The parallel pg_dump portion of this patch (i.e. the still-uncommitted part) no longer applies.  Please rebase. Here is a rebased version with some minor changes as well. I haven't tested it on Windows now but will do so

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-30 Thread Robert Haas
On Wed, Jan 19, 2011 at 12:45 AM, Joachim Wieland j...@mcknight.de wrote: On Mon, Jan 17, 2011 at 5:38 PM, Jaime Casanova ja...@2ndquadrant.com wrote: This one is the last version of this patch? if so, commitfest app should be updated to reflect that Here are the latest patches all of them

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-23 Thread Heikki Linnakangas
On 21.01.2011 19:11, Euler Taveira de Oliveira wrote: Em 21-01-2011 12:47, Andrew Dunstan escreveu: Maybe we could change the hint to say --file=DESTINATION or --file=FILENAME|DIRNAME ? ... --file=OUTPUT or --file=OUTPUTNAME. Ok, works for me. I've committed this patch now, with a whole

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Heikki Linnakangas
On 20.01.2011 17:22, Heikki Linnakangas wrote: (I'm working on this, no need to submit a new patch) Ok, here's a heavily refactored version of this (also available at git://git.postgresql.org/git/users/heikki/postgres.git, branch pg_dump_directory). The directory format is now identical to

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Robert Haas
On Fri, Jan 21, 2011 at 4:41 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: There's one UI thing that bothers me. The option to specify the target directory is called --file. But it's clearly not a file. OTOH, I'd hate to introduce a parallel --dir option just for this. Any

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Heikki Linnakangas
On 21.01.2011 15:35, Robert Haas wrote: On Fri, Jan 21, 2011 at 4:41 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: There's one UI thing that bothers me. The option to specify the target directory is called --file. But it's clearly not a file. OTOH, I'd hate to introduce a

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Andrew Dunstan
On 01/21/2011 10:34 AM, Heikki Linnakangas wrote: On 21.01.2011 15:35, Robert Haas wrote: On Fri, Jan 21, 2011 at 4:41 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: There's one UI thing that bothers me. The option to specify the target directory is called --file. But

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-21 Thread Euler Taveira de Oliveira
Em 21-01-2011 12:47, Andrew Dunstan escreveu: Maybe we could change the hint to say --file=DESTINATION or --file=FILENAME|DIRNAME ? ... --file=OUTPUT or --file=OUTPUTNAME. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Heikki Linnakangas
On 19.01.2011 16:01, Joachim Wieland wrote: On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here are the latest patches all of them also rebased to current HEAD. Will update the commitfest app as well. What's the idea of storing the file sizes

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Joachim Wieland
On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's part of the overall idea to make sure files are not inadvertently exchanged between different backups and that a file is not truncated. In the future I'd also like to add a checksum to the TOC so

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Heikki Linnakangas
On 20.01.2011 15:46, Joachim Wieland wrote: On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The header is there to identify a file, it contains the header that every other pgdump file contains, including the internal version number and the unique

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Florian Pflug
On Jan20, 2011, at 16:22 , Heikki Linnakangas wrote: You can put files in the archive in a certain order if you list them explicitly in the tar command line, like tar cf backup.tar toc.dat It's hard to know the right order, though. In practice you would need to do tar tf backup.tar

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Heikki Linnakangas
On 19.01.2011 07:45, Joachim Wieland wrote: On Mon, Jan 17, 2011 at 5:38 PM, Jaime Casanovaja...@2ndquadrant.com wrote: This one is the last version of this patch? if so, commitfest app should be updated to reflect that Here are the latest patches all of them also rebased to current HEAD.

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Joachim Wieland
On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here are the latest patches all of them also rebased to current HEAD. Will update the commitfest app as well. What's the idea of storing the file sizes in the toc file? It looks like it's not used

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-17 Thread Jaime Casanova
On Fri, Jan 7, 2011 at 3:18 PM, Joachim Wieland j...@mcknight.de wrote: Here's a new series of patches for the parallel dump/restore. They need to be applied on top of each other. This one is the last version of this patch? if so, commitfest app should be updated to reflect that -- Jaime