Re: [HACKERS] pg_dump --split patch

Marko Tiikkaja wrote: > On 16/11/2012 15:52, Dimitri Fontaine wrote: > > > >What happens if you have a table foo and another table "FoO"? > > They would go to the same file. If you think there are technical > issues behind that decision (e.g. the dump would not restore), I > would like to hear a

Re: [HACKERS] pg_dump --split patch

On 11/19/2012 09:07 AM, Dimitri Fontaine wrote: Dimitri Fontaine writes: pg_dump | pg_restore pg_export | psql While I agree that this idea - when implemented - would be nicer in practically every way, I'm not sure I want to volunteer to do all the necessary work. What I think need

Re: [HACKERS] pg_dump --split patch

Dimitri Fontaine writes: >>> pg_dump | pg_restore >>> pg_export | psql >> >> While I agree that this idea - when implemented - would be nicer in >> practically every way, I'm not sure I want to volunteer to do all the >> necessary work. > > What I think needs to happen now is a commiter's

Re: [HACKERS] pg_dump --split patch

Marko Tiikkaja writes: >> What happens if you have a table foo and another table "FoO"? > > They would go to the same file. If you think there are technical issues > behind that decision (e.g. the dump would not restore), I would like to hear > an example case. I didn't try the patch itself yet

Re: [HACKERS] pg_dump --split patch

Hi, On 16/11/2012 15:52, Dimitri Fontaine wrote: Marko Tiikkaja writes: The general output scheme looks like this: schemaname/OBJECT_TYPES/object_name.sql, I like this feature, I actually did have to code it myself in the past and several other people did so, so we already have at least

Re: [HACKERS] pg_dump --split patch

Hi, Marko Tiikkaja writes: > The general output scheme looks like this: >schemaname/OBJECT_TYPES/object_name.sql, I like this feature, I actually did have to code it myself in the past and several other people did so, so we already have at least 3 copies of `getddl` variants around. I really

Re: [HACKERS] pg_dump --split patch

Hi, Now that the (at least as far as I know) last ordering problem in pg_dump has been solved [1], I'm going to attempt resurrecting this old thread. It seemed to me that the biggest objections to this patch in the old discussions were directed at the implementation, which I have tried to i

Re: [HACKERS] pg_dump --split patch

On Mon, Jan 3, 2011 at 2:18 PM, Tom Lane wrote: > Robert Haas writes: >> On Mon, Jan 3, 2011 at 1:34 PM, Tom Lane wrote: >>> Yeah, that's exactly it.  I can think of some possible uses for >>> splitting up pg_dump output, but frankly "to ease diff-ing" is not >>> one of them.  For that problem,

Re: [HACKERS] pg_dump --split patch

Joel Jacobson wrote: To understand a change to my database functions, I would start by looking at the top-level, only focusing on the names of the functions modified/added/removed. At this stage, you want as little information as possible about each change, such as only the names of the functio

Re: [HACKERS] pg_dump --split patch

On 28.12.2010 17:00, Joel Jacobson wrote: Dear fellow hackers, Problem: A normal diff of two slightly different schema dump files (pg_dump -s), will not produce a user-friendly diff, as you get all changes in the same file. Another Solution: I have used a python script for spliiting "dump -s

Re: [HACKERS] pg_dump --split patch

On 28.12.2010 23:51, Tom Lane wrote: Andrew Dunstan writes: On 12/28/2010 04:44 PM, Joel Jacobson wrote: Perhaps abbreviations are to prefer, e.g., myfunc_i, myfunc_i_c, etc to reduce the need of truncating filenames. I think that's just horrible. Does the i stand for integer or inet? And it

Re: [HACKERS] pg_dump --split patch

On 28.12.2010 22:44, Joel Jacobson wrote: Sent from my iPhone On 28 dec 2010, at 21:45, Gurjeet Singh > wrote: The problem I see with suffixing a sequence id to the objects with name collision is that one day the dump may name myfunc(int) as myfunc.sql and a

Re: [HACKERS] pg_dump --split patch

On Mon, Jan 3, 2011 at 3:15 PM, Dimitri Fontaine wrote: >> On the other hand, I can certainly think of times when even a pretty >> dumb implementation of this would have saved me some time. > > You mean like those: > >  https://labs.omniti.com/labs/pgtreats/wiki/getddl >  https://github.com/dimitr

Re: [HACKERS] pg_dump --split patch

Robert Haas writes: > I have to admit I'm a bit unsold on the approach as well. It seems > like you could write a short Perl script which would transform a text > format dump into the proposed format pretty easily, and if you did > that and published the script, then the next poor shmuck who had

Re: [HACKERS] pg_dump --split patch

On Mon, Jan 3, 2011 at 2:46 PM, Joel Jacobson wrote: > My major concern of parsing the schema file is I would never fully > trust the output from the script, even if the regex is extremely > paranoid and really strict, there is still a risk it contains a bug. That could possibly be resolved by us

Re: [HACKERS] pg_dump --split patch

> Robert Haas writes: >> I have to admit I'm a bit unsold on the approach as well.  It seems >> like you could write a short Perl script which would transform a text >> format dump into the proposed format pretty easily, and if you did >> that and published the script, then the next poor shmuck wh

Re: [HACKERS] pg_dump --split patch

Robert Haas writes: > On Mon, Jan 3, 2011 at 1:34 PM, Tom Lane wrote: >> Yeah, that's exactly it.  I can think of some possible uses for >> splitting up pg_dump output, but frankly "to ease diff-ing" is not >> one of them.  For that problem, it's nothing but a crude kluge that >> only sort-of hel

Re: [HACKERS] pg_dump --split patch

On Mon, Jan 3, 2011 at 1:34 PM, Tom Lane wrote: > Robert Haas writes: >> On the specific issue of overloaded functions, I have a feeling that >> the only feasible option is going to be to put them all in the same >> file.  If you put them in different files, the names will either be >> very long

Re: [HACKERS] pg_dump --split patch

2011/1/3 Tom Lane : > pg_dump from dumping objects in a consistent order ... and once you do > that, you don't need this patch. > Yeah, that's exactly it.  I can think of some possible uses for > splitting up pg_dump output, but frankly "to ease diff-ing" is not > one of them.  For that problem, it

Re: [HACKERS] pg_dump --split patch

Robert Haas writes: > On the specific issue of overloaded functions, I have a feeling that > the only feasible option is going to be to put them all in the same > file. If you put them in different files, the names will either be > very long (because they'll have to include the argument types) or

Re: [HACKERS] pg_dump --split patch

2011/1/3 Robert Haas : > will become confusing for users and hard for us to maintain.  We're > going to need to agree on something that won't be perfect for > everyone, but will hopefully be a sufficient improvement for enough > people to be worth doing. Good point. I think we can at least agree t

Re: [HACKERS] pg_dump --split patch

On Mon, Jan 3, 2011 at 7:11 AM, Dmitry Koterov wrote: > To me, this is a wonderful feature, thanks! I think many people would be > happy if this patch woud be included to the mainstream (and it is quite > short and simple). > About name ordering - I think that the problem exists for objects: > 1.

Re: [HACKERS] pg_dump --split patch

To me, this is a wonderful feature, thanks! I think many people would be happy if this patch woud be included to the mainstream (and it is quite short and simple). About name ordering - I think that the problem exists for objects: 1. Stored functions. 2. Foreign keys/triggers (objects which has o

Re: [HACKERS] pg_dump --split patch

On tis, 2010-12-28 at 20:51 -0500, Andrew Dunstan wrote: > try: > > diff -F '^CREATE' ... This works about 67% of the time and still doesn't actually tell at a glance what changed. It will only tell you what the change you are currently looking at probably belongs to. -- Sent via pgsql-hack

Re: [HACKERS] pg_dump --split patch

On tis, 2010-12-28 at 12:33 -0500, Tom Lane wrote: > (2) randomly different ordering of rows within a table. Your patch > didn't address that, unless I misunderstood quite a bit. This issue here is just comparing schemas, so that part is a separate problem for someone else. > I think the correct

Re: [HACKERS] pg_dump --split patch

On Thu, Dec 30, 2010 at 2:13 AM, Joel Jacobson wrote: > 2010/12/29 Dimitri Fontaine > > Please have a look at getddl: >> >> https://github.com/dimitri/getddl >> >> > Nice! Looks like a nifty tool. > When I tried it, "./getddl.py -f -F /crypt/funcs -d glue", I got the error > "No such file or di

Re: [HACKERS] pg_dump --split patch

2010/12/29 Dimitri Fontaine > Please have a look at getddl: > > https://github.com/dimitri/getddl > > Nice! Looks like a nifty tool. When I tried it, "./getddl.py -f -F /crypt/funcs -d glue", I got the error "No such file or directory: 'sql/schemas.sql'". While the task of splitting objects int

Re: [HACKERS] pg_dump --split patch

Joel Jacobson writes: > Solution: I propose a new option to pg_dump, --split, which dumps each > object to a separate file in a user friendly directory structure: Please have a look at getddl: https://github.com/dimitri/getddl Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL

Re: [HACKERS] pg_dump --split patch

2010/12/29 Tom Lane > I think they're fundamentally different things, because the previously > proposed patch is an extension of the machine-readable archive format, > and has to remain so because of the expectation that people will want > to use parallel restore with it. Joel is arguing for a s

Re: [HACKERS] pg_dump --split patch

Aidan Van Dyk writes: > On Wed, Dec 29, 2010 at 9:11 AM, Gurjeet Singh > wrote: >> AFAIK, that applies to parallel dumps of data (may help in --schema-only >> dumps too), and what you are trying is for schema. > Right, but one of the things it does is break the dump in to parts, > and put them

Re: [HACKERS] pg_dump --split patch

On Wed, Dec 29, 2010 at 9:11 AM, Gurjeet Singh wrote: > On Wed, Dec 29, 2010 at 8:31 AM, Joel Jacobson wrote: >> >> >> 2010/12/29 Aidan Van Dyk >>> >>> On Wed, Dec 29, 2010 at 2:27 AM, Joel Jacobson >>> wrote: >>> >>> >>> >>> So, how different (or not) is this to the "directory" format that wa

Re: [HACKERS] pg_dump --split patch

On Wed, Dec 29, 2010 at 8:31 AM, Joel Jacobson wrote: > > > 2010/12/29 Aidan Van Dyk > > On Wed, Dec 29, 2010 at 2:27 AM, Joel Jacobson >> wrote: >> >> >> >> So, how different (or not) is this to the "directory" format that was >> coming out of the desire of a parallel pg_dump? >> > > Not sure

Re: [HACKERS] pg_dump --split patch

2010/12/29 Aidan Van Dyk > On Wed, Dec 29, 2010 at 2:27 AM, Joel Jacobson > wrote: > > > > So, how different (or not) is this to the "directory" format that was > coming out of the desire of a parallel pg_dump? > Not sure what format you are referring to? Custom, tar or plain text? I noticed t

Re: [HACKERS] pg_dump --split patch

On Wed, Dec 29, 2010 at 2:27 AM, Joel Jacobson wrote: So, how different (or not) is this to the "directory" format that was coming out of the desire of a parallel pg_dump? a. -- Aidan Van Dyk                                             Create like a god, ai...@highrise.ca                    

Re: [HACKERS] pg_dump --split patch

2010/12/29 Andrew Dunstan > try: > > diff -F '^CREATE' ... > > cheers > > andrew > Embarrasing, I'm sure I've done `man diff` before, must have missed that one, wish I'd known about that feature before, would have saved me many hours! :-) Thanks for the tip! There are some other real-life use-

Re: [HACKERS] pg_dump --split patch

On 12/28/2010 08:18 PM, Joel Jacobson wrote: 2010/12/29 Tom Lane mailto:t...@sss.pgh.pa.us>> If you've solved the deterministic-ordering problem, then this entire patch is quite useless. You can just run a normal dump and diff it. No, that's only half true. Diff will do a good job

Re: [HACKERS] pg_dump --split patch

2010/12/29 Tom Lane > > If you've solved the deterministic-ordering problem, then this entire > patch is quite useless. You can just run a normal dump and diff it. > > No, that's only half true. Diff will do a good job minimizing the "size" of the diff output, yes, but such a diff is still quit

Re: [HACKERS] pg_dump --split patch

David Wilson writes: > On Tue, Dec 28, 2010 at 2:39 PM, Joel Jacobson wrote: >> I didn't include the arguments in the file name, as it would lead to very >> long file names unless truncated, and since the problem is very limited, I >> think we shouldn't include it. It's cleaner with just the name

Re: [HACKERS] pg_dump --split patch

2010/12/29 David Wilson > Why not place all overloads of a function within the same file? Then, > assuming you order them deterministically within that file, we sidestep the > file naming issue and maintain useful diff capabilities, since a diff of the > function's file will show additions or rem

Re: [HACKERS] pg_dump --split patch

On Tue, Dec 28, 2010 at 2:39 PM, Joel Jacobson wrote: > > I think you are right about functions (and aggregates) being the only > desc-type where two objects can share the same name in the same schema. > This means the problem of dumping objects in different order is a very > limited problem, onl

Re: [HACKERS] pg_dump --split patch

On Tue, Dec 28, 2010 at 4:57 PM, Andrew Dunstan wrote: > > > On 12/28/2010 04:44 PM, Joel Jacobson wrote: > >> >> >> >> >> The problem I see with suffixing a sequence id to the objects with name >>> collision is that one day the dump may name myfunc(int) as myfunc.sql and >>> after an overloaded

Re: [HACKERS] pg_dump --split patch

Andrew Dunstan writes: > On 12/28/2010 04:44 PM, Joel Jacobson wrote: >> Perhaps abbreviations are to prefer, e.g., myfunc_i, myfunc_i_c, etc >> to reduce the need of truncating filenames. > I think that's just horrible. Does the i stand for integer or inet? And > it will get *really* ugly for

Re: [HACKERS] pg_dump --split patch

2010/12/28 Andrew Dunstan > > I think that's just horrible. Does the i stand for integer or inet? And it > will get *really* ugly for type names with spaces in them ... > > True, true. But while "c" is too short, I think "character varying" is too long. Is there some convenient lookup table to c

Re: [HACKERS] pg_dump --split patch

On 12/28/2010 04:44 PM, Joel Jacobson wrote: The problem I see with suffixing a sequence id to the objects with name collision is that one day the dump may name myfunc(int) as myfunc.sql and after an overloaded version is created, say myfunc(char, int), then the same myfunc(int) may be d

Re: [HACKERS] pg_dump --split patch

Sent from my iPhone On 28 dec 2010, at 21:45, Gurjeet Singh wrote: The problem I see with suffixing a sequence id to the objects with name collision is that one day the dump may name myfunc(int) as myfunc.sql and after an overloaded version is created, say myfunc(char, int), then the same myfunc

Re: [HACKERS] pg_dump --split patch

On Tue, Dec 28, 2010 at 2:39 PM, Joel Jacobson wrote: > 2010/12/28 Gurjeet Singh > >> I would suggest the directory structure as: >> >> /crypt/pg.dump-split/schema-name-1/VIEWS/view-name-1.sql >> /crypt/pg.dump-split/schema-name-1/TABLES/table-name-1.sql >> ... >> /crypt/pg.dump-split/schema-nam

Re: [HACKERS] pg_dump --split patch

2010/12/28 Gurjeet Singh > I would suggest the directory structure as: > > /crypt/pg.dump-split/schema-name-1/VIEWS/view-name-1.sql > /crypt/pg.dump-split/schema-name-1/TABLES/table-name-1.sql > ... > /crypt/pg.dump-split/schema-name-2/VIEWS/view-name-1.sql > /crypt/pg.dump-split/schema-name-2/TA

Re: [HACKERS] pg_dump --split patch

On Tue, Dec 28, 2010 at 11:59 AM, Joel Jacobson wrote: > I don't follow, what do you mean with "failure modes"? The oid in the > filename? I suggested to use a sequence instead but you didn't comment on > that. Are there any other failure modes which could cause a diff -r between > two different

Re: [HACKERS] pg_dump --split patch

On Tue, Dec 28, 2010 at 11:00 AM, Joel Jacobson wrote: > Dear fellow hackers, > > Problem: A normal diff of two slightly different schema dump files (pg_dump > -s), will not produce a user-friendly diff, as you get all changes in the > same file. > > Solution: I propose a new option to pg_dump, -

Re: [HACKERS] pg_dump --split patch

Joel Jacobson writes: > 2010/12/28 Tom Lane >> That has at least as many failure modes as the other representation. > I don't follow, what do you mean with "failure modes"? The oid in the > filename? I suggested to use a sequence instead but you didn't comment on > that. Are there any other fail

Re: [HACKERS] pg_dump --split patch

On 12/28/2010 11:59 AM, Joel Jacobson wrote: 2010/12/28 Tom Lane mailto:t...@sss.pgh.pa.us>> That has at least as many failure modes as the other representation. I don't follow, what do you mean with "failure modes"? The oid in the filename? I suggested to use a sequence instead but yo

Re: [HACKERS] pg_dump --split patch

2010/12/28 Tom Lane > > That has at least as many failure modes as the other representation. > > I don't follow, what do you mean with "failure modes"? The oid in the filename? I suggested to use a sequence instead but you didn't comment on that. Are there any other failure modes which could caus

Re: [HACKERS] pg_dump --split patch

Joel Jacobson writes: > 2010/12/28 Tom Lane >> Joel Jacobson writes: >>> Solution: I propose a new option to pg_dump, --split, which dumps each >>> object to a separate file in a user friendly directory structure: >> >> Um ... how does that solve the claimed problem exactly? > Because then you

Re: [HACKERS] pg_dump --split patch

2010/12/28 Tom Lane > Joel Jacobson writes: > > Dear fellow hackers, > > Problem: A normal diff of two slightly different schema dump files > (pg_dump > > -s), will not produce a user-friendly diff, as you get all changes in the > > same file. > > > Solution: I propose a new option to pg_dump, -

Re: [HACKERS] pg_dump --split patch

Joel Jacobson writes: > Dear fellow hackers, > Problem: A normal diff of two slightly different schema dump files (pg_dump > -s), will not produce a user-friendly diff, as you get all changes in the > same file. > Solution: I propose a new option to pg_dump, --split, which dumps each > object to