Re: [ADMIN] Postgres Backup Utility
Having been a C/C++ developer many years before being a DBA, and having written ITIL software; How is migrating structure from a Development database to a test database whilst maintaining test data backwards? Besides, the OP was asking how to diff to databases and create ddl, not asking for us to comment on why he's doing it. Personally, I'd rather not go trawling through what can only be described as hundreds of thousands of lines of PostgreSQL log to find THE RIGHT DDL statements. To me that's surely a recipe for disaster, and certainly not a role I would expect to complete as a DBA. Yes the Developers should've kept the DDL for the changes, however; I know from experience that this isn't always the case, and have found it necessary to Diff databases on a regular basis for one reason or another. Each time I've done something like this I have resulted to coding shell scripts, or C programs that are bespoke to accomplish this, as I've yet to find a tool that does this to my satisfaction levels. -Original Message- From: Scott Marlowe [mailto:scott.marl...@gmail.com] Sent: 20 January 2011 04:41 To: French, Martin Cc: Bradley Holbrook; pgsql-admin@postgresql.org Subject: Re: [ADMIN] Postgres Backup Utility On Wed, Jan 19, 2011 at 12:12 AM, French, Martin fren...@cromwell.co.uk wrote: Ok, you say that you cannot drop and recreate, so you need to do this via alter statements only? That's obviously going to complicate matters, as a straight dump, drop, recreate, restore would be the fastest and by far simplest method. So, Ideally, you'll need to do a table def comparison over the two databases, and generate the necessary sql to amend the tables in test accordingly? Sorry but this is exactly backwards from good procedure. What you do it have your developers checkin the SQL code they used to alter the tables / add rows in the test database, and you apply that to testing / QA / staging / production. Any other way is a recipe for both disaster and DBA burnout. To make it easier, you can always turn on ddl logging on the developer's databases and then just trawl the logs for what they did. Still easier than trying to compare schemas and figure out what's different and how to write the SQL to make it happen. ___ This email is intended for the named recipient. The information contained in it is confidential. You should not copy it for any purposes, nor disclose its contents to any other party. If you received this email in error, please notify the sender immediately via email, and delete it from your computer. Any views or opinions presented are solely those of the author and do not necessarily represent those of the company. PCI Compliancy: Please note, we do not send or wish to receive banking, credit or debit card information by email or any other form of communication. Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __ -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 5:00 AM, French, Martin fren...@cromwell.co.uk wrote: Having been a C/C++ developer many years before being a DBA, and having written ITIL software; How is migrating structure from a Development database to a test database whilst maintaining test data backwards? It's not. doing by running diffs on ddl / schema is. This is what the OP asked for: I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables). Besides, the OP was asking how to diff to databases and create ddl, not asking for us to comment on why he's doing it. Personally, I'd rather Re-read what he asked for. He did not ask how to run diffs of schema, he asked how to update it with changes from a test database. The best way to do this is to check in changes to the database and lookup tables so the DBA can then apply them when needed to the various other databases. But hey, feel free to answer a question the OP didn't ask with an answer he doesn't need. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 5:00 AM, French, Martin fren...@cromwell.co.uk wrote: Personally, I'd rather not go trawling through what can only be described as hundreds of thousands of lines of PostgreSQL log to find THE RIGHT DDL statements. Oh that's easy. Grep out the statements that start with alter etc, ptu them all in a big email, send them to all the developers and tell them to let you know which ones need applying to QA / Staging / etc. Can't get your DDL to the DBA? Your code won't work. You break the build because of that, you get to fix it. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 8:46 AM, Scott Marlowe scott.marl...@gmail.com wrote: On Thu, Jan 20, 2011 at 5:00 AM, French, Martin fren...@cromwell.co.uk wrote: Personally, I'd rather not go trawling through what can only be described as hundreds of thousands of lines of PostgreSQL log to find THE RIGHT DDL statements. Oh that's easy. Grep out the statements that start with alter etc, ptu them all in a big email, send them to all the developers and tell them to let you know which ones need applying to QA / Staging / etc. Can't get your DDL to the DBA? Your code won't work. You break the build because of that, you get to fix it. Note that I also mentioned turning on logging JUST ddl statements. Assumiing you're not logging EVERYTHING in development (and why would you) you should only have a few dozen lines of log to grep, and little or none of it will be things other than DDL. -- To understand recursion, one must first understand recursion. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
[ADMIN] State of multi-master replication options
Hi All; I suspect I know the answer to this... What's the current state of multi-master replication for PostgreSQL? Is Bucardo the only true master/master solution out there that might be worthy of a production push? Is Postres-R a candidate at this point (I suspect not)? Are there any other master/master or preferably multi-master (3+ masters) solutions available? Thanks in advance... -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] State of multi-master replication options
On Thu, Jan 20, 2011 at 9:53 AM, CS DBA cs_...@consistentstate.com wrote: Hi All; I suspect I know the answer to this... What's the current state of multi-master replication for PostgreSQL? Is Bucardo the only true master/master solution out there that might be worthy of a production push? Is Postres-R a candidate at this point (I suspect not)? Are there any other master/master or preferably multi-master (3+ masters) solutions available? Unless you roll your own with separate sequence ranges and inherited tables via slony, yes, Bucardo seems to be it right now. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
Thanks Scott... a couple comments. Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). The DDL Logging sounds like a sufficient solution, can it be configured to only record create and alter commands (or create or replace commands on functions or updates on sequences, etc)? I'd likely write a script to have this emailed to me at the end of every day. I'm going to google DDL logging (never heard of it), but any good resources off the top of your head? Martin French is right though, ask your developers to write down all their SQL struct changes and they look at you funny... and being a developer myself I'd look at me funny. If you forget just once you're screwed into a day sifting through tables and code. -Original Message- From: Scott Marlowe [mailto:scott.marl...@gmail.com] Sent: January-20-11 7:40 AM To: French, Martin Cc: Bradley Holbrook; pgsql-admin@postgresql.org Subject: Re: [ADMIN] Postgres Backup Utility On Thu, Jan 20, 2011 at 5:00 AM, French, Martin fren...@cromwell.co.uk wrote: Having been a C/C++ developer many years before being a DBA, and having written ITIL software; How is migrating structure from a Development database to a test database whilst maintaining test data backwards? It's not. doing by running diffs on ddl / schema is. This is what the OP asked for: I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables). Besides, the OP was asking how to diff to databases and create ddl, not asking for us to comment on why he's doing it. Personally, I'd rather Re-read what he asked for. He did not ask how to run diffs of schema, he asked how to update it with changes from a test database. The best way to do this is to check in changes to the database and lookup tables so the DBA can then apply them when needed to the various other databases. But hey, feel free to answer a question the OP didn't ask with an answer he doesn't need. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook operations_brad...@servillian.ca wrote: Thanks Scott... a couple comments. Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). The DDL Logging sounds like a sufficient solution, can it be configured to only record create and alter commands (or create or replace commands on functions or updates on sequences, etc)? I'd likely write a script to have this emailed to me at the end of every day. I'm going to google DDL logging (never heard of it), but any good resources off the top of your head? It's basically logging anything that changes the structure of the database. It would be easy enough to grep out what you do and don't want later. Martin French is right though, ask your developers to write down all their SQL struct changes and they look at you funny... and being a developer myself I'd look at me funny. If you forget just once you're screwed into a day sifting through tables and code. I've worked in three different shops now as a dev-dba and sysadmin, and in all three, all DDL changes had to be committed and / or handed over to the DBAs. period. Look funny all they want, they either give up the DDL or their code doesn't get pushed off dev servers onto anything else. At the very least they should be able to tell you which tables changed to go with which code changes, or you're not sure what code you can and can't push. I get both of your point on this, but it's a discipline issue that needs sorting out with the developers if you want to have reproduceable ddl changes in all your systems that match the code changes. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
Followup, note that you can set the log_statement='ddl' for an entire pg cluster, for a single database, or for a single user, if that helps. logging ddl does not log dml, or data changes, just structural changes. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] State of multi-master replication options
scott.marl...@gmail.com (Scott Marlowe) writes: On Thu, Jan 20, 2011 at 9:53 AM, CS DBA cs_...@consistentstate.com wrote: Hi All; I suspect I know the answer to this... What's the current state of multi-master replication for PostgreSQL? Is Bucardo the only true master/master solution out there that might be worthy of a production push? Is Postres-R a candidate at this point (I suspect not)? Are there any other master/master or preferably multi-master (3+ masters) solutions available? Unless you roll your own with separate sequence ranges and inherited tables via slony, yes, Bucardo seems to be it right now. Sounds right. There were several attempts to come up with Slony successors doing multimaster replication; it turns out to be a really hard problem to do this in a general fashion, and have efficient results. It probably fits in with Brewer's CAP Theorem: http://en.wikipedia.org/wiki/CAP_theorem CAP indicates you can choose between Consistency, Availability, and Partition Tolerance, and you can notably not have all three. When building would-be general purpose mechanisms for Postgres, it seems rather undesirable to throw out Consistency; we usually imagine that being able to require consistency was one of the reasons people thought it wise to use Postgres in the first place :-). But retaining consistency is pretty problematic. - Work on one node might be inconsistent with what's happening on another node. Two kinds of examples that would have differing kinds of behavior include: a) Maintaining foreign key relationships b) Managing account balances - Evaluating those sorts of consistency so as to *prevent* inconsistency is a Mighty Expensive thing to do. (Slony-II tried doing this, and found it punitively expensive to performance, as well as biting back at applications. Postgres-R is on the same road, so one may expect certain behaviors there.) - Bucardo takes the approach of having application rules where you have to pre-code what to do when it discovers inconsistencies when trying to apply changes to other nodes. That's not general purpose in the sense that you need to write rules specific to your system's behavior. -- http://linuxfinances.info/info/slony.html Rules of the Evil Overlord #14. The hero is not entitled to a last kiss, a last cigarette, or any other form of last request. http://www.eviloverlord.com/ -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 10:52 AM, Scott Marlowe scott.marl...@gmail.com wrote: Followup, note that you can set the log_statement='ddl' for an entire pg cluster, for a single database, or for a single user, if that helps. logging ddl does not log dml, or data changes, just structural changes. also also wik, you can use the log_line_prefix to make it so that you can grep out what you need easier too. #log_line_prefix = '' # special values: # %u = user name # %d = database name # %r = remote host and port # %h = remote host # %p = process ID # %t = timestamp without milliseconds # %m = timestamp with milliseconds # %i = command tag # %c = session ID # %l = session line number # %s = session start timestamp # %v = virtual transaction ID # %x = transaction ID (0 if none) # %q = stop here in non-session #processes # %% = '%' # e.g. '%u%%%d ' -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
-Original Message- From: Scott Marlowe [mailto:scott.marl...@gmail.com] Sent: Thursday, January 20, 2011 12:51 PM To: Bradley Holbrook Cc: French, Martin; pgsql-admin@postgresql.org Subject: Re: Postgres Backup Utility On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook operations_brad...@servillian.ca wrote: Thanks Scott... a couple comments. Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). The DDL Logging sounds like a sufficient solution, can it be configured to only record create and alter commands (or create or replace commands on functions or updates on sequences, etc)? I'd likely write a script to have this emailed to me at the end of every day. I'm going to google DDL logging (never heard of it), but any good resources off the top of your head? It's basically logging anything that changes the structure of the database. It would be easy enough to grep out what you do and don't want later. Martin French is right though, ask your developers to write down all their SQL struct changes and they look at you funny... and being a developer myself I'd look at me funny. If you forget just once you're screwed into a day sifting through tables and code. I've worked in three different shops now as a dev-dba and sysadmin, and in all three, all DDL changes had to be committed and / or handed over to the DBAs. period. Look funny all they want, they either give up the DDL or their code doesn't get pushed off dev servers onto anything else. At the very least they should be able to tell you which tables changed to go with which code changes, or you're not sure what code you can and can't push. I get both of your point on this, but it's a discipline issue that needs sorting out with the developers if you want to have reproduceable ddl changes in all your systems that match the code changes. Completely agree with Scott. Only want to add that in this kind of development environment: development/test/production - Source code versioning software is absolute necessity (there are many: CSV, SourceSafe, Perforce, ... - pick your choice). Regards, Igor Neyman -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook operations_brad...@servillian.ca wrote: Thanks Scott... a couple comments. Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). The DDL Logging sounds like a sufficient solution, can it be configured to only record create and alter commands (or create or replace commands on functions or updates on sequences, etc)? I'd likely write a script to have this emailed to me at the end of every day. I'm going to google DDL logging (never heard of it), but any good resources off the top of your head? It's basically logging anything that changes the structure of the database. It would be easy enough to grep out what you do and don't want later. Martin French is right though, ask your developers to write down all their SQL struct changes and they look at you funny... and being a developer myself I'd look at me funny. If you forget just once you're screwed into a day sifting through tables and code. I've worked in three different shops now as a dev-dba and sysadmin, and in all three, all DDL changes had to be committed and / or handed over to the DBAs. period. Look funny all they want, they either give up the DDL or their code doesn't get pushed off dev servers onto anything else. At the very least they should be able to tell you which tables changed to go with which code changes, or you're not sure what code you can and can't push. I get both of your point on this, but it's a discipline issue that needs sorting out with the developers if you want to have reproduceable ddl changes in all your systems that match the code changes. Completely agree with Scott. Me too. The idea that a developer can change the schema without telling anyone is laughable. The idea that someone else has to reverse engineer schema changes is ludicrous. We have a strict procedure that every schema change has to be accompanied by a script that applies the changes. It has to be checked into Subversion along with all the other code. Make a new rule: Developers have to document schema changes. Institute a zero-tolerance policy for omissions. If a developer can create the DDL to make the change, surely it's not too much trouble to spend another minute adding it to a file and checking that file in. Craig -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] binary logs: a location other than pg_xlog??
Tom Lane wrote: Lou Picciano loupicci...@comcast.net writes: Sure - of course - performance, too! It just seems like such an obvious candidate for a configurable option; I'm a bit surprised it isn't in there... L It *is* configurable: make pg_xlog a symlink. The reason we don't handle this via postgresql.conf is that you could too easily shoot yourself in the foot by breaking the link between a data directory and its xlogs. There are no postgresql.conf entries that say destroy_all_my_data_immediately_please = on but that's more or less what an easily changeable pg_xlog pointer would be. In short: this is not something to change if you don't know what you're doing. If you do, you should be able to handle it using the mechanism provided. Old thread, but no one mentioned that initdb has an option to create a symlink for you for pg_xlog: -X directory, --xlogdir=directory This option specifies the directory where the transaction log should be stored. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Jan 20, 2011, at 12:42 PM, Bradley Holbrook wrote: Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). Using a diff tool, how would you know if a column was dropped and another one added versus simply renaming a column? Won't that kind of difference matter when you want to apply the changes to your production database? John DeSoi, Ph.D. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Postgres Backup Utility
On Thu, Jan 20, 2011 at 6:08 PM, John DeSoi de...@pgedit.com wrote: On Jan 20, 2011, at 12:42 PM, Bradley Holbrook wrote: Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). Using a diff tool, how would you know if a column was dropped and another one added versus simply renaming a column? Won't that kind of difference matter when you want to apply the changes to your production database? Yikes! That could be bad. Drop column xyz. Hope that data wasn't important. To see if the column was renamed or something you'd have to examine its rows to see if they matched a previous version. I think that at the least I'd trawl the logs with ddl logging turned on to get changes. But seriously, I'd just lay down the law. You want changes in the dbs outside development, you give me, the DBA, your DDL statements. No statements, no code push. -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
[ADMIN] Read deleted record
Hi all.. Start from vacuum feature information from PGSQL helps documentation, it telling me that Postgresql didn't delete data permanently when we execute delete command, it just made the data invalid. By following this email archive : http://archives.postgresql.org/pgsql-admin/2005-01/msg00176.php I could get the table files, my question is, how could I read again the deleted record from that files. I have tried to post this in to pgsql-general, unfortunately got no response.. best regards, Soultani
