Re: [HACKERS] management of large patches
Robert Haas wrote: - MERGE - checkpoint improvements As far as these two go, the state of MERGE is still rougher than I would like. The code itself isn't too hard to read, and that the errors that are popping up tend to be caught by assertions (rather than just being mysterious crashes) makes me feel a little better that there's some defensive coding in there. It's still a 3648 line patch that touches grammar, planner, and executor bits though, and I've been doing mainly functional and coding style review so far. I'm afraid here's not too many committers in a good position to actually consume the whole scope of this thing for a commit level review. And the way larger patches tend to work here, I'd be surprised to find it passes through such a review without some as yet unidentified major beef appearing. Will see what we can do to help move this forward more before the CF start. The checkpoint changes I'm reworking are not really large from a code complexity or size perspective--I estimate around 350 lines of diff, with the rough version I submitted to CF2010-11 at 258. I suspect it will actually be the least complicated patch to consume from that list, from a committer perspective. The complexity there is mainly in the performance testing. I've been gearing up infrastructure the last couple weeks to automate and easily publish all the results I collect there. The main part that hasn't gone through any serious testing yet, auto-tuning the spread interval, will also be really easy to revert if a problem is found there. With Simon and I both reviewing each others work on this already, I hope we can keep this one from clogging the committer critical path you're worried about here. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services and Supportwww.2ndQuadrant.us PostgreSQL 9.0 High Performance: http://www.2ndQuadrant.com/books -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] contrib/snapshot
On Jan 2, 2011, at 6:50 PM, Joel Jacobson wrote: 2011/1/3 Joel Jacobson j...@gluefinance.com 2011/1/2 Jim Nasby j...@nasby.net Is it actually limited to functions? ISTM this concept would be valuable for anything that's not in pg_class (in other words, anything that doesn't have user data in it). Instead of limiting the support to functions, perhaps it would make more sense to limit it to all non-data objects? Is there a term for the group of object types not carrying any user data? My bad, I see you already answered both my questions. So, it does make sense, and the term for non-data object types is therefore non-pg_class, non-class or perhaps non-relation objects? The generic term for objects that keep their metadata in pg_class is relation. Actually, now that I think about it, existence in pg_class isn't a good determining factor, because there's stuff like types in there. Aside from tables and sequences, you might also want to exclude indexes, or at least provide the option to, since rebuilding them could take a significant amount of time. -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] page compression
On Jan 2, 2011, at 5:36 PM, Simon Riggs wrote: On Tue, 2010-12-28 at 09:10 -0600, Andy Colson wrote: I know its been discussed before, and one big problem is license and patent problems. Would like to see a design for that. There's a few different ways we might want to do that, and I'm interested to see if its possible to get compressed pages to be indexable as well. For example, if you compress 2 pages into 8Kb then you do one I/O and out pops 2 buffers. That would work nicely with ring buffers. Or you might try to have pages 8Kb in one block, which would mean decompressing every time you access the page. That wouldn't be much of a problem if we were just seq scanning. Or you might want to compress the whole table at once, so it can only be read by seq scan. Efficient, but not indexes. FWIW, last time I looked at how Oracle handled compression, it would only compress existing data. As soon as you modified a row, it ended up un-compressed, presumably in a different page that was also un-compressed. I wonder if it would be feasible to use a fork to store where a compressed page lives inside the heap... if we could do that I don't see any reason why indexes wouldn't work. The changes required to support that might not be too horrific either... -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery conflict monitoring
On Mon, Jan 3, 2011 at 00:23, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2010-12-27 at 14:39 +0100, Magnus Hagander wrote: On Thu, Dec 23, 2010 at 13:09, Magnus Hagander mag...@hagander.net wrote: This patch adds counters and views to monitor hot standby generated recovery conflicts. It extends the pg_stat_database view with one column with the total number of conflicts, and also creates a new view pg_stat_database_conflicts that contains a breakdown of exactly what caused the conflicts. Documentation still pending, but comments meanwhile is of course appreciated ;) Heikki pointed out over IM that it's pointless to count stats caused by recovery conflict with drop database - since we drop the stats record as soon as it arrives anyway. Here's an updated patch that removes that, and also adds some documentation. I like the patch, well inspired, code in the right places AFAICS. No code comments at all. Thanks for reviewing! Couple of thoughts: * are we safe to issue stats immediately before issuing FATAL? Won't some of them get lost? They shouldn't - not more than other stats messages. Those are often flushed from on_shmem_exit() which I think runs even later. * Not clear what I'd do with database level information, except worry a lot. Maybe an option to count conflicts per user would be better, since at least we'd know exactly who was affected by those. Just an idea. Depends on the usage scenario. In a lot of dedicated environments you really only have one database - but there are many environments where you do have multiple and it's quite useful to see them separately. And you can of course very easily sum() them up for a total count, since it's a view... Better keep the detail than throw it away, even if that part isn't useful in *all* cases... Grouping by user would potentially be helpful - I agree. However, that goes for most pgstat counters (number of seqscans, tuples read etc are interesting per user as well in some cases). So doing that right would mean adding per-user tracking all across pgstats in some smart way - something we don't do now at all. So I see that as a separate issue. * Would it better to have a log_standby_conflicts that allowed the opportunity to log the conflicting SQL, duration until cancelation etc? Logging is useful to figure out why you have a certain scenario, yes. But absolutely not as a *replacement* for the statistics counters, but as an addition. Just like we have (the now incorrectly named) pg_stat_bgwriter view *and* log_checkpoints... Different usecases for the same basic information. I'd rather have what you have than nothing at all though... the new hot_standby_feedback mode should be acting to reduce these, so it would be useful to have this patch enabled for testing that feature. It will help reduce it, but not take it away, right? Plus, it's an optional feature... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] contrib/snapshot
2011/1/3 Andrew Dunstan and...@dunslane.net contrib in PostgreSQL means a module maintained by the backend developers. But it's not clear to me that there is any particular reason why this should be in contrib. Then I definitively think contrib is the only possible place for this module. If the module will not be maintained by the backend developers, noone (including myself) will trust the module to perform the sensistive tasks in a mission critical production database. Since the module depends on pg_catalog system tables, it's must be updated if the they would change in future versions of PostgreSQL, and I wouldn't trust any other team than the backend developers to do it. I'm happy to continue hacking on the module until it's 100% working, stable, thoroughly tested and accepted by the backend developers. It's not working 100% yet, for instance, I'm currently working on making sure objects are created/dropped in an order not breaking any dependencies. -- Best regards, Joel Jacobson Glue Finance
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, 2011-01-03 at 01:53 -0500, Greg Smith wrote: In advance of the planned but not available yet ability to lock individual index key values, locking the whole table is the only possible implementation that can work correctly here I'm aware of. This was discussed here http://archives.postgresql.org/pgsql-hackers/2010-10/msg01903.php with suggested resolutions for this release here http://archives.postgresql.org/pgsql-hackers/2010-10/msg01907.php In summary, that means we can either 1. Lock the table for ShareRowExclusiveLock 2. throw a SERIALIZABLE error, if we come up against a row that cannot be neither MATCHED nor NON MATCHED. 3. Bounce the patch to 9.2, commit early and then work on a full concurrency solution before commit. The solution strawman is something like EvalPlanQual with a new snapshot for each re-checked row, emulating the pattern of snapshots/rechecks that would happen in a PL/pgSQL version of an UPSERT. Either way, we're saying that MERGE will not support concurrent operations safely, in this release. Given the continued lack of test cases for this patch, and the possible embarrassment over not doing concurrent actions, do we think (3) is the right road? -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Visual Studio 2010/Windows SDK 7.1 support
On Mon, Jan 3, 2011 at 08:19, Brar Piening b...@gmx.de wrote: Hi, i'v created a patch enables support for building PostgreSQL with Visual Studio 2010 or Microsoft Windows SDK for Windows 7 and .NET Framework 4 (Windows SDK 7.1). You can grab it from http://www.piening.info/VS2010.patch This patch does not apply at all to my repository. Every single hunk fails - I have a feeling it might have double line-ending encodings or something like that? Since you're using git (at least eh patch indicates so), have you perhaps published a git branch somewhere that I could try pulling from instead? Or if not - just to be sure, you are basing this off the master branch, I hope? It's diffed against current head + running perltidy -b -bl -nsfs -naws -l=100 -ole=unix *.pl *.pm as described in the README file (which seems not to have been run before committing Mkvcbuild.pm the last time). Yeah, it looks that way - it's missing the ordering of the contrib arrays. I'll run it once for that now, and then please rebase your patch on top of that - makes it easier to review it. If you still see any changes on top of that, then there's something strange going on.. It is problably neither the perfect way to introduce VS2010 support (my perl is better than my C but probably still not what you are used to) nor is it my way to try to make you officially support VS 2010. But perhaps it's something you could start with once you decide to upgrade the msvc toolchain. Oh, starting with a patch is definitely not the wrong way :-) it does look like they changed the file format extensively this time though - that's annoying, but hopefully they will stick to the new format this time... The SDK 7.1 build produces tons of warnings which are mostly macro redefinitions of EIDRM, EMSGSIZE, EAFNOSUPPORT, EWOULDBLOCK, ECONNRESET, EINPROGRESS, ENOBUFS, EPROTONOSUPPORT, ECONNREFUSED and EOPNOTSUPP which seem to have found their way into errno.h finally. Cutting those out of src\include\pg_config_os.h and src\interfaces\libpq\win32.h makes the project build pretty clean. I resisted the temptation to parse them out of those files during Mkvcbuild::mkvcbuild as this should probably be handled by some preprocessor defines. Yes, it's good that you resisted that :-) It should be done with something like #if _MSC_VER = 1400 (with 1400 replaced with whatever version is appropriate) There's at elast one example in c.h already, and several others spread through files and code. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery conflict monitoring
On Mon, 2011-01-03 at 10:03 +0100, Magnus Hagander wrote: I like the patch, well inspired, code in the right places AFAICS. No code comments at all. Thanks for reviewing! All good here. Test and commit please. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] management of large patches
Robert Haas robertmh...@gmail.com writes: - extensions - may need 2 or more commits I'm now basically done with coding, I'm writing the docs for the upgrade patch and preparing the upgrade SQL files for pre-9.1 to 9.1 upgrades of the contrib modules. Doing that, I've been cleaning up or reorganising some code: I will backport some of those changes to the main extension patch. So I expect to send both extension.v23.patch and extension-upgrade.v1.patch this week. As the main extension patch as received lots of detailed reviews (both user level and code level) by commiters already, I'm not expecting big surprises for the last commitfest. The upgrade patch design has been discussed in detail on-list too. Dust has settled here. Meanwhile, there's this bugfix for HEAD that I've sent: http://archives.postgresql.org/pgsql-hackers/2011-01/msg00078.php Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery conflict monitoring
Couple of doc suggestions: --- doc/src/sgml/high-availability.sgml + The number of query cancels and the reason for them can be viewed using + the structnamepg_stat_database_conflicts/ system view on the slave + server. For compleness sake, this should also mention the per-database summary, even though I'm not sure how valuable that view is. Also, on a standby server instead of on the slave server here. slave is mentioned once as a synonym in high-availability.sgml once, but that's it, and there can be more than one standby you want to pull these stats from. *** doc/src/sgml/monitoring.sgml ! number of rows returned, fetched, inserted, updated and deleted, and ! total number of queries cancelled due to conflict with recovery. This would be clearer if it said you're talking about standby recovery here, and possibly that this info is only available on the standby. I could see someone reading this and thinking it's possible for general database crash recovery to produce cancelled queries, instead of the way connections are actually blocked until that's done. ! entrystructnamepg_stat_database_conflicts/ ! entryOne row per database, showing database OID, database name and ! the number of queries that have been cancelled in this database due to ! dropped tablespaces, lock timeouts, old snapshots, pinned buffers and ! deadlocks. A clarification that you're talking about standby query cancellation here might be helpful too. I don't think that's necessary for all of the detailed pg_stat_get_* functions that regular users are less likely to care about, just these higher level ones. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services and Supportwww.2ndQuadrant.us PostgreSQL 9.0 High Performance: http://www.2ndQuadrant.com/books -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication as a separate permissions
On Fri, Dec 31, 2010 at 15:38, Magnus Hagander mag...@hagander.net wrote: On Thu, Dec 30, 2010 at 15:54, Peter Eisentraut pete...@gmx.net wrote: On ons, 2010-12-29 at 11:09 +0100, Magnus Hagander wrote: I've applied this version (with some minor typo-fixes). This page is now somewhat invalidated: http://developer.postgresql.org/pgdocs/postgres/role-attributes.html Hmm. Somehow I missed that page completely when looking through the docs. I'll go update that. BTW, shouldn't CONNECTION LIMIT be listed on that page? and INHERIT? And VALID UNTIL? They're all role attributes, no? At least the last two certainly interact with the authentication system... First, it doesn't mention the replication privilege, and second it continues to claim that superuser status bypasses all permission checks. Well, that was *already* wrong. superuser doesn't bypass NOLOGIN. That doesn't mean it shouldn't be fixed, but that's independent of the replication role. I've committed a fix for this. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep Design
On Sun, 2011-01-02 at 20:53 +, Simon Riggs wrote: On Sun, 2011-01-02 at 12:13 -0800, MARK CALLAGHAN wrote: On Thu, Dec 30, 2010 at 9:02 PM, Robert Haas robertmh...@gmail.com wrote: reads MySQL documentation I see now that you've tried to design this feature in a way that is similar to MySQL's offering, which does have some value. But it appears to me that the documentation you've written here is substantially similar to the MySQL 5.5 reference documentation. That could get us into a world of legal trouble - that documentation is not even open source, let alone BSD. http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html Sorry, only just read that bit. Are they that similar? My docs are about 3 times longer and cover all sorts of things. I didn't intentionally copy anything, but that doesn't really matter, what matters is that if you think they are similar, legal people might. I've only read the URL above, not the other links from it. Robert, Can you identify which paragraphs need to be re-written? I won't argue, I will just rewrite them or delete them and start afresh. Thanks for being eagle-eyed. Been through this to check. The first paragraph was pretty similar, so I've reworded that a little. Trying to make a readable paragraph that introduces the need for sync rep, based upon the deficiencies of async rep is going to be very similar, whatever we do. I can't see any other way to introduce a feature other than to explain the potential problem and then explain how the new feature resolves that. Parameters are set on both primary and standby. That design existed long before I looked at the MySQL manual, and yes there is similarity, which led to similar descriptions of how that works. Bear in mind that the two designs are very different. I can't see any other similarities. If anybody else can, please shout. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Recovery conflict monitoring
On Mon, Jan 3, 2011 at 11:35, Greg Smith g...@2ndquadrant.com wrote: Couple of doc suggestions: --- doc/src/sgml/high-availability.sgml + The number of query cancels and the reason for them can be viewed using + the structnamepg_stat_database_conflicts/ system view on the slave + server. For compleness sake, this should also mention the per-database summary, even though I'm not sure how valuable that view is. Also, on a standby server instead of on the slave server here. slave is mentioned once as a synonym in high-availability.sgml once, but that's it, and there can be more than one standby you want to pull these stats from. Good point, changed and added. *** doc/src/sgml/monitoring.sgml ! number of rows returned, fetched, inserted, updated and deleted, and ! total number of queries cancelled due to conflict with recovery. This would be clearer if it said you're talking about standby recovery here, and possibly that this info is only available on the standby. I could see someone reading this and thinking it's possible for general database crash recovery to produce cancelled queries, instead of the way connections are actually blocked until that's done. ! entrystructnamepg_stat_database_conflicts/ ! entryOne row per database, showing database OID, database name and ! the number of queries that have been cancelled in this database due to ! dropped tablespaces, lock timeouts, old snapshots, pinned buffers and ! deadlocks. A clarification that you're talking about standby query cancellation here might be helpful too. I don't think that's necessary for all of the detailed pg_stat_get_* functions that regular users are less likely to care about, just these higher level ones. Yeah, those both make sense - I've updated the docs and am running tests ATM. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Visual Studio 2010/Windows SDK 7.1 support
Original-Nachricht
Datum: Mon, 3 Jan 2011 10:44:19 +0100
Von: Magnus Hagander mag...@hagander.net
An: Brar Piening b...@gmx.de
CC: pgsql-hackers@postgresql.org
Betreff: Re: [HACKERS] Visual Studio 2010/Windows SDK 7.1 support
This patch does not apply at all to my repository. Every single hunk
fails - I have a feeling it might have double line-ending encodings or
something like that?
double line-endings indeed!
Surprisingly only some of them (CRCRLF follow some CRLF's).
I was quite unsure about the best line endings for this patch so I probably
messed it up by converting them forth and back. Still I'm not sure if there's
something wrong with my git config - perhaps due to using git-external-diff (I
wasn't able to find win32 binaries for filterdiff).
Sorry about that!
perl -e open(INFILE, 'VS2010.patch');open(OUTFILE,
'VS2010_fixed.patch');binmode(INFILE);binmode(OUTFILE);$/=\\015\015\012\;$\=\\015\012\;while(INFILE){chomp;
print OUTFILE;}close(INFILE);close(OUTFILE)
fixes the problem.
Yeah, it looks that way - it's missing the ordering of the contrib
arrays. I'll run it once for that now, and then please rebase your
patch on top of that - makes it easier to review it.
If you still see any changes on top of that, then there's something
strange going on..
No that was probably all.
My patch - fixed as described above - should apply to your repository once
you've run perltidy.
I'll rebase the patch as soon as I return from work unless you tell me
otherwise.
Best Regards
Brar
--
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility
On Sun, Jan 2, 2011 at 15:07, Peter Eisentraut pete...@gmx.net wrote: On tis, 2010-12-28 at 13:13 +0100, Magnus Hagander wrote: My pg_streamrecv no longer works with 9.1, because it returns PGRES_COPY_BOTH instead of PGRES_COPY_OUT when initating a copy. That's fine. So I'd like to make it work on both. Specifically, I would like it to check for PGRES_COPY_BOTH if the server is 9.1 and PGRES_COPY_OUT if it's 9.0. Which can be done by checking the server version. ISTM that the correct fix is to increment to protocol version number to 3.1 and send PGRES_COPY_OUT if the client requests version 3.0. That's what the version numbers are for, no? In a way - yes. I assume we didn't do that because it's considered internal. It still won't help in my situation though - I need to know what version of the libpq headers I have in order to even be able to *compile* the program. At runtime, I could check against the server version, and get around it. That said, if we are going to incorporate pg_streamrecv in the backend for 9.1, *my* problem goes away, as does the problem directly related to the change of PGRES_COPY_OUT. But the basic principle of the problem still remains... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
To me, this is a wonderful feature, thanks! I think many people would be happy if this patch woud be included to the mainstream (and it is quite short and simple). About name ordering - I think that the problem exists for objects: 1. Stored functions. 2. Foreign keys/triggers (objects which has owning objects). It is wonderful that you store all functions with the same name to the same file. To order them within this file we may simply compare the first definition line lexicographically (or - first line which differs one function definition from another). Foreign key/triggers ordering problem is described by me at http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg114586.html The big problem is with triggers: many triggers may have the same name, but be bound to different tables. It would be great to include these triggers to table's definition or, at least, have separated files for each trigger+table pair. On Wed, Dec 29, 2010 at 6:21 PM, Joel Jacobson j...@gluefinance.com wrote: 2010/12/29 Tom Lane t...@sss.pgh.pa.us I think they're fundamentally different things, because the previously proposed patch is an extension of the machine-readable archive format, and has to remain so because of the expectation that people will want to use parallel restore with it. Joel is arguing for a split-up of the text dump format. Yes, exactly. My patch is of course also a lot smaller :-) pg_dump-directory.diff.: 112 853 bytes pg-dump-split-plain-text-files-9.1devel.patch..: 5 579 bytes I just tried the pg_dump-directory.diff patch. The only thing is has in common with my patch is it writes data to different files, and it's only the data which is splitted into different files, the schema appears to go into the single file TOC. Example, pg_dump-directory.diff: $ ./pg_dump -f /crypt/dirpatch -F d -s glue $ ls -la /crypt/dirpatch/ TOC (1 file) $ rm -rf /crypt/dirpatch $ ./pg_dump -f /crypt/dirpatch -F d glue $ ls /crypt/dirpatch/ 6503.dat 6504.dat ...lots of files... 6871.dat 6872.dat 6873.dat 6874.dat TOC Example, pg_dump --split patch: $ pg_dump -f /crypt/splitpatch -F p --split -s glue $ ls /crypt/splitpatch* /crypt/splitpatch (file) /crypt/splitpatch-split: (directory) myschema1 myschema2 public $ ls /crypt/splitpatch-split/public/ AGGREGATE CONSTRAINT FK_CONSTRAINT FUNCTION INDEX SEQUENCE TABLE TRIGGER TYPE VIEW $ ls /crypt/splitpatch-split/public/FUNCTION/ myfunc.sql otherfunc.sql $ cat /crypt/splitpatch -- -- PostgreSQL database dump -- SET statement_timeout = 0; SET client_encoding = 'UTF8'; SET standard_conforming_strings = off; SET check_function_bodies = false; SET client_min_messages = warning; SET escape_string_warning = off; ...etc... \i /crypt/splitpatch-split/public/FUNCTION/myfunc.sql \i /crypt/splitpatch-split/public/FUNCTION/otherfunc.sql -- Best regards, Joel Jacobson Glue Finance
Re: [HACKERS] Problems with autovacuum and vacuum
Hello, Filip 2011/1/1 Filip Rembiałkowski plk.zu...@gmail.com 2010/12/30 JotaComm jota.c...@gmail.com Hello, Last week I had a serious problem with my PostgreSQL database. My autovacuum is OFF, but in September it started to prevent the transaction wraparoud; however last week the following message appeared continuously in my log: WARNING: database production must be vacuumed within 4827083 transactions HINT: To avoid a database shutdown, execute a full-database VACUUM in production. This message appeared for five to six hours; after that, the message disappeared from log. Any idea about what could have happened? probably another wraparaund-forced autovacuum worker did the job, so the warnings disappeared Every day the vacuum is executed on some tables; and on Sundays it's executed on all tables. But as the autovacuum is running since September, and it runs for a long time, the vacuum was blocked because autovacuum had been running on the same table. How should I procede in this case? hmm. single vacuum process runs for more than 3 months on a table with 10 rows? this is ... less than 128 rows/second, not good. I would rather terminate this old process, and start a VACUUM VERBOSE when the database is less loaded. How many INS/UPD/DEL you have on this table? About 15 millions rows inserted by day. PS. When you fix this, enable autovacuum, to avoid more problems... Regards, João Paulo -- JotaComm http://jotacomm.wordpress.com
[HACKERS] SSPI client authentication in non-Windows builds
Hello all,
this patch adds support for connecting to servers running on Windows
and requesting SSPI authentication. It does this by treating
AUTH_REQ_SSPI the same as AUTH_REQ_GSS if no native SSPI support is
available.
In addition to being generally useful, this is a workaround to a
problem with MIT KfW that I encountered back in September 2010 [1].
This change has been tested and works correctly on FreeBSD 8.1, using
the Kerberos and GSSAPI libraries from Heimdal 1.4. The server is
running PostgreSQL 9.0.2 on Windows 2008.
I originally fixed only the JDBC driver in this way [2], but then I
thought that I might try applying the same logic to libpq as well.
The preprocessor logic in that part of fe_sendauth() is quite
impenetrable; I hope I have not broken everything else.
[1]
http://archives.postgresql.org/message-id/i6cpc2%24m3h%241%40dough.gmane.org
[2]
http://archives.postgresql.org/message-id/i6org1%24mup%241%40dough.gmane.org
*** src/interfaces/libpq/fe-auth.c.orig Mon Jan 3 13:33:32 2011
--- src/interfaces/libpq/fe-auth.c Mon Jan 3 13:34:41 2011
***
*** 831,836
--- 831,839
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
case AUTH_REQ_GSS:
+ #if defined(ENABLE_GSS) !defined(ENABLE_SSPI)
+ case AUTH_REQ_SSPI:
+ #endif
{
int r;
***
*** 891,896
--- 894,902
#else
case AUTH_REQ_GSS:
case AUTH_REQ_GSS_CONT:
+ #ifndef ENABLE_SSPI
+ case AUTH_REQ_SSPI:
+ #endif
printfPQExpBuffer(conn-errorMessage,
libpq_gettext(GSSAPI authentication
not supported\n));
return STATUS_ERROR;
***
*** 913,923
}
pgunlock_thread();
break;
- #else
- case AUTH_REQ_SSPI:
- printfPQExpBuffer(conn-errorMessage,
- libpq_gettext(SSPI authentication
not supported\n));
- return STATUS_ERROR;
#endif
--- 919,924
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On 03.01.2011 11:37, Simon Riggs wrote: On Mon, 2011-01-03 at 01:53 -0500, Greg Smith wrote: In advance of the planned but not available yet ability to lock individual index key values, locking the whole table is the only possible implementation that can work correctly here I'm aware of. This was discussed here http://archives.postgresql.org/pgsql-hackers/2010-10/msg01903.php with suggested resolutions for this release here http://archives.postgresql.org/pgsql-hackers/2010-10/msg01907.php In summary, that means we can either 1. Lock the table for ShareRowExclusiveLock 2. throw a SERIALIZABLE error, if we come up against a row that cannot be neither MATCHED nor NON MATCHED. 3. Bounce the patch to 9.2, commit early and then work on a full concurrency solution before commit. The solution strawman is something like EvalPlanQual with a new snapshot for each re-checked row, emulating the pattern of snapshots/rechecks that would happen in a PL/pgSQL version of an UPSERT. Either way, we're saying that MERGE will not support concurrent operations safely, in this release. Given the continued lack of test cases for this patch, and the possible embarrassment over not doing concurrent actions, do we think (3) is the right road? This patch has never tried to implement concurrency-safe upsert. It implements the MERGE command as specified by the SQL standard, nothing more, nothing less. Let's not move the goalposts. Googling around, at least MS SQL Server's MERGE command is the same (http://weblogs.sqlteam.com/dang/archive/2009/01/31/UPSERT-Race-Condition-With-MERGE.aspx). There is nothing embarrassing about it, we just have to document it clearly. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, 2011-01-03 at 15:12 +0200, Heikki Linnakangas wrote: This patch has never tried to implement concurrency-safe upsert. It implements the MERGE command as specified by the SQL standard, nothing more, nothing less. Let's not move the goalposts. Googling around, at least MS SQL Server's MERGE command is the same (http://weblogs.sqlteam.com/dang/archive/2009/01/31/UPSERT-Race-Condition-With-MERGE.aspx). There is nothing embarrassing about it, we just have to document it clearly. That article says that SQLServer supplies a locking hint that completely removes the issue. Because they use locking, they are able to update in place, so there is no need for them to use snapshots. Our version won't allow a workaround yet, just for the record. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] V3: Idle in transaction cancellation
Is anybody working on this patch? -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] V3: Idle in transaction cancellation
Alvaro Herrera alvhe...@commandprompt.com wrote: Is anybody working on this patch? I'm not, but I sure hope someone is -- we could *really* use this in the SSI patch. With something providing the equivalent functionality to Andres's previous patch, and about one day's work in the SSI patch, SSI could guarantee that an immediate retry of a transaction rolled back with a serialization failure would not fail again on conflicts with the same transaction(s). This would be a very nice guarantee to be able to make. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 8:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2011-01-03 at 15:12 +0200, Heikki Linnakangas wrote: This patch has never tried to implement concurrency-safe upsert. It implements the MERGE command as specified by the SQL standard, nothing more, nothing less. Let's not move the goalposts. Googling around, at least MS SQL Server's MERGE command is the same (http://weblogs.sqlteam.com/dang/archive/2009/01/31/UPSERT-Race-Condition-With-MERGE.aspx). There is nothing embarrassing about it, we just have to document it clearly. That article says that SQLServer supplies a locking hint that completely removes the issue. Because they use locking, they are able to update in place, so there is no need for them to use snapshots. Our version won't allow a workaround yet, just for the record. Like Heikki, I'd rather have the feature without a workaround for the concurrency issues than no feature. But I have to admit that the discussion we've had thus far gives me very little confidence that this code is anywhere close to bug-free. So I think we're going to end up punting it to 9.2 not so much because it's not concurrency-safe as because it doesn't work. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] page compression
On Mon, Jan 3, 2011 at 4:02 AM, Jim Nasby j...@nasby.net wrote: FWIW, last time I looked at how Oracle handled compression, it would only compress existing data. As soon as you modified a row, it ended up un-compressed, presumably in a different page that was also un-compressed. IIUC, InnoDB basically compresses a block as small as it'll go, and then stores it in a regular size block. That leaves free space at the end, which can be used to cram additional tuples into the page. Eventually that free space is exhausted, at which point you try to recompress the whole page and see if that gives you room to cram in even more stuff. I thought that was a pretty clever approach. I wonder if it would be feasible to use a fork to store where a compressed page lives inside the heap... if we could do that I don't see any reason why indexes wouldn't work. The changes required to support that might not be too horrific either... At first blush, that sounds like a recipe for large amounts of undesirable random I/O. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] V3: Idle in transaction cancellation
On Monday, January 03, 2011 03:38:56 PM Alvaro Herrera wrote: Is anybody working on this patch? I am. Wont be finished in the next two days though (breakin last night...) Andres PS: Alvarro: commandprompt.com doesn't resolv anymore, so I can't send you the email directly... -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] and it's not a bunny rabbit, either
On Sun, Jan 2, 2011 at 4:45 PM, Peter Eisentraut pete...@gmx.net wrote: On lör, 2011-01-01 at 17:21 -0500, Robert Haas wrote: I don't see anything wrong with having 20 or 30 messages of variants of foo cannot be used on bar without placeholders. Well, that's OK with me. It seems a little grotty, but manageably so. Questions: 1. Should we try to include the name of the object? If so, how? Hmm. There is a bit of a difference in my mind between, say, constraints cannot be used on sequences constraint foo cannot be used on sequence bar the latter leaving open the question whether some other combination might work. Yeah, that's no good. Maybe there's a good way to clear things up with an errdetail(), though I'm having a hard time thinking how to phrase it. ERROR: sequence %s does not support the requested operation DETAIL: Constraints are not supported on sequences. ERROR: constraints are not supported on sequences DETAIL: %s is a sequence. ERROR: %s is a sequence DETAIL: Constraints and sequences are like water and oil, dude. 2. Can we have a variant with an SQL-command-fragment parameter? %s cannot be used on views where %s might be CLUSTER, DROP COLUMN, etc. That's OK; we do that in several other places. Cool. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Scanning pg_tablespace from walsender
I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). In order to do that, I need to scan pg_tablespace. However, scanning that from walsender gives me: FATAL: cannot read pg_class without having selected a database Which means I somehow have to get pg_tablespace into the cache without reading pg_class, I guess. Similar to how we do for pg_database for example. Can someone throw me a pointer or two on how to actually do that? :-) Am I correct in assuming I need to add it to RelationCacheInitializePhase2(), and to do that, need to figure out how to define a TableSpaceRelation_Rowtype_Id in the headers? Or is there an easier way I'm missing? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
Magnus Hagander mag...@hagander.net writes: I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). In order to do that, I need to scan pg_tablespace. Wait a minute. Isn't this problem about to metastasize into I need to read *every* global catalog from walsender? If not, why not? If so, I think we need another answer. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] and it's not a bunny rabbit, either
Excerpts from Robert Haas's message of lun ene 03 12:21:44 -0300 2011: Yeah, that's no good. Maybe there's a good way to clear things up with an errdetail(), though I'm having a hard time thinking how to phrase it. ERROR: sequence %s does not support the requested operation DETAIL: Constraints are not supported on sequences. This seems backwards to me: the detail is more general than the main message. ERROR: constraints are not supported on sequences DETAIL: %s is a sequence. This one seems good -- the detail message gives the detail. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 10:25 AM, Magnus Hagander mag...@hagander.net wrote: I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). Can you get that directly from the filesystem layout? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Visual Studio 2010/Windows SDK 7.1 support
On 01/03/2011 06:26 AM, Brar Piening wrote: No that was probably all. My patch - fixed as described above - should apply to your repository once you've run perltidy. I'll rebase the patch as soon as I return from work unless you tell me otherwise. Please, this time let's backpatch the fixes to all supported branches (from the buildfarm's POV that means back to 8.3 on for MSVC). I have just been wrestling with the fact that we didn't do that with the VS 2008 changes, and these changes are much, much more extensive. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 16:29, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). In order to do that, I need to scan pg_tablespace. Wait a minute. Isn't this problem about to metastasize into I need to read *every* global catalog from walsender? If not, why not? If so, I think we need another answer. Um, why would I need that? I need to be able to find all files, which means I need to find all tablespaces. I don't see how that would turn into every global catalog? (It already needs pg_authid and similar for the login, but that's shared with all others) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 16:34, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 3, 2011 at 10:25 AM, Magnus Hagander mag...@hagander.net wrote: I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). Can you get that directly from the filesystem layout? Hmm. I guess we could enumerate the pg_tblspc directory, and call readlink() on all the symlinks in there. Assuming all platforms can do readlink() (we'd obviously need a special windows implementation, but that's doable I guess). I just figured it'd be a lot cleaner to read it from our own catalogs... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 10:37 AM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jan 3, 2011 at 16:34, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 3, 2011 at 10:25 AM, Magnus Hagander mag...@hagander.net wrote: I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). Can you get that directly from the filesystem layout? Hmm. I guess we could enumerate the pg_tblspc directory, and call readlink() on all the symlinks in there. Assuming all platforms can do readlink() (we'd obviously need a special windows implementation, but that's doable I guess). I just figured it'd be a lot cleaner to read it from our own catalogs... I don't even see why you'd need readlink. Can't you just traverse the symlinks? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 16:40, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 3, 2011 at 10:37 AM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jan 3, 2011 at 16:34, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 3, 2011 at 10:25 AM, Magnus Hagander mag...@hagander.net wrote: I'm working on completing Heikki's patch for streaming base backups, and have run into a problem: In order to dump all tablespaces properly, I have to know where they are (d'uh). Can you get that directly from the filesystem layout? Hmm. I guess we could enumerate the pg_tblspc directory, and call readlink() on all the symlinks in there. Assuming all platforms can do readlink() (we'd obviously need a special windows implementation, but that's doable I guess). I just figured it'd be a lot cleaner to read it from our own catalogs... I don't even see why you'd need readlink. Can't you just traverse the symlinks? Well, they need to be put back in the same location on the other machine (slave in case of replication, tarball otherwise). If I just traverse the symlinks, they'll just appears as a subdirectory of pg_tblspc on the other machine, won't they? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
Excerpts from Magnus Hagander's message of lun ene 03 12:25:28 -0300 2011: Can someone throw me a pointer or two on how to actually do that? :-) Am I correct in assuming I need to add it to RelationCacheInitializePhase2(), and to do that, need to figure out how to define a TableSpaceRelation_Rowtype_Id in the headers? Or is there an easier way I'm missing? I think you just need to add BKI_ROWTYPE_OID and BKI_SCHEMA_MACRO to pg_tablespace.h (yes, and pick a suitable OID for the rowtype). Then figure out formrdesc. HTH -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Visual Studio 2010/Windows SDK 7.1 support
On Mon, 3 Jan 2011 10:44:19 +0100, Magnus Hagander mag...@hagander.net wrote: Yeah, it looks that way - it's missing the ordering of the contrib I'll run it once for that now, and then please rebase your patch on top of that - makes it easier to review it. The rebased patch can be grabbed from http://www.piening.info/VS2010v2.patch It's actually the same patch as before with the corrupted line-endings parsed out. git diff master VS2010 ../VS2010.patch reproduceabyl generates those on my (windows) system when I use git-external-diff configured as described in http://wiki.postgresql.org/wiki/Working_with_Git#Context_diffs_with_Git to produce context diffs. Unified diffs (git diff master VS2010 --no-ext-diff ../VS2010_unified.patch) only contain valid unix linfeeds Perhaps it has got something to do with core.autocrlf=true but I don't really care as long as I can get rid of it with one line of perl ;-) For the records I'm using $ git --version git version 1.7.3.1.msysgit.0 and the configuration of the postgresql repository is $ git config --list core.symlinks=false core.autocrlf=true color.diff=auto pack.packsizelimit=2g help.format=html http.sslcainfo=/bin/curl-ca-bundle.crt sendemail.smtpserver=/bin/msmtp.exe user.name=Brar Piening user.email=[myemail] core.repositoryformatversion=0 core.filemode=false core.bare=false core.logallrefupdates=true core.symlinks=false core.ignorecase=true core.hidedotfiles=dotGitOnly remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* remote.origin.url=git://git.postgresql.org/git/postgresql.git branch.master.remote=origin branch.master.merge=refs/heads/master diff.external=git-external-diff Best regards, Brar -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
Magnus Hagander mag...@hagander.net writes: On Mon, Jan 3, 2011 at 16:29, Tom Lane t...@sss.pgh.pa.us wrote: Wait a minute. Isn't this problem about to metastasize into I need to read *every* global catalog from walsender? If not, why not? If so, I think we need another answer. Um, why would I need that? I need to be able to find all files, which means I need to find all tablespaces. I don't see how that would turn into every global catalog? Well, if you just need to find all the files, scan the symlinks in $PGDATA/pg_tblspc/. Don't turn a filesystem problem into a catalog problem. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 10:42 AM, Magnus Hagander mag...@hagander.net wrote: Well, they need to be put back in the same location on the other machine (slave in case of replication, tarball otherwise). If I just traverse the symlinks, they'll just appears as a subdirectory of pg_tblspc on the other machine, won't they? Sure, I guess you'd need to read the links if you want it to work that way. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
* Robert Haas (robertmh...@gmail.com) wrote: Like Heikki, I'd rather have the feature without a workaround for the concurrency issues than no feature. I'm still trying to figure out the problem with having the table-level lock, unless we really think people will be doing concurrent MERGE's where they won't overlap..? I'm also a bit nervous about if the result of concurrent MERGE's would actually be correct if we're not taking a bigger lock than row-level (I assume we're taking row-level locks as it goes through..). In general, I also thought/expected to have some kind of UPSERT type capability with our initial MERGE support, even if it requires a big lock and won't operate concurrently, etc. But I have to admit that the discussion we've had thus far gives me very little confidence that this code is anywhere close to bug-free. So I think we're going to end up punting it to 9.2 not so much because it's not concurrency-safe as because it doesn't work. That's certainly a concern. :/ Stephen signature.asc Description: Digital signature
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On 03.01.2011 17:56, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: Like Heikki, I'd rather have the feature without a workaround for the concurrency issues than no feature. I'm still trying to figure out the problem with having the table-level lock, unless we really think people will be doing concurrent MERGE's where they won't overlap..? I'm also a bit nervous about if the result of concurrent MERGE's would actually be correct if we're not taking a bigger lock than row-level (I assume we're taking row-level locks as it goes through..). In general, I also thought/expected to have some kind of UPSERT type capability with our initial MERGE support, even if it requires a big lock and won't operate concurrently, etc. You can of course LOCK TABLE as a work-around, if that's what you want. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication as a separate permissions
On Mon, Jan 3, 2011 at 6:00 AM, Magnus Hagander mag...@hagander.net wrote: On Fri, Dec 31, 2010 at 15:38, Magnus Hagander mag...@hagander.net wrote: On Thu, Dec 30, 2010 at 15:54, Peter Eisentraut pete...@gmx.net wrote: On ons, 2010-12-29 at 11:09 +0100, Magnus Hagander wrote: I've applied this version (with some minor typo-fixes). This page is now somewhat invalidated: http://developer.postgresql.org/pgdocs/postgres/role-attributes.html Hmm. Somehow I missed that page completely when looking through the docs. I'll go update that. BTW, shouldn't CONNECTION LIMIT be listed on that page? and INHERIT? And VALID UNTIL? They're all role attributes, no? +1. First, it doesn't mention the replication privilege, and second it continues to claim that superuser status bypasses all permission checks. Well, that was *already* wrong. superuser doesn't bypass NOLOGIN. That doesn't mean it shouldn't be fixed, but that's independent of the replication role. I've committed a fix for this. I still think this is the wrong approach. Saying superuser doesn't bypass nologin is like saying that it doesn't bypass the need to enter the correct password to authenticate to it. You have to BE the superuser before you start bypassing permissions checks, and NOLOGIN and a possible password prompts control WHO CAN BECOME superuser. On the other hand, the REPLICATION privilege is denying you the right to perform an operation *even though you already are authenticated as a superuser*. I don't think there's anywhere else in the system where we allow a privilege to non-super-users but deny that same privilege to super-users, and I don't think we should be starting now. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 10:58 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 03.01.2011 17:56, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: Like Heikki, I'd rather have the feature without a workaround for the concurrency issues than no feature. I'm still trying to figure out the problem with having the table-level lock, unless we really think people will be doing concurrent MERGE's where they won't overlap..? I'm also a bit nervous about if the result of concurrent MERGE's would actually be correct if we're not taking a bigger lock than row-level (I assume we're taking row-level locks as it goes through..). In general, I also thought/expected to have some kind of UPSERT type capability with our initial MERGE support, even if it requires a big lock and won't operate concurrently, etc. You can of course LOCK TABLE as a work-around, if that's what you want. That work-around completely fails to solve the concurrency problem. Just because you have a lock on the table doesn't mean that there aren't already tuples in the table which are invisible to your snapshot (for example because the inserting transactions haven't committed yet). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On 01/03/2011 10:58 AM, Heikki Linnakangas wrote: In general, I also thought/expected to have some kind of UPSERT type capability with our initial MERGE support, even if it requires a big lock and won't operate concurrently, etc. You can of course LOCK TABLE as a work-around, if that's what you want. I think we need to state this in large red letters in the docs, if that's the requirement. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On 03.01.2011 18:02, Robert Haas wrote: On Mon, Jan 3, 2011 at 10:58 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 03.01.2011 17:56, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: Like Heikki, I'd rather have the feature without a workaround for the concurrency issues than no feature. I'm still trying to figure out the problem with having the table-level lock, unless we really think people will be doing concurrent MERGE's where they won't overlap..? I'm also a bit nervous about if the result of concurrent MERGE's would actually be correct if we're not taking a bigger lock than row-level (I assume we're taking row-level locks as it goes through..). In general, I also thought/expected to have some kind of UPSERT type capability with our initial MERGE support, even if it requires a big lock and won't operate concurrently, etc. You can of course LOCK TABLE as a work-around, if that's what you want. That work-around completely fails to solve the concurrency problem. Just because you have a lock on the table doesn't mean that there aren't already tuples in the table which are invisible to your snapshot (for example because the inserting transactions haven't committed yet). It works in read committed mode, because you acquire a new snapshot after the LOCK TABLE, and anyone else who modified the table must commit before the lock is granted. In serializable mode you get a serialization error. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
Robert Haas robertmh...@gmail.com writes: On Mon, Jan 3, 2011 at 10:42 AM, Magnus Hagander mag...@hagander.net wrote: Well, they need to be put back in the same location on the other machine (slave in case of replication, tarball otherwise). If I just traverse the symlinks, they'll just appears as a subdirectory of pg_tblspc on the other machine, won't they? Sure, I guess you'd need to read the links if you want it to work that way. Well, you're quietly ignoring a whole bunch of issues there, like whether the tablespaces *should* be in the identical locations on the other machine and how you'll deal with it if not. Eventually there's going to need to be some sort of tablespace mapping option for replication. But anyway, taking a base backup is fundamentally defined as scan the filesystem, paying no attention to catalogs and ISTM that it obviously should be the same way for tablespaces. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 17:14, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jan 3, 2011 at 10:42 AM, Magnus Hagander mag...@hagander.net wrote: Well, they need to be put back in the same location on the other machine (slave in case of replication, tarball otherwise). If I just traverse the symlinks, they'll just appears as a subdirectory of pg_tblspc on the other machine, won't they? Sure, I guess you'd need to read the links if you want it to work that way. Well, you're quietly ignoring a whole bunch of issues there, like whether the tablespaces *should* be in the identical locations on the other machine and how you'll deal with it if not. Eventually there's going to need to be some sort of tablespace mapping option for replication. But anyway, taking a base backup is fundamentally defined as scan the filesystem, paying no attention to catalogs and ISTM that it obviously should be the same way for tablespaces. I'm doing that now, and it works fine on my linux box. Haven't looked at a win32 implementation yet, but that can certainly be done. As for relocating tablespaces - yes, that would be very useful. But at this point, we *do* require them to be at the same place on the box you restore to (whether it's a backup or a slave). That said, it seems we don't actually ever *care* - from my quick grep of the source, it seems we never ever read the location from the catalog - we just store it there for reference. So in theory, we should be able to relocate a tablespace by just changing the symlink. But that would leave pg_tablespace and the filesystem out of sync, so we probably shouldn't do that. Either way, relocating tablespaces is for the future, let's start with being able to do streaming base backups at all. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
* Robert Haas (robertmh...@gmail.com) wrote: On Mon, Jan 3, 2011 at 10:42 AM, Magnus Hagander mag...@hagander.net wrote: Well, they need to be put back in the same location on the other machine (slave in case of replication, tarball otherwise). If I just traverse the symlinks, they'll just appears as a subdirectory of pg_tblspc on the other machine, won't they? Sure, I guess you'd need to read the links if you want it to work that way. Have to admit, I'm not entirely sure if this is really the behavior that makes the most sense. My gut reaction to this is that it'd make more sense for them to end up as directories rather than symlinks to places that might not exist on the slave, or that might not be writable by PG on the slave. I can see arguments either way though and so I really don't like the idea of it being forced one way or the other. Here's my 2c- make it optional on the slave side and then don't complain if the symlink already exists (even if it goes somewhere else). My thinking is that if someone needs to have the tablespaces reside somewhere else on the slave, they could say don't create the symlinks in the recovery config, and then manually create the symlinks where they need them to go. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Scanning pg_tablespace from walsender
On Mon, Jan 3, 2011 at 17:17, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Mon, Jan 3, 2011 at 10:42 AM, Magnus Hagander mag...@hagander.net wrote: Well, they need to be put back in the same location on the other machine (slave in case of replication, tarball otherwise). If I just traverse the symlinks, they'll just appears as a subdirectory of pg_tblspc on the other machine, won't they? Sure, I guess you'd need to read the links if you want it to work that way. Have to admit, I'm not entirely sure if this is really the behavior that makes the most sense. My gut reaction to this is that it'd make more sense for them to end up as directories rather than symlinks to places that might not exist on the slave, or that might not be writable by PG on the slave. I can see arguments either way though and so I really don't like the idea of it being forced one way or the other. Here's my 2c- make it optional on the slave side and then don't complain if the symlink already exists (even if it goes somewhere else). My thinking is that if someone needs to have the tablespaces reside somewhere else on the slave, they could say don't create the symlinks in the recovery config, and then manually create the symlinks where they need them to go. It's already a basic requirement that they need to be the same - replicating over a CREATE TABLESPACE is going to fail otherwise.. Being able to deal with that in a nice way would be good, of course, but we don't have that today. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication as a separate permissions
Robert Haas robertmh...@gmail.com writes: On the other hand, the REPLICATION privilege is denying you the right to perform an operation *even though you already are authenticated as a superuser*. I don't think there's anywhere else in the system where we allow a privilege to non-super-users but deny that same privilege to super-users, and I don't think we should be starting now. You might want to reflect on rolcatupdate a bit before asserting that there are no cases where privileges are ever denied to superusers. However, that precedent would suggest that the default should be to grant the replication bit to superusers. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 11:08 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 03.01.2011 18:02, Robert Haas wrote: On Mon, Jan 3, 2011 at 10:58 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 03.01.2011 17:56, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: Like Heikki, I'd rather have the feature without a workaround for the concurrency issues than no feature. I'm still trying to figure out the problem with having the table-level lock, unless we really think people will be doing concurrent MERGE's where they won't overlap..? I'm also a bit nervous about if the result of concurrent MERGE's would actually be correct if we're not taking a bigger lock than row-level (I assume we're taking row-level locks as it goes through..). In general, I also thought/expected to have some kind of UPSERT type capability with our initial MERGE support, even if it requires a big lock and won't operate concurrently, etc. You can of course LOCK TABLE as a work-around, if that's what you want. That work-around completely fails to solve the concurrency problem. Just because you have a lock on the table doesn't mean that there aren't already tuples in the table which are invisible to your snapshot (for example because the inserting transactions haven't committed yet). It works in read committed mode, because you acquire a new snapshot after the LOCK TABLE, and anyone else who modified the table must commit before the lock is granted. Oh, I forgot we hold the ROW EXCLUSIVE lock until commit. That might be OK, then. In serializable mode you get a serialization error. I don't think this part is true. You can certainly do this: CREATE TABLE test (a int); BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE; SELECT * FROM test; in another session, insert (1) into test LOCK TABLE test IN SHARE MODE; -- or just LOCK TABLE test, if you prefer SELECT * FROM test; -- still ain't there INSERT INTO test VALUES (1); I don't see what would make MERGE immune to this. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication as a separate permissions
On Mon, Jan 3, 2011 at 11:20 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On the other hand, the REPLICATION privilege is denying you the right to perform an operation *even though you already are authenticated as a superuser*. I don't think there's anywhere else in the system where we allow a privilege to non-super-users but deny that same privilege to super-users, and I don't think we should be starting now. You might want to reflect on rolcatupdate a bit before asserting that there are no cases where privileges are ever denied to superusers. Oh, huh. I wasn't aware of that. However, that precedent would suggest that the default should be to grant the replication bit to superusers. Yes it would. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Scanning pg_tablespace from walsender
* Magnus Hagander (mag...@hagander.net) wrote: It's already a basic requirement that they need to be the same - replicating over a CREATE TABLESPACE is going to fail otherwise.. Being able to deal with that in a nice way would be good, of course, but we don't have that today. If CREATE TABLESPACE replication also looked at the flag I was proposing, it could work. :) Of course, the admin wouldn't be able to move the directory/change the symlink to where they actually want it to be w/o taking the replication server down, but I'm not sure that's a show-stopper... It's certainly not the cleanest/nicest approach, don't get me wrong, but I really hate the idea of forcing people to have an identical filesystem layout on the slave that they have on the master. Stephen signature.asc Description: Digital signature
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, 2011-01-03 at 18:08 +0200, Heikki Linnakangas wrote: It works in read committed mode, because you acquire a new snapshot after the LOCK TABLE, and anyone else who modified the table must commit before the lock is granted. In serializable mode you get a serialization error. If its not safe without this LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE then we should do that automatically, and document that. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On 03.01.2011 18:29, Simon Riggs wrote: On Mon, 2011-01-03 at 18:08 +0200, Heikki Linnakangas wrote: It works in read committed mode, because you acquire a new snapshot after the LOCK TABLE, and anyone else who modified the table must commit before the lock is granted. In serializable mode you get a serialization error. If its not safe without this LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE then we should do that automatically, and document that. No we should not. The SQL standard doesn't require that, and it would unnecessarily restrict concurrent updates on unrelated rows in the table. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Jan3, 2011, at 17:21 , Robert Haas wrote: On Mon, Jan 3, 2011 at 11:08 AM, Heikki Linnakangas In serializable mode you get a serialization error. I don't think this part is true. You can certainly do this: CREATE TABLE test (a int); BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE; SELECT * FROM test; in another session, insert (1) into test LOCK TABLE test IN SHARE MODE; -- or just LOCK TABLE test, if you prefer SELECT * FROM test; -- still ain't there INSERT INTO test VALUES (1); In SERIALIZABLE mode, you need to take any table-level locks before obtaining a snapshot. There's even a warning about this in the docs somewhere IIRC... best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, 2011-01-03 at 18:35 +0200, Heikki Linnakangas wrote:
On 03.01.2011 18:29, Simon Riggs wrote:
On Mon, 2011-01-03 at 18:08 +0200, Heikki Linnakangas wrote:
It works in read committed mode, because you acquire a new snapshot
after the LOCK TABLE, and anyone else who modified the table must commit
before the lock is granted. In serializable mode you get a serialization
error.
If its not safe without this
LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE
then we should do that automatically, and document that.
No we should not. The SQL standard doesn't require that, and it would
unnecessarily restrict concurrent updates on unrelated rows in the table.
If we do that, then we definitely need a catch-all WHEN statement, so
that we can say
WHEN NOT MATCHED
INSERT
WHEN MATCHED
UPDATE
ELSE
{ INSERT into another table so we can try again in a minute
or RAISE error }
Otherwise we will silently drop rows. Throwing an error every time isn't
useful behaviour.
Of course, that then breaks the standard, just as all existing
implementations do.
--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On 03.01.2011 18:49, Simon Riggs wrote:
On Mon, 2011-01-03 at 18:35 +0200, Heikki Linnakangas wrote:
On 03.01.2011 18:29, Simon Riggs wrote:
On Mon, 2011-01-03 at 18:08 +0200, Heikki Linnakangas wrote:
It works in read committed mode, because you acquire a new snapshot
after the LOCK TABLE, and anyone else who modified the table must commit
before the lock is granted. In serializable mode you get a serialization
error.
If its not safe without this
LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE
then we should do that automatically, and document that.
No we should not. The SQL standard doesn't require that, and it would
unnecessarily restrict concurrent updates on unrelated rows in the table.
If we do that, then we definitely need a catch-all WHEN statement, so
that we can say
WHEN NOT MATCHED
INSERT
WHEN MATCHED
UPDATE
ELSE
{ INSERT into another table so we can try again in a minute
or RAISE error }
Otherwise we will silently drop rows. Throwing an error every time isn't
useful behaviour.
An ELSE clause would be nice, but it's not related to the question at
hand. Only some serialization anomalities result in a row that matches
neither WHEN MATCHED nor WHEN NOT MATCHED. Others result in a duplicate
key exception, for example.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 11:36 AM, Florian Pflug f...@phlo.org wrote: On Jan3, 2011, at 17:21 , Robert Haas wrote: On Mon, Jan 3, 2011 at 11:08 AM, Heikki Linnakangas In serializable mode you get a serialization error. I don't think this part is true. You can certainly do this: CREATE TABLE test (a int); BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE; SELECT * FROM test; in another session, insert (1) into test LOCK TABLE test IN SHARE MODE; -- or just LOCK TABLE test, if you prefer SELECT * FROM test; -- still ain't there INSERT INTO test VALUES (1); In SERIALIZABLE mode, you need to take any table-level locks before obtaining a snapshot. There's even a warning about this in the docs somewhere IIRC... That should be safe, if people do it that way. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] back branches vs. VS 2008
The following patch allows me to build the 8.3 and 8.4 branches using
Visual Studio 2008, once the build system is patched. But I don't really
know why. HEAD and 9.0 build fine without it. But those branches
branches fail with a complaint about IPPROTO_IPV6 being undefined.
The patch seems harmless enough. But I'd like to know why it's
happening. Does anyone have a clue?
cheers
andrew
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 06aece3..c1775ea 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -383,7 +383,7 @@ StreamServerPort(int family, char *hostName,
unsigned short portNumber,
}
#endif
-#ifdef IPV6_V6ONLY
+#if defined(IPV6_V6ONLY) defined(IPPROTO_IPV6)
if (addr-ai_family == AF_INET6)
{
if (setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY,
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On Mon, Jan 3, 2011 at 18:15, Andrew Dunstan and...@dunslane.net wrote: The following patch allows me to build the 8.3 and 8.4 branches using Visual Studio 2008, once the build system is patched. But I don't really know why. HEAD and 9.0 build fine without it. But those branches branches fail with a complaint about IPPROTO_IPV6 being undefined. The patch seems harmless enough. But I'd like to know why it's happening. Does anyone have a clue? Umm. Since when do we backpatch new features/platforms? I don't know exactly why that is happening, but it's a good indicator that backpatching it isn't necessarily safe - what else can be missed? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 12:01 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
If we do that, then we definitely need a catch-all WHEN statement, so
that we can say
WHEN NOT MATCHED
INSERT
WHEN MATCHED
UPDATE
ELSE
{ INSERT into another table so we can try again in a minute
or RAISE error }
Otherwise we will silently drop rows. Throwing an error every time isn't
useful behaviour.
An ELSE clause would be nice, but it's not related to the question at hand.
Only some serialization anomalities result in a row that matches neither
WHEN MATCHED nor WHEN NOT MATCHED. Others result in a duplicate key
exception, for example.
I must be missing something. A source row is either matched or not
matched. ELSE doesn't exist because the writers of the spec thought
there might be some third matched-invisible-row case, but rather
because you might have written WHEN [NOT MATCHED] AND some qual, and
you might fall through a list of all such clauses.
I think we're focusing on the wrong problem here. MERGE creates some
syntax to let you do with SQL something that people are currently
doing with application-side logic. I've written the application-side
logic to do this kind of thing more times than I care to remember, and
yeah there are concurrency issues, but:
- sometimes there's only one writer, so it doesn't matter
- sometimes there can technically be more than one writer, but the
usage is so low that nothing actually breaks
- sometimes you know that a given writer will only operate on rows
customer_id = some constant; so you only need to prevent two
concurrent writers *for the same customer*, not any two concurrent
writers
- and sometimes you need a full table lock.
The third case, in particular, is quite common in my experience, and a
very good reason not to impose a full table lock. Users hate having
to do explicit locking (especially users whose names rhyme with Bevin
Bittner) but they hate *unnecessary* full-table locks even more. A
problem that you can fix by adding a LOCK TABLE statement is annoying;
a problem that you can fix only be removing an implicit lock table
operation that the system performs under the hood is a lot worse. In
the fourth case above, which IME is quite common, you could EITHER
take a full-table lock, if that performs OK, OR you could arrange to
take an advisory lock that protects the records for the particular
customer whose data you want to update. If we always take a
full-table lock, then the user loses the option to do something else.
The point we ought to be focusing on is that the patch doesn't work.
Unless someone is prepared to put in some time to fix THAT, the rest
of this discussion is academic.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, 2011-01-03 at 19:01 +0200, Heikki Linnakangas wrote:
If we do that, then we definitely need a catch-all WHEN statement, so
that we can say
WHEN NOT MATCHED
INSERT
WHEN MATCHED
UPDATE
ELSE
{ INSERT into another table so we can try again in a minute
or RAISE error }
Otherwise we will silently drop rows. Throwing an error every time isn't
useful behaviour.
An ELSE clause would be nice, but it's not related to the question at
hand. Only some serialization anomalities result in a row that matches
neither WHEN MATCHED nor WHEN NOT MATCHED.
Concurrent UPDATEs, DELETEs, MERGE
Others result in a duplicate
key exception, for example.
Concurrent INSERTs, MERGE
So an ELSE clause is very relevant to handling anomalies in a useful
way.
--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On 01/03/2011 12:43 PM, Magnus Hagander wrote: On Mon, Jan 3, 2011 at 18:15, Andrew Dunstanand...@dunslane.net wrote: The following patch allows me to build the 8.3 and 8.4 branches using Visual Studio 2008, once the build system is patched. But I don't really know why. HEAD and 9.0 build fine without it. But those branches branches fail with a complaint about IPPROTO_IPV6 being undefined. The patch seems harmless enough. But I'd like to know why it's happening. Does anyone have a clue? Umm. Since when do we backpatch new features/platforms? I don't know exactly why that is happening, but it's a good indicator that backpatching it isn't necessarily safe - what else can be missed? This isn't a new platform, any more than a new version of gcc is a new platform. And I certainly don't understand your reference to new features. I'm not suggesting backporting one. I'm not going to maintain more than one buildfarm member doing MSVC, and and if we were to adopt your policy I would not be able to use a modern-ish version of the compiler/SDK and also build all the live branches. That seems quite unnecessary. If we'd backported the changes to support VS2008 when they were made a year or two ago, as we should have (the changes are pretty trivial), we'd probably have discovered this back then. I'm putting in this effort because Tom complained about lack of buildfarm coverage that occurred when I recently lost the machine my buildfarm members were running on, and I'm trying to get back the coverage they had. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
Robert Haas robertmh...@gmail.com wrote: Users hate having to do explicit locking (especially users whose names rhyme with Bevin Bittner) :-) Before you decide to taunt me again, I guess I should point out a few things here. Should SSI and MERGE both make it into 9.1, there's every reason to believe that running just about any DML, including MERGE, at REPEATABLE READ would generate the same behavior which running at REPEATABLE READ or SERIALIZABLE does now. If MERGE is susceptible to such anomalies as testing for the presence of a row and then trying to UPDATE it if found, only to update zero rows because it was concurrently deleted, SERIALIZABLE would prevent that with a serialization failure. I'd kind of expect that already with a write-write conflict, but if that isn't happening, the SSI patch should help. Well, help prevent anomalies -- if you want it to work out how to continue without rolling back it won't help at all. The fact that SSI introduces predicate locking may ultimately allow MERGE to do something more clever in terms of UPSERT logic, but I *REALLY* think it's too late in the release cycle to start looking at that. Predicate locking for SSI was done exactly as was most useful for SSI, on the basis (generally popular on this list) that trying to generalize something with only one use case is doomed to failure. Trying to bend it in an additional direction this late in the release would pretty much ensure that neither MERGE nor SSI could make it in. On the other hand, if we put SSI in with predicate locking more or less as it is, and put MERGE in with more primitive concurrency control, I fully expect that someone could figure out how to tease apart SSI and its predicate locking during the next release cycle, so that the predicate locking was more generally useful. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
On Mon, Jan 3, 2011 at 7:11 AM, Dmitry Koterov dmi...@koterov.ru wrote: To me, this is a wonderful feature, thanks! I think many people would be happy if this patch woud be included to the mainstream (and it is quite short and simple). About name ordering - I think that the problem exists for objects: 1. Stored functions. 2. Foreign keys/triggers (objects which has owning objects). It is wonderful that you store all functions with the same name to the same file. To order them within this file we may simply compare the first definition line lexicographically (or - first line which differs one function definition from another). Foreign key/triggers ordering problem is described by me at http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg114586.html The big problem is with triggers: many triggers may have the same name, but be bound to different tables. It would be great to include these triggers to table's definition or, at least, have separated files for each trigger+table pair. Hmm... this sounds awfully specific to your particular environment. My triggers never have the same names... and on the flip side, I don't see why someone couldn't want one function per file rather than all same-named functions in one file. I think the problem with this patch is that different people are likely to want slightly different things, and there may not be any single format that pleases everyone, and supporting too many variants will become confusing for users and hard for us to maintain. We're going to need to agree on something that won't be perfect for everyone, but will hopefully be a sufficient improvement for enough people to be worth doing. On the specific issue of overloaded functions, I have a feeling that the only feasible option is going to be to put them all in the same file. If you put them in different files, the names will either be very long (because they'll have to include the argument types) or fairly incomprehensible (if you did something like hash the argument types and append 8 hex digits to the function name) or not all that static (if you use OIDs; or if you number them sequentially, like foo1.sql, foo2.sql, foo3.sql, then foo3.sql might end up as foo2.sql on a system where there are only two variants of foo, making diff not work very well). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On Mon, Jan 3, 2011 at 19:08, Andrew Dunstan and...@dunslane.net wrote: On 01/03/2011 12:43 PM, Magnus Hagander wrote: On Mon, Jan 3, 2011 at 18:15, Andrew Dunstanand...@dunslane.net wrote: The following patch allows me to build the 8.3 and 8.4 branches using Visual Studio 2008, once the build system is patched. But I don't really know why. HEAD and 9.0 build fine without it. But those branches branches fail with a complaint about IPPROTO_IPV6 being undefined. The patch seems harmless enough. But I'd like to know why it's happening. Does anyone have a clue? Umm. Since when do we backpatch new features/platforms? I don't know exactly why that is happening, but it's a good indicator that backpatching it isn't necessarily safe - what else can be missed? This isn't a new platform, any more than a new version of gcc is a new platform. And I certainly don't understand your reference to new features. I'm not suggesting backporting one. It most definitely is a new platform in a *lot* more ways than a new version of gcc. It's the whole PlatformSDK. Why else did it require patches to the code? And it is a new feature *to the msvc build system*. I'm not going to maintain more than one buildfarm member doing MSVC, and and if we were to adopt your policy I would not be able to use a modern-ish version of the compiler/SDK and also build all the live branches. That seems quite unnecessary. If we'd backported the changes to support VS2008 when they were made a year or two ago, as we should have (the changes are pretty trivial), we'd probably have discovered this back then. Well, it's perfectly possible to have more tha none version of MSVC on the machine. And we're not going to be changing the version that's actually used for the official binary builds, so all you'll accomplish then is to have the buildfarm test something different form what we're shipping. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
2011/1/3 Robert Haas robertmh...@gmail.com: will become confusing for users and hard for us to maintain. We're going to need to agree on something that won't be perfect for everyone, but will hopefully be a sufficient improvement for enough people to be worth doing. Good point. I think we can at least agree the bare minimum is splitting per namespace, object type and name. On the specific issue of overloaded functions, I have a feeling that the only feasible option is going to be to put them all in the same file. If you put them in different files, the names will either be very long (because they'll have to include the argument types) or fairly incomprehensible (if you did something like hash the argument types and append 8 hex digits to the function name) or not all that static (if you use OIDs; or if you number them sequentially, like foo1.sql, foo2.sql, foo3.sql, then foo3.sql might end up as foo2.sql on a system where there are only two variants of foo, making diff not work very well). I agree. Even if the overloaded functions are not written in the same order, you will quickly and easily note function(s) of this particular name has been changed, which should narrow down your mind-mapping-change-grasping-exercise quite a lot. -- Best regards, Joel Jacobson Glue Finance -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
Robert Haas robertmh...@gmail.com writes: On the specific issue of overloaded functions, I have a feeling that the only feasible option is going to be to put them all in the same file. If you put them in different files, the names will either be very long (because they'll have to include the argument types) or fairly incomprehensible (if you did something like hash the argument types and append 8 hex digits to the function name) or not all that static (if you use OIDs; or if you number them sequentially, like foo1.sql, foo2.sql, foo3.sql, then foo3.sql might end up as foo2.sql on a system where there are only two variants of foo, making diff not work very well). If you put all the variants in the same file, diff is *still* not going to work very well. At least not unless you solve the problems that keep pg_dump from dumping objects in a consistent order ... and once you do that, you don't need this patch. I think the problem with this patch is that different people are likely to want slightly different things, and there may not be any single format that pleases everyone, and supporting too many variants will become confusing for users and hard for us to maintain. Yeah, that's exactly it. I can think of some possible uses for splitting up pg_dump output, but frankly to ease diff-ing is not one of them. For that problem, it's nothing but a crude kluge that only sort-of helps. If we're to get anywhere on this, we need a better-defined problem statement that everyone can agree is worth solving and is well solved with this particular approach. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 1:18 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: Users hate having to do explicit locking (especially users whose names rhyme with Bevin Bittner) :-) Before you decide to taunt me again, I guess I should point out a few things here. Sorry, that was intended as good-natured humor, not taunting. I think that the work you are doing on the serializability stuff is *exactly* the right fix for the concurrency issues associated with MERGE. Coming up with a fix that is specific to MERGE doesn't impress me much. I don't believe that hacking up MERGE will lead to anything other than an ugly mess; it's just a syntax wrapper around an operation that's fundamentally not too easy to make concurrent. SSI will handle it, though, along with, well, all the other cases that are worth worrying about. I don't have quite as much of an allergy to explicit locking as you do, but I'm quite clear that it isn't nearly as good as it just works. Should SSI and MERGE both make it into 9.1, [...] So far the thread on large patches has lead to a status report from most of the people working on large patches, and no volunteers to take the lead on reviewing/committing any of them. Although I think both of those patches are worthwhile, and although I intend to spend a very, very large amount of time doing CF work in the next 43 days, I don't foresee committing either of them, and I probably will not have time for a detailed review of either one, either. I feel pretty bad about that, but I just don't have any more bandwidth. :-( -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
2011/1/3 Tom Lane t...@sss.pgh.pa.us: pg_dump from dumping objects in a consistent order ... and once you do that, you don't need this patch. Yeah, that's exactly it. I can think of some possible uses for splitting up pg_dump output, but frankly to ease diff-ing is not one of them. For that problem, it's nothing but a crude kluge that only sort-of helps. If we're to get anywhere on this, we need a better-defined problem statement that everyone can agree is worth solving and is well solved with this particular approach. The problem statement is only partly diffing, I think the two major other problems with one-single-giant-schema-file is: 1. Automatically version controlling your production database schema using a file based approach is not possible. Splitting the schema would mean it's very simple to setup a cronjob which automatically commits the schema changes every night. You would be able to follow the changes by simply looking at the vcs log, instead of grepping the log files and trying to figure out what changed. 2. Splitting is a single option which reduces the need for any other imaginable exclude/include options, such as only export these functions or do not export views etc. Tables are of course the most common thing you want to include/exclude in a dump, but there are quite a lot of different object types, however, no single object type is important enough to motivate a new pg_dump option to allow a exclusion/inclusion option, but I think the sum of the need for such an option for all object types is greather than the extra complexity introduced by a patch consisting of only a few lines of code. -- Best regards, Joel Jacobson Glue Finance -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
Magnus Hagander mag...@hagander.net writes: On Mon, Jan 3, 2011 at 19:08, Andrew Dunstan and...@dunslane.net wrote: I'm not going to maintain more than one buildfarm member doing MSVC, and and if we were to adopt your policy I would not be able to use a modern-ish version of the compiler/SDK and also build all the live branches. Well, it's perfectly possible to have more tha none version of MSVC on the machine. And we're not going to be changing the version that's actually used for the official binary builds, so all you'll accomplish then is to have the buildfarm test something different form what we're shipping. Are you speaking for EDB on that? Do you even know what they're using to build the Windows installers? We've made backpatches before to support building/running older branches on newer platforms. We do it all the time in fact. (The latest instance was hacking the Linux wal_sync_method defaults. If you think this isn't a necessary activity, try building a 7.1 or so branch with a modern gcc.) It might be reasonable to argue that this particular patch is too invasive to be safe to back-patch, but I don't agree with the premise that it isn't a reasonable topic for a back-patch. I do have some concern about loss of buildfarm coverage for older VS versions, but if Andrew isn't going to cover those, perhaps someone else will step up for that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On Mon, Jan 3, 2011 at 19:50, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Mon, Jan 3, 2011 at 19:08, Andrew Dunstan and...@dunslane.net wrote: I'm not going to maintain more than one buildfarm member doing MSVC, and and if we were to adopt your policy I would not be able to use a modern-ish version of the compiler/SDK and also build all the live branches. Well, it's perfectly possible to have more tha none version of MSVC on the machine. And we're not going to be changing the version that's actually used for the official binary builds, so all you'll accomplish then is to have the buildfarm test something different form what we're shipping. Are you speaking for EDB on that? Do you even know what they're using to build the Windows installers? Yes and yes. Well, I'm not actually speaking for them, so I guess we'll need a +1 from Dave. But given that the principle has held for all the previous releases made, I assume it still does. The second yes is not pending a +1, I know exaclty what they use. We've made backpatches before to support building/running older branches on newer platforms. We do it all the time in fact. (The latest instance was hacking the Linux wal_sync_method defaults. If you think this isn't a necessary activity, try building a 7.1 or so branch with a modern gcc.) It might be reasonable to argue that this particular patch is too invasive to be safe to back-patch, but I don't agree with the premise that it isn't a reasonable topic for a back-patch. Fair enough. I don't care enough to object more :-) I do have some concern about loss of buildfarm coverage for older VS versions, but if Andrew isn't going to cover those, perhaps someone else will step up for that. I'd be worried if we don't have coverage for the versions that are actually used to build the installers. But those may be what Dave has covered on his animals already - Dave? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
Robert Haas robertmh...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Before you decide to taunt me again, I guess I should point out a few things here. Sorry, that was intended as good-natured humor, not taunting. Oh, I took it that way. I guess my attempt at humor through an oblique reference to a line from Monty Python and the Holy Grail fell flat. :-/ I guess I should have said before you taunt me a second time to make it more readily recognizable... I think that the work you are doing on the serializability stuff is *exactly* the right fix for the concurrency issues associated with MERGE. It's got a nice consistency with current behavior, with reads never blocking or being blocked, but I can see why people would want a MERGE which could dance around the concurrency problems and always succeed with UPSERT behavior. Various topics have come up which seem like they might benefit from predicate locking. I don't know how many would need locks which introduce blocking. I think it will actually be very easy to adapt the predicate locking for such things as transactional cache invalidation (which is what drew the interest of the MIT folks). I'm not sure how much work it would be to adapt it to use for the type of blocking locks which seem to be needed based on some of the MERGE discussions I've read. It think it will be non-trivial but possible. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
On Mon, Jan 3, 2011 at 1:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On the specific issue of overloaded functions, I have a feeling that the only feasible option is going to be to put them all in the same file. If you put them in different files, the names will either be very long (because they'll have to include the argument types) or fairly incomprehensible (if you did something like hash the argument types and append 8 hex digits to the function name) or not all that static (if you use OIDs; or if you number them sequentially, like foo1.sql, foo2.sql, foo3.sql, then foo3.sql might end up as foo2.sql on a system where there are only two variants of foo, making diff not work very well). If you put all the variants in the same file, diff is *still* not going to work very well. At least not unless you solve the problems that keep pg_dump from dumping objects in a consistent order ... and once you do that, you don't need this patch. That's not really true. It's a whole lot easier to look a diff of two 100-line files and then repeat that N times than to look at a single diff of two N*100 line files. I certainly spend enough of my patch-review doing git diff master some particular source file, and then if what's going on isn't clear you can look at just that file in more detail without worrying about every other source file in the system. And I have encountered this problem when comparing database schemas (and sometimes data) also. Yes, I've done that using diff. Yes, it did suck. Yes, I got it done before my boss fired me. I think the problem with this patch is that different people are likely to want slightly different things, and there may not be any single format that pleases everyone, and supporting too many variants will become confusing for users and hard for us to maintain. Yeah, that's exactly it. I can think of some possible uses for splitting up pg_dump output, but frankly to ease diff-ing is not one of them. For that problem, it's nothing but a crude kluge that only sort-of helps. If we're to get anywhere on this, we need a better-defined problem statement that everyone can agree is worth solving and is well solved with this particular approach. I have to admit I'm a bit unsold on the approach as well. It seems like you could write a short Perl script which would transform a text format dump into the proposed format pretty easily, and if you did that and published the script, then the next poor shmuck who had the same problem could either use the script as-is or hack it up to meet some slightly different set of requirements. Or maybe you'd be better off basing such a script on the custom or tar format instead, in order to avoid the problem of misidentifying a line beginning with --- as a comment when it's really part of a data item. Or maybe even writing a whole schema diff tool that would take two custom-format dumps as inputs. On the other hand, I can certainly think of times when even a pretty dumb implementation of this would have saved me some time. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Upgrading Extension, version numbers (was: [HACKERS] Extensions, patch v16)
On Dec 29, 2010, at 2:01 PM, Dimitri Fontaine wrote: # lo comment = 'managing Large Objects' version = '9.1devel' relocatable = true upgrade_from_null = 'null = lo.upgrade.sql' Here, any property that begins with 'upgrade_from_' is considered as an upgrade setup and the part after the prefix is not considered. The value is meant to have two parts separated by '=', first is either null or a regexp matched against currently installed version number, second part is the upgrade script name to use at ALTER EXTENSION ... UPGRADE. I thought we were going to try to avoid having entries for upgrades in the control file. We support 'null' version number to be able to upgrade from existing code which is not organized as an extension yet. The aim is to be able to: CREATE EMPTY EXTENSION lo; -- version is null here ALTER EXTENSION lo UPGRADE; And run a script containing lines that will look like this: alter domain @extsch...@.lo set extension lo; alter function @extsch...@.lo_oid(lo) set extension lo; alter function @extsch...@.lo_manage() set extension lo; Note that we always need to support the placeholder here, because of course following dependencies at this point isn't possible. I thought placeholders were going away, too. Did I lose track? Well, the way I see things, it's already too late and there's nothing we can easily do to prevent that. What I mean is that the user will typically upgrade the OS-level package first, then apply the upgrade on the database(s). $ apt-get install postgresql-9.1-prefix $ psql -U postgres -c 'alter extension prefix upgrade' somedb At the time you tell PostgreSQL about the new extension, the shared object file has been in place for some time already, and the upgrade SQL script has not been ran yet. That sounds dangerous. What I hope extension authors will do is document whether any upgrade requires a restart or will otherwise be responsible for instability in the server for backend started with the newer .so before the upgrade script has been run. So that users/DBA will know whether the upgrade calls for a maintenance window. But if a new connection comes in, the .so will be loaded into the new child, no? Very dangerous. I could see us trying to shoehorn such information into the control file too, but would ERRORing out on LOAD be any better than taking the compatibility chance? Knowing that the compatibility in most cases depends a lot on the actual call paths? The new .so should not be installed until the upgrade is been run. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid
On Mon, Jan 3, 2011 at 2:01 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Before you decide to taunt me again, I guess I should point out a few things here. Sorry, that was intended as good-natured humor, not taunting. Oh, I took it that way. I guess my attempt at humor through an oblique reference to a line from Monty Python and the Holy Grail fell flat. :-/ I guess I should have said before you taunt me a second time to make it more readily recognizable... Ah! I missed that. I have actually seen that movie, but it's been, well... OK, I feel old now. I think that the work you are doing on the serializability stuff is *exactly* the right fix for the concurrency issues associated with MERGE. It's got a nice consistency with current behavior, with reads never blocking or being blocked, but I can see why people would want a MERGE which could dance around the concurrency problems and always succeed with UPSERT behavior. I think the right thing to do about wanting UPSERT is to implement UPSERT, though personally I prefer the name REPLACE from my long-ago days as a MySQL user. It may be easier to solve a special case of the concurrency problem than to solve it in its full generality (and fixing MERGE is pretty close to solving it in its full generality). And even if it isn't, the MERGE syntax is insane if what you really want to do is insert or update ONE record. If all we have is MERGE, people will keep doing it with a PL/pgsql stored procedure or some crummy application logic just so that they don't have to spend several days trying to understand the syntax. Heck, I understand the syntax (or I think I do) and I still think it's more trouble than its worth. There is certainly a use case for an F-15 fighter jet but sometimes what you really want is a model rocket and a small bottle of silver paint. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extension upgrade, patch v0: debug help needed
On Jan 1, 2011, at 2:30 PM, Dimitri Fontaine wrote: To support that is quite simple in fact, as the following commands will do the trick: CREATE WRAPPER EXTENSION ...;-- don't run the script ALTER OBJECT ... SET EXTENSION ...; -- that's in the upgrade script ALTER EXTENSION ... UPGRADE; -- as usual I rather doubt that WRAPPER will be accepted as a reserved word in the grammar. Here's an example: dim=# \i ~/pgsql/exts/share/contrib/lo.sql CREATE DOMAIN CREATE FUNCTION CREATE FUNCTION dim=# create wrapper extension lo; CREATE EXTENSION What happened to your UPGRADE from NULL idea? The WRAPPER keyword meaning is that we're only creating the catalog entry, forcing version to NULL and not running the extension's script. This keyword looked like the best choice from existing ones in our grammar, please feel free to pick any other phrase here. I don't see why it's necessary at all. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
Robert Haas robertmh...@gmail.com writes: On Mon, Jan 3, 2011 at 1:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah, that's exactly it. I can think of some possible uses for splitting up pg_dump output, but frankly to ease diff-ing is not one of them. For that problem, it's nothing but a crude kluge that only sort-of helps. If we're to get anywhere on this, we need a better-defined problem statement that everyone can agree is worth solving and is well solved with this particular approach. I have to admit I'm a bit unsold on the approach as well. It seems like you could write a short Perl script which would transform a text format dump into the proposed format pretty easily, and if you did that and published the script, then the next poor shmuck who had the same problem could either use the script as-is or hack it up to meet some slightly different set of requirements. Or maybe you'd be better off basing such a script on the custom or tar format instead, in order to avoid the problem of misidentifying a line beginning with --- as a comment when it's really part of a data item. Or maybe even writing a whole schema diff tool that would take two custom-format dumps as inputs. On the other hand, I can certainly think of times when even a pretty dumb implementation of this would have saved me some time. The basic objection that I have to this patch is that it proposes to institutionalize a pretty dumb implementation. And, as you mentioned, once it's in there it'll be more or less set in stone because we aren't going to want to support umpteen variants. I like the idea of a postprocessing script a lot better --- it seems like it wouldn't get in the way of people making their own variants. And as you say it'd likely be pretty trivial to do. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On 01/03/2011 01:50 PM, Tom Lane wrote: It might be reasonable to argue that this particular patch is too invasive to be safe to back-patch, but I don't agree with the premise that it isn't a reasonable topic for a back-patch. The patch for the non-buildsystem code is one line. The rest is about 20 lines. I do have some concern about loss of buildfarm coverage for older VS versions, but if Andrew isn't going to cover those, perhaps someone else will step up for that. The machine involved already has three buildfarm critters. If I have to have three versions of VS installed (since we're now talking about installing a new one) that will grow to five, on one VM currently running on a small not very powerful Athlon X2 machine. It's already a pain in the neck to manage. Some time in the future I might have resources to run more, but right now I do not. Incidentally, I just went looking for VS2005/Express on microsoft.com. I don't know if they still make it available, but if they do it's fairly well hidden. I could find VS2008/Express and VS2010/Express very easily. ISTM that having support on the live branches for the compilers/SDKs that Microsoft apparently actually supports and distributes is not a bad thing to have. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On Mon, Jan 3, 2011 at 2:23 PM, Andrew Dunstan and...@dunslane.net wrote: Incidentally, I just went looking for VS2005/Express on microsoft.com. I don't know if they still make it available, but if they do it's fairly well hidden. I could find VS2008/Express and VS2010/Express very easily. ISTM that having support on the live branches for the compilers/SDKs that Microsoft apparently actually supports and distributes is not a bad thing to have. I agree. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Upgrading Extension, version numbers (was: [HACKERS] Extensions, patch v16)
David E. Wheeler da...@kineticode.com writes: On Dec 29, 2010, at 2:01 PM, Dimitri Fontaine wrote: At the time you tell PostgreSQL about the new extension, the shared object file has been in place for some time already, and the upgrade SQL script has not been ran yet. That sounds dangerous. It is, but I don't see any alternative. As Dimitri said, the .so will typically be installed by a packaging system, so we don't have any opportunity to run SQL code beforehand. In any case ... The new .so should not be installed until the upgrade is been run. ... that flat out doesn't work. If the upgrade script tries to add functions that didn't exist in the old .so, it'll fail. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Upgrading Extension, version numbers (was: [HACKERS] Extensions, patch v16)
On Jan 3, 2011, at 11:42 AM, Tom Lane wrote: It is, but I don't see any alternative. As Dimitri said, the .so will typically be installed by a packaging system, so we don't have any opportunity to run SQL code beforehand. In any case ... The new .so should not be installed until the upgrade is been run. ... that flat out doesn't work. If the upgrade script tries to add functions that didn't exist in the old .so, it'll fail. Right, what I'm saying is that `ALTER EXTENSION foo UPGRADE;` should install the .so, too, just before it runs the upgrade scripts. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
Robert Haas robertmh...@gmail.com writes: I have to admit I'm a bit unsold on the approach as well. It seems like you could write a short Perl script which would transform a text format dump into the proposed format pretty easily, and if you did that and published the script, then the next poor shmuck who had the same problem could either use the script as-is or hack it up to meet some slightly different set of requirements. Or maybe you'd be better That's actually what I first did a couple of months ago, https://github.com/gluefinance/parse_schema/blob/master/parse_schema.pl My major concern of parsing the schema file is I would never fully trust the output from the script, even if the regex is extremely paranoid and really strict, there is still a risk it contains a bug. If you cannot trust the output from the schema parse script, it's not safe to use it to do a partial restore of objects. Let's say you want to restore only a few functions from your backup schema. Without --split, you would need to restore them manually somehow. With the --split option, you could simply restore them from the indivudual files, at least functions where only the source code has been modified and not the arguments. 2011/1/3 Tom Lane t...@sss.pgh.pa.us: The basic objection that I have to this patch is that it proposes to institutionalize a pretty dumb implementation. And, as you mentioned, What's dumb about it? It's simple and it works. Please give me an idea of what a smart implementation would be, and I will see if I can rewrite the patch. once it's in there it'll be more or less set in stone because we aren't going to want to support umpteen variants. Yes, that's why it should be a bare minimum solution to the problem, which is a lot better than no solution at all. What other variants than the discussion on the path name structure can you think of? I like the idea of a postprocessing script a lot better --- it seems like it wouldn't get in the way of people making their own variants. And as you say it'd likely be pretty trivial to do. I'm quite sure such a postprocessing script has been written hundreds of times over the past years by different postgres users, not even submitting a question to the mailing list, since it's a quite managable task for anyone with some regex and scripting experience. Why not just provide a simple bare minimum solution to the problem and let the remaining x % of the users who have a very specific need write their own specialized script if they need to? It would save a lot of time for all future users who need to write their own script to solve the problem or google until they find my script or any other script solving the same problem. Multiple almost identical solutions to exactly the same problem is kind of time waste. -- Best regards, Joel Jacobson Glue Finance -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Upgrading Extension, version numbers
David E. Wheeler da...@kineticode.com writes: I thought we were going to try to avoid having entries for upgrades in the control file. Not what I have understood. http://archives.postgresql.org/pgsql-hackers/2010-12/msg01014.php http://archives.postgresql.org/pgsql-hackers/2010-12/msg01045.php AS there was no answer, the meaning for me is that it was ok to proceed. On this list people agreeing often remain silent. Note that we always need to support the placeholder here, because of course following dependencies at this point isn't possible. I thought placeholders were going away, too. Did I lose track? Oh, dear, yes :) See the documentation for the relocatable parameter. We know handle two kinds of extensions, some of them you can't offer better than placeholders to allow users to define the schema where they will land. Also, at upgrade time, I don't see any other way to solve the problem. Do you? http://pgsql.tapoueh.org/extensions/doc/html/extend-extension.html At the time you tell PostgreSQL about the new extension, the shared object file has been in place for some time already, and the upgrade SQL script has not been ran yet. That sounds dangerous. Been doing that countless times. Yet to see a case where the new .so is not compatible at all with the previous .sql and the author don't give you any warning about the situation. In theory that's possible, in practice we value upgrades high enough around here. Other than that, I don't have another idea to make it work reliably. I'm still reading, though. Meanwhile I've done what seems like a good compromise and to follow practical use cases. What I hope extension authors will do is document whether any upgrade requires a restart or will otherwise be responsible for instability in the server for backend started with the newer .so before the upgrade script has been run. So that users/DBA will know whether the upgrade calls for a maintenance window. But if a new connection comes in, the .so will be loaded into the new child, no? Very dangerous. Yeah. Before extension existed, it has always been working like that, our users already depend on such a behavior, nothing new here. I just don't see how extension could solve that is all I'm saying. The new .so should not be installed until the upgrade is been run. Nice statement. How do you make that happen? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extension upgrade, patch v0: debug help needed
David E. Wheeler da...@kineticode.com writes: I rather doubt that WRAPPER will be accepted as a reserved word in the grammar. It's already in the grammar, and I didn't change its level. dim=# create wrapper extension lo; CREATE EXTENSION What happened to your UPGRADE from NULL idea? You upgrade an installed extension, right? The WRAPPER keyword meaning is that we're only creating the catalog entry, forcing version to NULL and not running the extension's script. This keyword looked like the best choice from existing ones in our grammar, please feel free to pick any other phrase here. I don't see why it's necessary at all. Care to offer an alternative or go in any level of details about what you have in mind and how *exactly* you think it should work? I've not been offered psychic powers this Christmas, sorry… Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extension upgrade, patch v0: debug help needed
On Jan 3, 2011, at 11:49 AM, Dimitri Fontaine wrote: David E. Wheeler da...@kineticode.com writes: I rather doubt that WRAPPER will be accepted as a reserved word in the grammar. It's already in the grammar, and I didn't change its level. Okay. dim=# create wrapper extension lo; CREATE EXTENSION What happened to your UPGRADE from NULL idea? You upgrade an installed extension, right? What? The WRAPPER keyword meaning is that we're only creating the catalog entry, forcing version to NULL and not running the extension's script. This keyword looked like the best choice from existing ones in our grammar, please feel free to pick any other phrase here. I don't see why it's necessary at all. Care to offer an alternative or go in any level of details about what you have in mind and how *exactly* you think it should work? I've not been offered psychic powers this Christmas, sorry… That's what I understood your original UPGRADE from NULL being. Did I misread you? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Upgrading Extension, version numbers (was: [HACKERS] Extensions, patch v16)
David E. Wheeler da...@kineticode.com writes: On Jan 3, 2011, at 11:42 AM, Tom Lane wrote: ... that flat out doesn't work. If the upgrade script tries to add functions that didn't exist in the old .so, it'll fail. Right, what I'm saying is that `ALTER EXTENSION foo UPGRADE;` should install the .so, too, just before it runs the upgrade scripts. 1. Doesn't work if you're upgrading an installation that has more than one database using the extension. There's only one library directory. 2. Not possible from a permissions standpoint. Even if you think it'd be smart to have the postgres daemon privileged enough to overwrite its own executables, there is 0 chance of getting that past any distro. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Upgrading Extension, version numbers (was: [HACKERS] Extensions, patch v16)
On Jan 3, 2011, at 11:51 AM, Tom Lane wrote: 1. Doesn't work if you're upgrading an installation that has more than one database using the extension. There's only one library directory. 2. Not possible from a permissions standpoint. Even if you think it'd be smart to have the postgres daemon privileged enough to overwrite its own executables, there is 0 chance of getting that past any distro. Okay, got it. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extension upgrade, patch v0: debug help needed
David E. Wheeler da...@kineticode.com writes: That's what I understood your original UPGRADE from NULL being. Did I misread you? Are the docs about the feature, available handy in HTML so that you don't have to read them in SGML at my git repository, are they *that* bad? http://pgsql.tapoueh.org/extensions/doc/html/extend-extension.html Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
On Mon, Jan 3, 2011 at 2:46 PM, Joel Jacobson j...@gluefinance.com wrote: My major concern of parsing the schema file is I would never fully trust the output from the script, even if the regex is extremely paranoid and really strict, there is still a risk it contains a bug. That could possibly be resolved by using the custom or tar formats. If you cannot trust the output from the schema parse script, it's not safe to use it to do a partial restore of objects. Let's say you want to restore only a few functions from your backup schema. I don't think this is the right solution to that problem; or at least, it's not the solution I want personally. I want something like --function=glob (and yes I do want that same thing for every other object type, too, but PARTICULARLY for functions). This might be a next-best solution to that problem, but I'm still holding out for the home run. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extension upgrade, patch v0: debug help needed
On Jan 3, 2011, at 11:54 AM, Dimitri Fontaine wrote: That's what I understood your original UPGRADE from NULL being. Did I misread you? Are the docs about the feature, available handy in HTML so that you don't have to read them in SGML at my git repository, are they *that* bad? http://pgsql.tapoueh.org/extensions/doc/html/extend-extension.html I was responding to your email mentioning it, which did not reference said docs. David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Upgrading Extension, version numbers
On Jan 3, 2011, at 11:46 AM, Dimitri Fontaine wrote: Not what I have understood. http://archives.postgresql.org/pgsql-hackers/2010-12/msg01014.php http://archives.postgresql.org/pgsql-hackers/2010-12/msg01045.php AS there was no answer, the meaning for me is that it was ok to proceed. On this list people agreeing often remain silent. There were several of us who were not silent. http://archives.postgresql.org/pgsql-hackers/2010-12/msg00804.php http://archives.postgresql.org/pgsql-hackers/2010-12/msg00796.php The fact that the last two messages in the thread say something else does not mean that they represent the consensus. Note that we always need to support the placeholder here, because of course following dependencies at this point isn't possible. I thought placeholders were going away, too. Did I lose track? Oh, dear, yes :) See the documentation for the relocatable parameter. We know handle two kinds of extensions, some of them you can't offer better than placeholders to allow users to define the schema where they will land. Also, at upgrade time, I don't see any other way to solve the problem. Do you? http://pgsql.tapoueh.org/extensions/doc/html/extend-extension.html Right, I forgot about the relocatable parameter. I kind of expect that most extensions *would* be relocatable, though. Maybe it should be expected to be true if it's not present? Or perhaps require non-relocatable extensions to have a fixed_schema control key or something? Either will work, just trying to find the likely convention to avoid configuration in most cases. Maybe I'm wrong, though, and most extensions wouldn't be relocatable? Yeah. Before extension existed, it has always been working like that, our users already depend on such a behavior, nothing new here. I just don't see how extension could solve that is all I'm saying. Fair enough. The new .so should not be installed until the upgrade is been run. Nice statement. How do you make that happen? Nope. David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On Mon, Jan 3, 2011 at 6:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Mon, Jan 3, 2011 at 19:08, Andrew Dunstan and...@dunslane.net wrote: I'm not going to maintain more than one buildfarm member doing MSVC, and and if we were to adopt your policy I would not be able to use a modern-ish version of the compiler/SDK and also build all the live branches. Well, it's perfectly possible to have more tha none version of MSVC on the machine. And we're not going to be changing the version that's actually used for the official binary builds, so all you'll accomplish then is to have the buildfarm test something different form what we're shipping. Are you speaking for EDB on that? He's not speaking *for* us, but he's absolutely right. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
Dave Page dp...@pgadmin.org writes: On Mon, Jan 3, 2011 at 6:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: And we're not going to be changing the version that's actually used for the official binary builds, so all you'll accomplish then is to have the buildfarm test something different form what we're shipping. Are you speaking for EDB on that? He's not speaking *for* us, but he's absolutely right. OK, so what about the next question: is EDB running buildfarm members that will test the VS version(s) you are using? I don't think Andrew is under any obligation to do that for you. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --split patch
Robert Haas robertmh...@gmail.com writes: I have to admit I'm a bit unsold on the approach as well. It seems like you could write a short Perl script which would transform a text format dump into the proposed format pretty easily, and if you did that and published the script, then the next poor shmuck who had the same problem could either use the script as-is or hack it up to meet some slightly different set of requirements. Or maybe you'd be better off basing such a script on the custom or tar format instead, in order to avoid the problem of misidentifying a line beginning with --- as a comment when it's really part of a data item. Or maybe even writing a whole schema diff tool that would take two custom-format dumps as inputs. On the other hand, I can certainly think of times when even a pretty dumb implementation of this would have saved me some time. You mean like those: https://labs.omniti.com/labs/pgtreats/wiki/getddl https://github.com/dimitri/getddl In this case hacking the perl version was impossible for me to hack on so my version is in python, but still is somewhat of a fork. I even have an intermediate shell version somewhere that a colleague wrote, but I much prefer maintaining python code. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extension upgrade, patch v0: debug help needed
David E. Wheeler da...@kineticode.com writes: http://pgsql.tapoueh.org/extensions/doc/html/extend-extension.html I was responding to your email mentioning it, which did not reference said docs. Fair enough, I'm still interested in you telling me if I get to rewrite them all or if it's explaining the thing well enough :) Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: starting to review the Extend NOT NULL representation to pg_constraint patch
Bernd, are you still working on this? On fre, 2010-10-15 at 13:36 -0400, Tom Lane wrote: Bernd Helmle maili...@oopsware.de writes: Here is an updated version of the patch. It fixes the following issues Andrew discovered during his review cycle: I looked through this a bit. It's not ready to commit unfortunately. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Upgrading Extension, version numbers
David E. Wheeler da...@kineticode.com writes: The fact that the last two messages in the thread say something else does not mean that they represent the consensus. Yeah, but as I'm the one writing the code, I gave myself more than one vote. And did consider the alternatives but didn't like them so much. Right, I forgot about the relocatable parameter. I kind of expect that most extensions *would* be relocatable, though. Maybe it should be expected to be true if it's not present? Or perhaps require non-relocatable extensions to have a fixed_schema control key or something? Either will work, just trying to find the likely convention to avoid configuration in most cases. Maybe I'm wrong, though, and most extensions wouldn't be relocatable? Most are, but it's not for granted. See adminpack. Or earthdistance that I had to make not-relocatable for lack of dependency support, as it depends on cube and ALTER EXTENSION earthdistance SET SCHEMA foo would have relocated cube too. We said dependency can wait until v2. I don't see the benefit of having the 'relocatable' property optional in the control file, but I see a huge drawback. Requiring it will force extension authors to at least have a glance at the docs to understand how to set it. It's important not to overlook it. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] back branches vs. VS 2008
On 01/03/2011 03:04 PM, Tom Lane wrote: Dave Pagedp...@pgadmin.org writes: On Mon, Jan 3, 2011 at 6:50 PM, Tom Lanet...@sss.pgh.pa.us wrote: Magnus Hagandermag...@hagander.net writes: And we're not going to be changing the version that's actually used for the official binary builds, so all you'll accomplish then is to have the buildfarm test something different form what we're shipping. Are you speaking for EDB on that? He's not speaking *for* us, but he's absolutely right. OK, so what about the next question: is EDB running buildfarm members that will test the VS version(s) you are using? I don't think Andrew is under any obligation to do that for you. They have baiji (Vista/MSVC 2005 Pro) and mastodon (WS2K3R2/MSVC 2005 Express). EDB have been pretty good about buildfarm support. But more importantly, the buildfarm is about more than just official build platform support. Suppose that you're running 8.4 in your enterprise, and you want to run a patched Postgres, or one with an extra module you wrote. You want to be able to build it with current tools, no matter what the official builds are made with, and you don't want to be forced to upgrade before you're ready. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
