Re: [HACKERS] Snapshot synchronization, again...
Jim Nasby writes: > Dumb question: Is this something that could be solved by having the > postmaster track this information in it's local memory and make it available > via a variable-sized IPC mechanism, such as a port or socket? That would > eliminate the need to clean things up after a crash; I'm not sure if there > would be other benefits. Involving the postmaster in this is entirely *not* reasonable. The postmaster cannot do anything IPC-wise that the stats collector couldn't do, and every additional function we load onto the postmaster is another potential source of unrecoverable database-wide failures. The PM is reliable only because it doesn't do much. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Snapshot synchronization, again...
On Feb 28, 2011, at 11:59 AM, Tom Lane wrote: > Robert Haas writes: >> On Sun, Feb 27, 2011 at 8:33 PM, Joachim Wieland wrote: >>> Remember that it's not only about saving shared memory, it's also >>> about making sure that the snapshot reflects a state of the database >>> that has actually existed at some point in the past. > >> But you can do all of this with files too, can't you? Just remove or >> truncate the file when the snapshot is no longer valid. > > Yeah. I think adopting a solution similar to 2PC state files is a very > reasonable way to go here. This isn't likely to be a high-usage or > performance-critical feature, so it's not essential to keep the > information in shared memory for performance reasons. Dumb question: Is this something that could be solved by having the postmaster track this information in it's local memory and make it available via a variable-sized IPC mechanism, such as a port or socket? That would eliminate the need to clean things up after a crash; I'm not sure if there would be other benefits. -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On 03/01/2011 05:19 PM, Jan Urbański wrote: On 01/03/11 22:07, Andrew Dunstan wrote: On 03/01/2011 03:53 PM, Jan Urbański wrote: On 01/03/11 21:35, Tom Lane wrote: Josh Berkus writes: I'm ok with closing things as of the end of the 15 days, say Thursday or Friday. It might be a good idea to make a list of what we have left to do before we can wrap an alpha. Here are some things on my list. Not all of them are necessarily release blockers, but we need to discuss which ones are: * Regression test failures from recent plpython patches. These are affecting enough machines to make them "must fix before alpha", IMO. There are some variations in error message wording, which are not too terrible but also not exactly hard to fix. The python assert failure that some Fedora machines are reporting is considerably more disturbing. I agree. I'm looking into the crash, no luck so long. Is there anything you need that would help you? Could you try this patch and see if it fixes the failures? I'm at a loss as to why this happens, but judging from the traceback the spiexceptions module is getting unreffed somewhere and when garbage collection kicks it it barfs on an object with refcount 0. So I'm forcing an incref of the module to confirm that. I tried various tricks on 32 bit Debian, with Python 2.6, 2.7, Python compiled from Fedora's SRPM and I never saw anything wrong. Will keep on trying, but tommorrow evening, time to sleep :( Thanks. That seems to have fixed it, so I have applied the patch. Would you like to supply some comments to got with it? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
On Tue, Mar 1, 2011 at 3:11 PM, Heikki Linnakangas wrote: > Heck, you can just put an Assert(!ImmediateInterruptOK) there, although it > will fire in the authentication phase because of the issue with > ClientAuthentication. You can set debug_assertions=off in postgresql.conf > and enable it again with SET after logging in to get around that. That doesn't sound like a bad idea. We could Assert(!ImmediateInterruptOK || ImmediateInterruptEnabledInQuestionablePlace) at the beginning of a bunch of basic low-level routines like AllocSetAlloc. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
> > No, the question is why is the ImmediateInterruptOK flag set. Whether > the interrupt actually happens isn't that relevant. You could try > setting a watchpoint on the flag variable. > >> But adding hold/resume interrrupts in mcxt.c (not aset.c, since we >> want to be agnostic to the underlying layer) should be good enough to >> handle this non-re-entrant issue, no? > > We are not doing that, because that would be only a band-aid patch for > approximately 0.1% of the problems that can arise from running random > code with ImmediateInterruptOK set. We need to find out what's leaving > that set and fix it. > Got it. Thanks Tom and Heikki. Will investigate this further. @Andres Apologies all. I should have mentioned upfront that this is occurring on 8.3.13, with some custom modifications done, but probably not in this area.. Regards, Nikhils -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, 2011-03-01 at 12:42 -0800, Josh Berkus wrote: > > That response is just dodging the hard question, so whatever. Tom's > > cleanup is not going to break things, or at least it's going to fix > > more than it breaks on net. Sync rep, on the other hand, is going to > > do the opposite, probably by a large margin. > > Well, we need to at least give Simon and Fujii a chance to respond > during their respective daytime hours. Friday is a reasonable deadline. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] OSSP gone missing? Fate of UUID?
On Mar 1, 2011, at 17:15, Hiroshi Saito wrote: > Ooops, > It is some trobles now. > please see Ralf-san's comment. Thanks, Hiroshi! Michael Glaesemann grzm seespotcode net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On 01/03/11 22:07, Andrew Dunstan wrote:
>
>
> On 03/01/2011 03:53 PM, Jan Urbański wrote:
>> On 01/03/11 21:35, Tom Lane wrote:
>>> Josh Berkus writes:
I'm ok with closing things as of the end of the 15 days, say
Thursday or
Friday.
>>> It might be a good idea to make a list of what we have left to do before
>>> we can wrap an alpha. Here are some things on my list. Not all of them
>>> are necessarily release blockers, but we need to discuss which ones are:
>>>
>>> * Regression test failures from recent plpython patches. These are
>>> affecting enough machines to make them "must fix before alpha", IMO.
>>> There are some variations in error message wording, which are not too
>>> terrible but also not exactly hard to fix. The python assert failure
>>> that some Fedora machines are reporting is considerably more disturbing.
>
> I agree.
>
>> I'm looking into the crash, no luck so long.
>>
>>
>
> Is there anything you need that would help you?
Could you try this patch and see if it fixes the failures?
I'm at a loss as to why this happens, but judging from the traceback the
spiexceptions module is getting unreffed somewhere and when garbage
collection kicks it it barfs on an object with refcount 0. So I'm
forcing an incref of the module to confirm that.
I tried various tricks on 32 bit Debian, with Python 2.6, 2.7, Python
compiled from Fedora's SRPM and I never saw anything wrong. Will keep on
trying, but tommorrow evening, time to sleep :(
Cheers,
Jan
diff --git a/src/pl/plpython/plpython.c b/src/pl/plpython/plpython.c
index 4cc0708..a2ebd22 100644
--- a/src/pl/plpython/plpython.c
+++ b/src/pl/plpython/plpython.c
@@ -3936,6 +3936,7 @@ PLy_add_exceptions(PyObject *plpy)
#endif
if (PyModule_AddObject(plpy, "spiexceptions", excmod) < 0)
PLy_elog(ERROR, "failed to add the spiexceptions module");
+ Py_INCREF(excmod);
PLy_exc_error = PyErr_NewException("plpy.Error", NULL, NULL);
PLy_exc_fatal = PyErr_NewException("plpy.Fatal", NULL, NULL);
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 4:58 PM, Robert Haas wrote: > > So where is the new patch, and the test reports from all the people > who found problems in the last version? > Simon do this in his public repo here: git://github.com/simon2ndQuadrant/postgres.git It has been just one day from this so i expect he will post a patch soon, meanwhile i'm following his repo and testing and seems i'm not the only one because Yeb Havinga has added two columns in pg_stat_replication to know which standby is the synch standby and which ones are potential synch standbys -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] toast tables on system catalogs
Excerpts from Tom Lane's message of mar mar 01 19:03:35 -0300 2011: > Alvaro Herrera writes: > > Strangely, we made pg_database have a toast table, and the only reason > > for this is datacl. Should we create toast tables for the remaining > > catalogs? > > As I commented on your blog, this is nonsense. pg_database has a TOAST > table becase we thought it might need one for datconfig[]. Now that > that's gone, it'd be consistent to remove the toast table, but it didn't > occur to us to do that. Yeah, it occured to me to troll the git logs just after sending the email and I promptly noticed the bug in my conclusion -- there was no datacl back then; and pg_db_role_settings is very new. > aclitem entries wide enough to need toasting are going to suck for all > sorts of reasons (IIRC there are some O(N^2) algorithms in there, not > to mention the cost of pulling in entries from a toast table on every > access) so I am not excited about encouraging people to use them. I agree on not supporting large numbers of privileges, though the error message leaves a bit to be desired. Should we remove the toast table declaration for pg_database? (BTW with the relmapper mechanism, do we still need to declare the toast table OIDs?) -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] OSSP gone missing? Fate of UUID?
Ooops, It is some trobles now. please see Ralf-san's comment. >On 01.03.11 15:49, Hiroshi Saito wrote: >> Hi Ralf-san. Thomas-san. >> >> Um... cannot be accessed here. >> http://www.ossp.org/pkg/lib/uuid/ >> >> Are some you in a trouble? > >The server went down. Thanks for the hint. >Now fixed. > > Ralf S. Engelschall > r...@engelschall.com > www.engelschall.com Best regards, Hiroshi Saito >On Mon, Feb 28, 2011 at 4:08 PM, Michael Glaesemann > wrote: >> A couple of weeks ago when installing uuid-ossp on a new server, I noticed >> that the ossp site is gone. I haven't found anything on the web to indicate >> what happened. >> >> Anyone know? > >Maybe it's time to provide our own UUID generator. > >-- >Robert Haas >EnterpriseDB: http://www.enterprisedb.com >The Enterprise PostgreSQL Company > >-- >Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) >To make changes to your subscription: >http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Porting PostgreSQL to DragonFly BSD
On Tuesday 1. of March 2011 22:44:16 Peter Eisentraut wrote: > On tis, 2011-03-01 at 22:22 +0100, Rumko wrote: > > Well, wouldn't consider it ugly, but the patch (attached and available at > > http://www.rumko.net/0001-DragonFly-BSD-support-linked.patch ) is a lot > > shorter. > > > > Uses freebsd's template and defines the linker in Makefile.shlib. > > The piece in Makefile.shlib you add is dead code because PORTNAME will > never be "dragonfly" (it would be "freebsd"). Ah, good to know. > I see there is a > difference between the existing freebsd code and what you propose to add > in that freebsd doesn't use shared object minor versions. Is that also > or not the case on DragonFly BSD? Due to pkgsrc being the default on NetBSD and DragonFly BSD, it should create the libs in the same way ... maybe instead of using PORTNAME, we could use host_os to differentiate? -- Regards, Rumko signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] toast tables on system catalogs
Alvaro Herrera writes: > Strangely, we made pg_database have a toast table, and the only reason > for this is datacl. Should we create toast tables for the remaining > catalogs? As I commented on your blog, this is nonsense. pg_database has a TOAST table becase we thought it might need one for datconfig[]. Now that that's gone, it'd be consistent to remove the toast table, but it didn't occur to us to do that. aclitem entries wide enough to need toasting are going to suck for all sorts of reasons (IIRC there are some O(N^2) algorithms in there, not to mention the cost of pulling in entries from a toast table on every access) so I am not excited about encouraging people to use them. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ADMIN] PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On Tue, Mar 01, 2011 at 12:00:54AM +0200, Heikki Linnakangas wrote: > On 28.02.2011 23:28, daveg wrote: > >On Wed, Jan 12, 2011 at 10:46:14AM +0200, Heikki Linnakangas wrote: > >>We'll likely need to go back and forth a few times with various > >>debugging patches until we get to the heart of this.. > > > >Anything new on this? I'm seeing at on one of my clients production boxes. > > I haven't heard anything from the OP since. > > >Also, what is the significance, ie what is the risk or damage potential if > >this flag is set incorrectly? > > Sequential scans will honor the flag, so you might see some dead rows > incorrectly returned by a sequential scan. That's the only "damage", but > an incorrectly set flag could be a sign of something more sinister, like > corrupt tuple headers. The flag should never be set incorrectly, so if > you see that message you have hit a bug in PostgreSQL, or you have bad > hardware. > > This flag is quite new, so a bug in PostgreSQL is quite possible. If you > still have a backup that contains those incorrectly set flags, I'd like > to see what the page looks like. I ran vacuums on all the affected tables last night. I plan to take a downtime to clear the buffer cache and then to run vacuums on all the dbs in the cluster. Most but not all the tables involved are catalogs. However, I could probably pick up your old patch sometime next week if it recurrs and send you page images. -dg -- David Gould da...@sonic.net 510 536 1443510 282 0869 If simplicity worked, the world would be overrun with insects. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ADMIN] PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On Mon, Feb 28, 2011 at 07:43:39PM -0600, David Christensen wrote: > > On Feb 28, 2011, at 3:28 PM, daveg wrote: > > > Anything new on this? I'm seeing at on one of my clients production boxes. > > Also, what is the significance, ie what is the risk or damage potential if > > this flag is set incorrectly? > > > Was this cluster upgraded to 8.4.4 from 8.4.0? It sounds to me like a known > bug in 8.4.0 which was fixed by this commit: > > commit 7fc7a7c4d082bfbd579f49e92b046dd51f1faf5f > Author: Tom Lane > Date: Mon Aug 24 02:18:32 2009 + > > Fix a violation of WAL coding rules in the recent patch to include an > "all tuples visible" flag in heap page headers. The flag update *must* > be applied before calling XLogInsert, but heap_update and the tuple > moving routines in VACUUM FULL were ignoring this rule. A crash and > replay could therefore leave the flag incorrectly set, causing rows > to appear visible in seqscans when they should not be. This might explain > recent reports of data corruption from Jeff Ross and others. > > In passing, do a bit of editorialization on comments in visibilitymap.c. > > oy:postgresql machack$ git describe --tag > 7fc7a7c4d082bfbd579f49e92b046dd51f1faf5f > REL8_4_0-190-g7fc7a7c > > If the flag got twiddled while running as 8.4.0, the incorrect PD_ALL_VISIBLE > flag would (obviously) not be fixed by the upgrade to 8.4.4. (Is this a > separate issue?) This cluster was installed with 8.4.4. So it is still an existing problem. Also, to my recollection, this cluster has never crashed. -dg -- David Gould da...@sonic.net 510 536 1443510 282 0869 If simplicity worked, the world would be overrun with insects. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 4:55 PM, Jaime Casanova wrote: > On Tue, Mar 1, 2011 at 3:40 PM, Robert Haas wrote: >> Sync rep, on the other hand, is going to >> do the opposite, probably by a large margin. >> > > Are you sure of that? the code is very centralized and the bugs we > found last week weren't that difficult to address, the prove for that > is that Simon fixed them in one day... So where is the new patch, and the test reports from all the people who found problems in the last version? > And for the open items, most of them are definitions about the behaviour... Like what? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 3:40 PM, Robert Haas wrote: > Sync rep, on the other hand, is going to > do the opposite, probably by a large margin. > Are you sure of that? the code is very centralized and the bugs we found last week weren't that difficult to address, the prove for that is that Simon fixed them in one day... And for the open items, most of them are definitions about the behaviour... -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] toast tables on system catalogs
Hi, Someone on IRC recently noticed that you can't grant USAGE privileges on a table to a large number of roles. (My experiment says 2466 works, 2467 doesn't). Of course, this is wrong and all that. I wrote a blog article about this: http://www.commandprompt.com/blogs/alvaro_herrera/2011/03/grant_schema_usage_to_2500_users_no_can_do/ The reason for this is that pg_namespace doesn't have a toast table; and neither do other 10 catalogs that have "acl" columns: pg_attribute pg_default_acl pg_largeobject_metadata pg_pltemplate pg_tablespace pg_class pg_foreign_data_wrapper pg_namespace pg_foreign_server pg_user_mapping pg_language select relname, reltoastrelid from pg_class where oid in ( select attrelid from pg_attribute where (attname like '%acl' and atttypid = 'aclitem[]'::regtype) or (attname like '%options' and atttypid = 'text[]'::regtype)) and relkind = 'r'; Strangely, we made pg_database have a toast table, and the only reason for this is datacl. Should we create toast tables for the remaining catalogs? -- Álvaro Herrera -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ADMIN] PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
Hi On Tue, Mar 1, 2011 at 11:00 AM, Heikki Linnakangas < heikki.linnakan...@enterprisedb.com> wrote: > On 28.02.2011 23:28, daveg wrote: > >> On Wed, Jan 12, 2011 at 10:46:14AM +0200, Heikki Linnakangas wrote: >> >>> We'll likely need to go back and forth a few times with various >>> debugging patches until we get to the heart of this.. >>> >> >> Anything new on this? I'm seeing at on one of my clients production boxes. >> > > I haven't heard anything from the OP since. > > > Also, what is the significance, ie what is the risk or damage potential if >> this flag is set incorrectly? >> > > Sequential scans will honor the flag, so you might see some dead rows > incorrectly returned by a sequential scan. That's the only "damage", but an > incorrectly set flag could be a sign of something more sinister, like > corrupt tuple headers. The flag should never be set incorrectly, so if you > see that message you have hit a bug in PostgreSQL, or you have bad hardware. > > This flag is quite new, so a bug in PostgreSQL is quite possible. If you > still have a backup that contains those incorrectly set flags, I'd like to > see what the page looks like. > > > -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com > Sorry, I was a bit busy lately. This effect on my production servers seems have floating nature (some weeks it happen each day many time and some weeks - not at all... phase of the moon may be). Last 2 weeks I don't see that error in database logs so I can't add anything more. What make situation even harder to debug it is I can't compile and install patched version of database engine (because it is production server and I physically don't have access to changing software there). So I trying to reproduce error on single connect (so I can attach GDB and look inside) but without luck during last week. -- Maxim Boguk Senior Postgresql DBA. Skype: maxim.boguk Jabber: maxim.bo...@gmail.com LinkedIn profile: http://nz.linkedin.com/in/maximboguk If they can send one man to the moon... why can't they send them all? МойКруг: http://mboguk.moikrug.ru/ Сила солому ломит, но не все в нашей жизни - солома, да и сила далеко не все.
Re: [HACKERS] Porting PostgreSQL to DragonFly BSD
On tis, 2011-03-01 at 22:22 +0100, Rumko wrote: > Well, wouldn't consider it ugly, but the patch (attached and available at > http://www.rumko.net/0001-DragonFly-BSD-support-linked.patch ) is a lot > shorter. > > Uses freebsd's template and defines the linker in Makefile.shlib. The piece in Makefile.shlib you add is dead code because PORTNAME will never be "dragonfly" (it would be "freebsd"). I see there is a difference between the existing freebsd code and what you propose to add in that freebsd doesn't use shared object minor versions. Is that also or not the case on DragonFly BSD? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
"David E. Wheeler" writes: > On Mar 1, 2011, at 1:12 PM, Tom Lane wrote: >> The question is what collation property the >> citext type needs to have, and how we get it to have that setting during >> an upgrade from 9.0. See >> http://archives.postgresql.org/message-id/11548.1297983...@sss.pgh.pa.us > Ah, I remember now. That lead to this: > http://archives.postgresql.org/pgsql-hackers/2011-02/msg01824.php > So it looks like what Peter said about updating the catalog is what needs to > be done, and is simple, yes? But then pg_dump needs more collation-juju. Am I > right? Yeah, the real problem in my mind is not so much citext as whether the current representation of a type's collation property is sane in the first place. Once we've thrashed that out, I'll be happy with a simple "UPDATE pg_type" kluge in the citext upgrade script. Doing anything cleaner-looking than that is a future project (and might never happen, seeing that we've never felt a need for ALTER TYPE commands for other properties of a basic type). So really I guess the release-blocker issue is http://archives.postgresql.org/message-id/27152.1299015...@sss.pgh.pa.us regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TYPE COLLATABLE?
Peter Eisentraut writes: > On fre, 2011-02-18 at 11:45 -0500, Tom Lane wrote: >> While testing a fix for this, I observe that pg_dump is entirely >> broken on the subject, because it fails to dump anything at all about >> the typcollation property when dumping a base type. > This is now fixed. >> I also rather wonder >> exactly what pg_dump would dump to restore a value of >> pg_type.typcollation that's not either 0 or 100. > It does what pg_dump does with other unrecognized or invalid type > attributes: it ignores them. I can't say that this makes me think any better of the design here. If a boolean true/false is a sufficient representation of a type's collation property, why isn't the column in pg_type just a boolean? If the idea of storing an OID is to allow reference to a choice of collations, why are we painting ourselves into a corner by dumping it as a boolean? ISTM there are exactly two sane choices here: 1. Change the pg_type column to a boolean. 2. Change the CREATE TYPE command's representation of the COLLATION property to be the name of the referenced collation. If there is a reason why the current design is actually correct, please explain it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Porting PostgreSQL to DragonFly BSD
On Tuesday 1. of March 2011 16:07:47 Tom Lane wrote: > Rumko writes: > > On Sunday 27. of February 2011 23:50:17 Peter Eisentraut wrote: > >> It seems to me that it would be easier to just map dragonfly to the > >> freebsd template. > > > > I didn't see a precedence for that kind of introduction of a new platform > > (all others seem to have their own templates), so thought it would've had > > less chance of being accepted. > > When I looked at that patch, I actually couldn't see anything different > at all from freebsd. We aren't interested in maintaining 99% duplicate > code --- it seems much easier from a maintenance standpoint to introduce > an ifdef or two into the freebsd code, rather than deal with a large cut > and paste job. Please try it that way and see what you come up with. > (It's possible that it'll be so ugly that we agree your original patch > is better, but we need to see the results of the experiment.) > > regards, tom lane Well, wouldn't consider it ugly, but the patch (attached and available at http://www.rumko.net/0001-DragonFly-BSD-support-linked.patch ) is a lot shorter. Uses freebsd's template and defines the linker in Makefile.shlib. -- Regards, Rumko From cf98b549799fac991ee627f16f6131c092913101 Mon Sep 17 00:00:00 2001 From: Rumko Date: Sun, 27 Feb 2011 20:56:11 +0100 Subject: [PATCH] DragonFly BSD support. Based on FreeBSD port with minimal changes. --- configure |1 + configure.in |1 + src/Makefile.shlib | 11 +++ 3 files changed, 13 insertions(+), 0 deletions(-) diff --git a/configure b/configure index 0bddb5f..b4da314 100755 --- a/configure +++ b/configure @@ -2196,6 +2196,7 @@ case $host_os in darwin*) template=darwin ;; dgux*) template=dgux ;; freebsd*) template=freebsd ;; +dragonfly*) template=freebsd ;; hpux*) template=hpux ;; irix*) template=irix ;; linux*|gnu*|k*bsd*-gnu) diff --git a/configure.in b/configure.in index 6aae504..5178570 100644 --- a/configure.in +++ b/configure.in @@ -60,6 +60,7 @@ case $host_os in darwin*) template=darwin ;; dgux*) template=dgux ;; freebsd*) template=freebsd ;; +dragonfly*) template=freebsd ;; hpux*) template=hpux ;; irix*) template=irix ;; linux*|gnu*|k*bsd*-gnu) diff --git a/src/Makefile.shlib b/src/Makefile.shlib index a5cf6c6..a78c98c 100644 --- a/src/Makefile.shlib +++ b/src/Makefile.shlib @@ -198,6 +198,17 @@ ifeq ($(PORTNAME), netbsd) endif endif +ifeq ($(PORTNAME), dragonfly) + ifdef ELF_SYSTEM +LINK.shared= $(COMPILER) -shared +ifdef soname + LINK.shared += -Wl,-x,-soname,$(soname) +endif + else +LINK.shared= $(LD) -x -Bshareable -Bforcearchive + endif +endif + ifeq ($(PORTNAME), hpux) ifdef SO_MAJOR_VERSION shlib = lib$(NAME)$(DLSUFFIX).$(SO_MAJOR_VERSION) -- 1.7.3.5 signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On Tue, Mar 1, 2011 at 22:20, Dimitri Fontaine wrote: > Tom Lane writes: >> Any other "must fix" items on people's minds? > > I'd like it if Magnus could commit his last round of work on > pg_basebackup to stream the WALs in a subprocess. It's been about ready > and waiting for more tests and code review while I've been ill. I > should be able to get there by Friday, on the parts where I can help. I'd love to, but there's at least one bug still in there that needs to be tracked down. And I'm doing training most of this week, so unless someone else can pitch in, it may not be done on time. And while it's nice, I don't think it falls under "*must* fix". -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
Tom Lane writes: > Any other "must fix" items on people's minds? I'd like it if Magnus could commit his last round of work on pg_basebackup to stream the WALs in a subprocess. It's been about ready and waiting for more tests and code review while I've been ill. I should be able to get there by Friday, on the parts where I can help. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On Mar 1, 2011, at 1:12 PM, Tom Lane wrote: > Collation, not correlation. Yeah, I'm fat-fingered today. > The question is what collation property the > citext type needs to have, and how we get it to have that setting during > an upgrade from 9.0. See > http://archives.postgresql.org/message-id/11548.1297983...@sss.pgh.pa.us Ah, I remember now. That lead to this: http://archives.postgresql.org/pgsql-hackers/2011-02/msg01824.php So it looks like what Peter said about updating the catalog is what needs to be done, and is simple, yes? But then pg_dump needs more collation-juju. Am I right? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
"David E. Wheeler" writes: > On Mar 1, 2011, at 12:35 PM, Tom Lane wrote: >> * Collation-related changes still needed in contrib/citext. If we don't >> have this done before alpha4, I'm afraid we'll have to generate a 1.1 >> update script to fix it for alpha4 users. I'd just as soon not deal >> with that overhead. > What needs to happen here, exactly? I'm ignorant about about the correlation > changes, but need to learn, and of course am willing to help out with citext, > especially if correlation support might make it better. Collation, not correlation. The question is what collation property the citext type needs to have, and how we get it to have that setting during an upgrade from 9.0. See http://archives.postgresql.org/message-id/11548.1297983...@sss.pgh.pa.us regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python tracebacks
On tis, 2011-03-01 at 21:10 +0100, Jan Urbański wrote: > So you end up with a context message saying "PL/Python function %s" > and a detail message with the saved detail (if it's present) *and* the > traceback. The problem is that the name of the function is already in > the traceback, so there's no need for the context *if* there's a > traceback present. I wouldn't actually worry about that bit of redundancy so much. Getting proper context for nested calls is much more important. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On tis, 2011-03-01 at 12:50 -0800, David E. Wheeler wrote: > On Mar 1, 2011, at 12:35 PM, Tom Lane wrote: > > > * Collation-related changes still needed in contrib/citext. If we don't > > have this done before alpha4, I'm afraid we'll have to generate a 1.1 > > update script to fix it for alpha4 users. I'd just as soon not deal > > with that overhead. > > What needs to happen here, exactly? I'm ignorant about about the correlation > changes, but need to learn, and of course am willing to help out with citext, > especially if correlation support might make it better. I think you just need to put something like UPDATE pg_type SET typcollation = (SELECT oid FROM pg_collation WHERE collname = 'default') WHERE typname = 'citext'; into the upgrade script. The discussion last time was whether there should be ALTER TYPE support for that, but I guess we settled on not bothering. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On 03/01/2011 03:53 PM, Jan Urbański wrote: On 01/03/11 21:35, Tom Lane wrote: Josh Berkus writes: I'm ok with closing things as of the end of the 15 days, say Thursday or Friday. It might be a good idea to make a list of what we have left to do before we can wrap an alpha. Here are some things on my list. Not all of them are necessarily release blockers, but we need to discuss which ones are: * Regression test failures from recent plpython patches. These are affecting enough machines to make them "must fix before alpha", IMO. There are some variations in error message wording, which are not too terrible but also not exactly hard to fix. The python assert failure that some Fedora machines are reporting is considerably more disturbing. I agree. I'm looking into the crash, no luck so long. Is there anything you need that would help you? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On 01/03/11 21:35, Tom Lane wrote: > Josh Berkus writes: >> I'm ok with closing things as of the end of the 15 days, say Thursday or >> Friday. > > It might be a good idea to make a list of what we have left to do before > we can wrap an alpha. Here are some things on my list. Not all of them > are necessarily release blockers, but we need to discuss which ones are: > > * Regression test failures from recent plpython patches. These are > affecting enough machines to make them "must fix before alpha", IMO. > There are some variations in error message wording, which are not too > terrible but also not exactly hard to fix. The python assert failure > that some Fedora machines are reporting is considerably more disturbing. I'm looking into the crash, no luck so long. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On Mar 1, 2011, at 12:35 PM, Tom Lane wrote: > * Collation-related changes still needed in contrib/citext. If we don't > have this done before alpha4, I'm afraid we'll have to generate a 1.1 > update script to fix it for alpha4 users. I'd just as soon not deal > with that overhead. What needs to happen here, exactly? I'm ignorant about about the correlation changes, but need to learn, and of course am willing to help out with citext, especially if correlation support might make it better. David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On 3/1/11 12:35 PM, Tom Lane wrote: > * Generate alpha release notes. This is at least half a day's work > for somebody, I think, even with our fairly low standards for alpha > release notes. I can help with this. Possibly Selena can too. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
> That response is just dodging the hard question, so whatever. Tom's > cleanup is not going to break things, or at least it's going to fix > more than it breaks on net. Sync rep, on the other hand, is going to > do the opposite, probably by a large margin. Well, we need to at least give Simon and Fujii a chance to respond during their respective daytime hours. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
Robert Haas writes: > Bruce has been going through the open items for the past several weeks > (at least) and tells me that he hasn't found very much. I'm not sure > what your thought is on what's required to get us from here to beta, > but I am thinking it could be done in a few weeks. With a concerted > effort and some sustained focus, I don't see why we could get this > release out the door in, say, three months. Taking in a feature > that's going to take another month to sort out is going to push that > out, and I am really not excited about another round of > spend-all-summer-waiting-for-people-to-get-back-from-vacation-and-release-in-September. Yeah, it would be really nice to get the release out in June rather than September. If we wait any longer for Sync Rep I'm pretty sure it's going to be the latter not the former. See my nearby message for a start at a list of what we "must" do to get to alpha4. Any features we want to cram in at this stage go on top of that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 3:36 PM, Josh Berkus wrote: > >> That's still missing the point, which is that the code is unlikely to >> be up to our usual standards at that point. > > I was talking about "when do we close the CF and start building an > alpha", not synch rep particularly. Tom said he wants until Friday > anyway just for some cleanup. That response is just dodging the hard question, so whatever. Tom's cleanup is not going to break things, or at least it's going to fix more than it breaks on net. Sync rep, on the other hand, is going to do the opposite, probably by a large margin. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
> That's still missing the point, which is that the code is unlikely to > be up to our usual standards at that point. I was talking about "when do we close the CF and start building an alpha", not synch rep particularly. Tom said he wants until Friday anyway just for some cleanup. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
Josh Berkus writes: > I'm ok with closing things as of the end of the 15 days, say Thursday or > Friday. It might be a good idea to make a list of what we have left to do before we can wrap an alpha. Here are some things on my list. Not all of them are necessarily release blockers, but we need to discuss which ones are: * Regression test failures from recent plpython patches. These are affecting enough machines to make them "must fix before alpha", IMO. There are some variations in error message wording, which are not too terrible but also not exactly hard to fix. The python assert failure that some Fedora machines are reporting is considerably more disturbing. * Collation-related regression failure on buildfarm member pika. This is clearly a bug we need to identify, but maybe we can ship the alpha without a fix --- for one thing, getting more than one report of the problem would be helpful. * Collation-related changes still needed in contrib/citext. If we don't have this done before alpha4, I'm afraid we'll have to generate a 1.1 update script to fix it for alpha4 users. I'd just as soon not deal with that overhead. * Rearrange pg_proc and pg_operator comments as suggested here: http://archives.postgresql.org/pgsql-docs/2010-10/msg00041.php While this isn't a "must fix", the very end of a development cycle is clearly the best time for such a patch, since at any other time there are going to be a larger number of pending patches that would have to be adjusted. So I'd kind of like to get this done, if I can spare a day for it. * Generate alpha release notes. This is at least half a day's work for somebody, I think, even with our fairly low standards for alpha release notes. There are other things I'd like to do, like review and perhaps commit the btree_gist patch, but they're not at the level of "must fix". Any other "must fix" items on people's minds? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 3:01 PM, Josh Berkus wrote: > On 3/1/11 11:58 AM, Tom Lane wrote: >> If we do hold up the release, I'll probably go back and reopen the >> postgresql_fdw patch as well as btree_gist. So I won't run out of >> things to do. But I'm not terribly satisfied with the decision-making >> process here. > > OK, well, we gave it an extra 15 days, which was all we promised. > > I'm ok with closing things as of the end of the 15 days, say Thursday or > Friday. That's still missing the point, which is that the code is unlikely to be up to our usual standards at that point. So far I don't hear anyone arguing strongly that we should accept sync rep in 9.1, and several people arguing the reverse. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python tracebacks
On Mar 1, 2011, at 12:10 PM, Jan Urbański wrote:
> So you end up with a context message saying "PL/Python function %s" and
> a detail message with the saved detail (if it's present) *and* the
> traceback. The problem is that the name of the function is already in
> the traceback, so there's no need for the context *if* there's a
> traceback present.
>
> The problem I'm having is technical: since the callback is already set
> when the code reaches the traceback-printing stage, you can't really
> unset it. AFAICS the elog code calls *all* callbacks from
> error_context_stack. So I can't prevent the context message from
> appearing. If I make the traceback part of the context as well, it's
> just going to appear together with the message from the callback.
I remember going through a lot of pain getting this done "right"
in pg-python[pl/py].
SELECT it_blows_up();
ERROR: function's "main" raised a Python exception
CONTEXT: [exception from Python]
Traceback (most recent call last):
File "public.it_blows_up()", line 13, in main
three()
File "public.it_blows_up()", line 10, in three
return two()
File "public.it_blows_up()", line 7, in two
return one()
File "public.it_blows_up()", line 4, in one
raise OverflowError("there's water everywhere")
OverflowError: there's water everywhere
[public.it_blows_up()]
IIRC, I unconditionally print the "[public.it_blows_up()]" part iff it's
not an ERROR. If it is an ERROR, I let the traceback rendering part of the code
handle it on the PL's entry point exit. It was really tricky to do this because
I
was rendering the traceback *after* the error_context_stack had been called.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, 2011-03-01 at 14:26 -0500, Robert Haas wrote: > On Tue, Mar 1, 2011 at 2:12 PM, Josh Berkus wrote: > > > >>> Since we appear to be still holding the commitfest open for Sync Rep, > >>> I guess this ought to get reviewed. > >> > >> Or else we should close the CommitFest and cut alpha4. Anyone have an > >> opinion on which way to go? > > > > I think we can give Sync Rep until the 15th, given the pace of work on > > it. It is a major feature, and a complicated one. > > Sure, but there are other features, major and minor, that we have > postponed to 9.2. In the normal course of events, sync rep would have > been marked Returned with Feedback a month ago. I like the feature, > but I have to say I'm not very pleased that we seem to have fallen > into a pattern of believing that some major features are somehow > exempted from the scheduling deadline and others are not. Neither am I, I mean, we were actively fixing and bringing Fk-Locks up to date and we got pushed but sync rep gets in? We may have had fk-locks baked by now. I want syncrep as much as the next person, but this isn't really fair to any of the other submitters. JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 2:58 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Mar 1, 2011 at 2:45 PM, Tom Lane wrote: >>> I'd say that if there's a plausible chance that Sync Rep will be >>> committable by the end of *this* week (and I mean Friday not Sunday), >>> I'm willing to wait that long for it. Otherwise, it's 9.2 material. > >> I am quite sure that Simon will be able to get something committed >> ahead of whatever deadline we choose to set. Whether that commit will >> be up to our usual standards is another question altogether. The last >> version posted to the list was trivial to break, and that was several >> weeks ago. > > Yeah, there's that. It's difficult to believe that anything committed > in the very short term wouldn't be rushed to completion rather than > really ready. That holds whether the deadline is this week or two > weeks out. Yep. > The other issue is that, as Robert says, we have already cut Sync Rep > more slack than any other patch in the commitfest. It does not seem > fair to hold up the release process another week or two for it, even > assuming that we get a high-quality feature at the end of that. Yep. > If we do hold up the release, I'll probably go back and reopen the > postgresql_fdw patch as well as btree_gist. So I won't run out of > things to do. But I'm not terribly satisfied with the decision-making > process here. Well, we haven't actually made a decision here yet. We're just talking about what decision we ought to make. Frankly, I avoided trying to mark Sync Rep Returned with Feedback mostly for the reason that I knew Simon would object, and his commit bit gives him a certain degree of latitude to ignore the CF process anyway. But I am really not that keen on having Sync Rep go in and then spending another month fixing all the bugs, and that's what I think will happen. Bruce has been going through the open items for the past several weeks (at least) and tells me that he hasn't found very much. I'm not sure what your thought is on what's required to get us from here to beta, but I am thinking it could be done in a few weeks. With a concerted effort and some sustained focus, I don't see why we could get this release out the door in, say, three months. Taking in a feature that's going to take another month to sort out is going to push that out, and I am really not excited about another round of spend-all-summer-waiting-for-people-to-get-back-from-vacation-and-release-in-September. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI bug?
On Tue, Mar 01, 2011 at 07:07:42PM +0200, Heikki Linnakangas wrote: > Was there test cases for any of the issues fixed by this patch that we > should add to the suite? Some of these issues are tricky to test, e.g. some of the code about transferring predicate locks to a new target doesn't get exercised unless an index page gets split while there are concurrent transactions holding locks on that page. I have not been able to find a good way to test these other than system-level testing using a concurrent workload (usually the DBT-2 benchmark). I'd certainly be open to suggestions! Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python tracebacks
On 01/03/11 20:35, Tom Lane wrote: > =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: >> Currently the traceback is added to the detail and the original >> errdetail is preserved. So you'd get the detail line and the traceback >> below it. > > Hm? I'm talking about plpython_error_callback() and friends, which > AFAICS you haven't changed the behavior of at all. And it would > certainly be completely inappropriate to do what's said above for > an errdetail with a non-plpython origin. Not sure if I understand the problem. PL/Python sets plpython_error_callback right after entering the function call handler, so any elog thrown while the function is executing has a "PL/Python function %s" context message. If plpython calls into SQL with SPI and something there throws an elog(ERROR) with an errdetail, that detail is saved inside the exception and a Python error is then thrown. If this Python error reaches the top of the Python stack, the error reporting code kicks in, extracts the saved errdetail value from the Python exception, and then extract the stack trace and also adds it to the errdetail. So you end up with a context message saying "PL/Python function %s" and a detail message with the saved detail (if it's present) *and* the traceback. The problem is that the name of the function is already in the traceback, so there's no need for the context *if* there's a traceback present. The problem I'm having is technical: since the callback is already set when the code reaches the traceback-printing stage, you can't really unset it. AFAICS the elog code calls *all* callbacks from error_context_stack. So I can't prevent the context message from appearing. If I make the traceback part of the context as well, it's just going to appear together with the message from the callback. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 3:08 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Mar 1, 2011 at 2:52 PM, Josh Berkus wrote: >>> Ideally, we want to have some binaries/packages for the "final alpha". >>> Those broaden testing considerably. > >> When we have a version that needs that treatment, we can simply call >> it beta1. If it's too half-baked for that, then I don't see the point >> in going to a lot of trouble to build packages. > > We (or more precisely EDB) made Windows installers for alpha1: > http://www.enterprisedb.com/products-services-training/pgdevdownload > And IIRC they did installers for alphas in the 9.0 cycle too. And > certainly Devrim and others have been building binary packages for > alphas. If this alpha is so much less baked than the previous ones > that that's not worthwhile, there's something very wrong with the > process. The last alpha ought to be in testable condition. Oh, really? OK. I wasn't aware that alphas got installers. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
Robert Haas writes: > On Tue, Mar 1, 2011 at 2:52 PM, Josh Berkus wrote: >> Ideally, we want to have some binaries/packages for the "final alpha". >> Those broaden testing considerably. > When we have a version that needs that treatment, we can simply call > it beta1. If it's too half-baked for that, then I don't see the point > in going to a lot of trouble to build packages. We (or more precisely EDB) made Windows installers for alpha1: http://www.enterprisedb.com/products-services-training/pgdevdownload And IIRC they did installers for alphas in the 9.0 cycle too. And certainly Devrim and others have been building binary packages for alphas. If this alpha is so much less baked than the previous ones that that's not worthwhile, there's something very wrong with the process. The last alpha ought to be in testable condition. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On 3/1/11 11:58 AM, Tom Lane wrote: > If we do hold up the release, I'll probably go back and reopen the > postgresql_fdw patch as well as btree_gist. So I won't run out of > things to do. But I'm not terribly satisfied with the decision-making > process here. OK, well, we gave it an extra 15 days, which was all we promised. I'm ok with closing things as of the end of the 15 days, say Thursday or Friday. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
Robert Haas writes: > On Tue, Mar 1, 2011 at 2:45 PM, Tom Lane wrote: >> I'd say that if there's a plausible chance that Sync Rep will be >> committable by the end of *this* week (and I mean Friday not Sunday), >> I'm willing to wait that long for it. Otherwise, it's 9.2 material. > I am quite sure that Simon will be able to get something committed > ahead of whatever deadline we choose to set. Whether that commit will > be up to our usual standards is another question altogether. The last > version posted to the list was trivial to break, and that was several > weeks ago. Yeah, there's that. It's difficult to believe that anything committed in the very short term wouldn't be rushed to completion rather than really ready. That holds whether the deadline is this week or two weeks out. The other issue is that, as Robert says, we have already cut Sync Rep more slack than any other patch in the commitfest. It does not seem fair to hold up the release process another week or two for it, even assuming that we get a high-quality feature at the end of that. If we do hold up the release, I'll probably go back and reopen the postgresql_fdw patch as well as btree_gist. So I won't run out of things to do. But I'm not terribly satisfied with the decision-making process here. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 2:52 PM, Josh Berkus wrote: > Ideally, we want to have some binaries/packages for the "final alpha". > Those broaden testing considerably. When we have a version that needs that treatment, we can simply call it beta1. If it's too half-baked for that, then I don't see the point in going to a lot of trouble to build packages. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
> As I understand it, it requires only the steps described here: > > http://wiki.postgresql.org/wiki/Alpha_release_process > > That all looks pretty straightforward, assuming you can log into > developer.postgresql.org, which I can't. I think I remember having an > ftp account at some point, but it's not accepting connections on port > 22, so there is doubtless some secret sauce I am missing here. Ideally, we want to have some binaries/packages for the "final alpha". Those broaden testing considerably. We don't need them for all platforms, of course. Really, the critical ones for testing are Windows and OSX. Linux/BSD/Solaris users are pretty good at make/make install. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
Hi, On Tuesday, March 01, 2011 11:50:42 AM Nikhil Sontakke wrote: > >> Will try to get the call stack if needed. > > > > Yes, please. > Here is the stack trace: Thats not a stock postgres is it? Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 2:38 PM, Magnus Hagander wrote: >> Frankly, I think we should be aiming to get a beta out in April, not >> another alpha. > > That would be good, but in order to get there, +1 for cutting a new > alpha even if syncrep isn't ready for it yet. That way we can at least > get some more testing on all the non-syncrep code. > > That is assuming that cutting an alpha release isn't all that much > work, but IIRC it's not. (Hey, I know it's not much work for me, but > someone who actually does the work should comment on the total > amount...) As I understand it, it requires only the steps described here: http://wiki.postgresql.org/wiki/Alpha_release_process That all looks pretty straightforward, assuming you can log into developer.postgresql.org, which I can't. I think I remember having an ftp account at some point, but it's not accepting connections on port 22, so there is doubtless some secret sauce I am missing here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 2:45 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Mar 1, 2011 at 2:12 PM, Josh Berkus wrote: >>> I think we can give Sync Rep until the 15th, given the pace of work on >>> it. It is a major feature, and a complicated one. > >> Sure, but there are other features, major and minor, that we have >> postponed to 9.2. In the normal course of events, sync rep would have >> been marked Returned with Feedback a month ago. I like the feature, >> but I have to say I'm not very pleased that we seem to have fallen >> into a pattern of believing that some major features are somehow >> exempted from the scheduling deadline and others are not. > > Yes. What are the rest of us supposed to do for the next two weeks, > twiddle our thumbs? > > Personally I've got a couple of days' worth of cleanup tasks before I'd > want to see us cut an alpha anyway, especially if we're going to try > to accept the btree_gist KNNgist patch. Two weeks is too much though. > > I'd say that if there's a plausible chance that Sync Rep will be > committable by the end of *this* week (and I mean Friday not Sunday), > I'm willing to wait that long for it. Otherwise, it's 9.2 material. I am quite sure that Simon will be able to get something committed ahead of whatever deadline we choose to set. Whether that commit will be up to our usual standards is another question altogether. The last version posted to the list was trivial to break, and that was several weeks ago. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mysql2pgsql.perl update
On Mon, Feb 28, 2011 at 11:26 AM, Josh Berkus wrote: > >> does foundry support git or should I just link to a repo on github? > > If you prefer using git, the latter. Ok, will do. Assign the project and I'll update stuff. Andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Native XML
On 03/01/2011 02:15 PM, Kevin Grittner wrote: Given that there were similar issues for other hierarchical data types, perhaps we need something similar to tsvector, but for hierarchical data. The extra layer of abstraction might not cost much when used for XML compared to the possible benefit with other data. It seems likely to be a very nice fit with GiST indexes. So under this idea, you would always have the text (or maybe byte array?) version of the XML, and you could "shard" it to a separate column for fast searches. Tsearch should be able to handle XML now. It certainly knows how to recognize XML tags. I apparently didn't express myself very well, since you seem to have *completely* missed my point. I know we can do tsearch2 searches against XML, or JSON, or YAML, or (insert next week's new favorite format here). What we can't currently do efficiently is search for particular values in some particular place in the hierarchy of a document. I've had loads of fun approximating it with regular expressions, but some days I'd like life to be easier. What I was arguing for is a new type which would represent the structure in a fashion which was independent of the particular text format and was efficient to traverse hierarchically. Done right, that would map well to GiST. Although, thinking about that some more, perhaps there would be a way to create a GiST index suitable for that straight from the XML text, and avoid the sharded column. A GiST index actually seems pretty close to what such a structure would look like anyway I probably didn't read your suggestion closely enough. I think hierarchical data really only scratches the surface of the problem. It would be nice to be able to specify all sorts of context for searches: * foo after bar * foo near bar * foo and bar in the same paragraph * foo as a parent/child/ancestor/descendent/sibling/cousin of bar cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
Robert Haas writes: > On Tue, Mar 1, 2011 at 2:12 PM, Josh Berkus wrote: >> I think we can give Sync Rep until the 15th, given the pace of work on >> it. It is a major feature, and a complicated one. > Sure, but there are other features, major and minor, that we have > postponed to 9.2. In the normal course of events, sync rep would have > been marked Returned with Feedback a month ago. I like the feature, > but I have to say I'm not very pleased that we seem to have fallen > into a pattern of believing that some major features are somehow > exempted from the scheduling deadline and others are not. Yes. What are the rest of us supposed to do for the next two weeks, twiddle our thumbs? Personally I've got a couple of days' worth of cleanup tasks before I'd want to see us cut an alpha anyway, especially if we're going to try to accept the btree_gist KNNgist patch. Two weeks is too much though. I'd say that if there's a plausible chance that Sync Rep will be committable by the end of *this* week (and I mean Friday not Sunday), I'm willing to wait that long for it. Otherwise, it's 9.2 material. > Frankly, I think we should be aiming to get a beta out in April, not > another alpha. Quite. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 20:26, Robert Haas wrote: > On Tue, Mar 1, 2011 at 2:12 PM, Josh Berkus wrote: >> Since we appear to be still holding the commitfest open for Sync Rep, I guess this ought to get reviewed. >>> >>> Or else we should close the CommitFest and cut alpha4. Anyone have an >>> opinion on which way to go? >> >> I think we can give Sync Rep until the 15th, given the pace of work on >> it. It is a major feature, and a complicated one. > > Sure, but there are other features, major and minor, that we have > postponed to 9.2. In the normal course of events, sync rep would have > been marked Returned with Feedback a month ago. I like the feature, > but I have to say I'm not very pleased that we seem to have fallen > into a pattern of believing that some major features are somehow > exempted from the scheduling deadline and others are not. I am sure > there are plenty of other people who would have liked a six week > extension of the usual CommitFest deadlines too, but they didn't get > it (and for the most part, were pretty gracious about that). > >> We could even cut a pre-synch-rep Alpha4 *now*, and follow that with a >> post-synch-rep Alpha5 sometime around April 1. >> >> That'll put us in a good position for beta, and also to see what >> specific issue SynR adds. > > Frankly, I think we should be aiming to get a beta out in April, not > another alpha. That would be good, but in order to get there, +1 for cutting a new alpha even if syncrep isn't ready for it yet. That way we can at least get some more testing on all the non-syncrep code. That is assuming that cutting an alpha release isn't all that much work, but IIRC it's not. (Hey, I know it's not much work for me, but someone who actually does the work should comment on the total amount...) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python tracebacks
=?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: > Currently the traceback is added to the detail and the original > errdetail is preserved. So you'd get the detail line and the traceback > below it. Hm? I'm talking about plpython_error_callback() and friends, which AFAICS you haven't changed the behavior of at all. And it would certainly be completely inappropriate to do what's said above for an errdetail with a non-plpython origin. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 2:12 PM, Josh Berkus wrote: > >>> Since we appear to be still holding the commitfest open for Sync Rep, >>> I guess this ought to get reviewed. >> >> Or else we should close the CommitFest and cut alpha4. Anyone have an >> opinion on which way to go? > > I think we can give Sync Rep until the 15th, given the pace of work on > it. It is a major feature, and a complicated one. Sure, but there are other features, major and minor, that we have postponed to 9.2. In the normal course of events, sync rep would have been marked Returned with Feedback a month ago. I like the feature, but I have to say I'm not very pleased that we seem to have fallen into a pattern of believing that some major features are somehow exempted from the scheduling deadline and others are not. I am sure there are plenty of other people who would have liked a six week extension of the usual CommitFest deadlines too, but they didn't get it (and for the most part, were pretty gracious about that). > We could even cut a pre-synch-rep Alpha4 *now*, and follow that with a > post-synch-rep Alpha5 sometime around April 1. > > That'll put us in a good position for beta, and also to see what > specific issue SynR adds. Frankly, I think we should be aiming to get a beta out in April, not another alpha. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Native XML
"Kevin Grittner" writes: > I apparently didn't express myself very well, since you seem to have > *completely* missed my point. I know we can do tsearch2 searches > against XML, or JSON, or YAML, or (insert next week's new favorite > format here). What we can't currently do efficiently is search for > particular values in some particular place in the hierarchy of a > document. I've had loads of fun approximating it with regular > expressions, but some days I'd like life to be easier. Check. > What I was arguing for is a new type which would represent the > structure in a fashion which was independent of the particular text > format and was efficient to traverse hierarchically. Done right, > that would map well to GiST. Although, thinking about that some > more, perhaps there would be a way to create a GiST index suitable > for that straight from the XML text, and avoid the sharded column. > A GiST index actually seems pretty close to what such a structure > would look like anyway FWIW, GIN might be a more natural match, at least for the cases where "place in the document" has a scalar value. If you need to search for "place" with something other than equality or prefix match semantics, maybe not. But in any case I think your point is that this is an indexing problem, and whether the full document in the table column is pre-parsed or not isn't all that relevant for performance. I agree. tsearch2 is really a precedent for your argument, not a distinct approach, because it doesn't expect pre-parsed text columns either. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python tracebacks
On 01/03/11 20:15, Tom Lane wrote: > =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: >> After thinking about it more I believe that the context field should >> keep on being a one line indication of which function the message comes >> from (and that's how it's done in PL/pgSQL for instance), and the detail >> field should be used for the details of the message, like the traceback >> that comes with it, and that's what the attached patch does. > > To me, none of those arguments seem good. Traceback is the sort of > thing that belongs in errcontext, and arbitarily deciding that plpython > isn't going to play by the rules doesn't sit well here. I agree that > what you are showing is redundant with the current errcontext printout, > but the solution for that is to change the errcontext printout, not to > add redundant and inappropriate errdetail. > > An example of the reasoning for this is the situation where a plpython > function calls back into SQL, and something there throws an ereport > (which might include an errdetail). It would be useful to include the > Python traceback in the errcontext stack, since there might be multiple > levels of Python function call within what PG sees as just a "plpython > function". But you can't get there with this approach. Currently the traceback is added to the detail and the original errdetail is preserved. So you'd get the detail line and the traceback below it. But OK, since there are more voices in favour of putting tracebacks in the context field, I'll keep on looking for a solution. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python tracebacks
=?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: > I looked into putting the tracebacks in the context field, but IMHO it > doesn't really play out nice. PL/Python uses a errcontext callback to > populate the context field, so the reduntant information (the name of > the function) is always there. If that callback would be removed, the > context information will not appear in plpy.warning output, which I > think would be bad. > So: the context is currently put unconditionally into every elog > message, which I think is good. In case of errors, the traceback already > includes the procedure name (because of how Python tracebacks are > typically formatted), which makes the traceback contain redundant > information to the context field. Replacing the context field with the > traceback is difficult, because it is populated by the error context > callback. > After thinking about it more I believe that the context field should > keep on being a one line indication of which function the message comes > from (and that's how it's done in PL/pgSQL for instance), and the detail > field should be used for the details of the message, like the traceback > that comes with it, and that's what the attached patch does. To me, none of those arguments seem good. Traceback is the sort of thing that belongs in errcontext, and arbitarily deciding that plpython isn't going to play by the rules doesn't sit well here. I agree that what you are showing is redundant with the current errcontext printout, but the solution for that is to change the errcontext printout, not to add redundant and inappropriate errdetail. An example of the reasoning for this is the situation where a plpython function calls back into SQL, and something there throws an ereport (which might include an errdetail). It would be useful to include the Python traceback in the errcontext stack, since there might be multiple levels of Python function call within what PG sees as just a "plpython function". But you can't get there with this approach. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Native XML
Andrew Dunstan wrote: > On 02/28/2011 05:28 PM, Kevin Grittner wrote: >> Anton wrote: >> >>> it was actually the focal point of my considerations: whether to >>> store plain text or 'something else'. > > There seems to be an almost universal assumption that storing XML > in its native form (i.e. a text stream) is going to produce > inefficient results. Well, certainly not in all cases. Finding those rows which satisfy an XPath search among a few million rows with 20KB XML fields might benefit from sort of indexing, though. > unless we implemented our own XPath processor to work with our own > XML format (do we really want to do that?), to evaluate an XPath > expression for a piece of XML we'd actually need to produce the > text format from our internal format before passing it to some > external library to parse into its internal format and then > process the XPath expression. My suggestion was that you would store the text format, and allow the developer to create a sharded format in a different column with a different type if desired, not the other way around. As I said, similar to what a developer would do for tsvector to allow text searches. I agree that creating the text from an internal format doesn't sound good. >> Given that there were similar issues for other hierarchical data >> types, perhaps we need something similar to tsvector, but for >> hierarchical data. The extra layer of abstraction might not cost >> much when used for XML compared to the possible benefit with >> other data. It seems likely to be a very nice fit with GiST >> indexes. >> >> So under this idea, you would always have the text (or maybe byte >> array?) version of the XML, and you could "shard" it to a >> separate column for fast searches. > Tsearch should be able to handle XML now. It certainly knows how > to recognize XML tags. I apparently didn't express myself very well, since you seem to have *completely* missed my point. I know we can do tsearch2 searches against XML, or JSON, or YAML, or (insert next week's new favorite format here). What we can't currently do efficiently is search for particular values in some particular place in the hierarchy of a document. I've had loads of fun approximating it with regular expressions, but some days I'd like life to be easier. What I was arguing for is a new type which would represent the structure in a fashion which was independent of the particular text format and was efficient to traverse hierarchically. Done right, that would map well to GiST. Although, thinking about that some more, perhaps there would be a way to create a GiST index suitable for that straight from the XML text, and avoid the sharded column. A GiST index actually seems pretty close to what such a structure would look like anyway -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
>> Since we appear to be still holding the commitfest open for Sync Rep, >> I guess this ought to get reviewed. > > Or else we should close the CommitFest and cut alpha4. Anyone have an > opinion on which way to go? I think we can give Sync Rep until the 15th, given the pace of work on it. It is a major feature, and a complicated one. We could even cut a pre-synch-rep Alpha4 *now*, and follow that with a post-synch-rep Alpha5 sometime around April 1. That'll put us in a good position for beta, and also to see what specific issue SynR adds. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Change pg_last_xlog_receive_location not to move backwards
On 26.02.2011 16:58, Robert Haas wrote: It sounds like the only thing we have definite agreement about from all this is that apply_location should be renamed to replay_location in pg_stat_replication, a point fairly incidental to the what this patch is about. It seems fairly unsatisfying to just change that and punt the rest of this, but I'm not sure what the alternative is. After reading the discussions, I don't see any actual opposition to Fujii-san's patch. And I think it makes sense, the new definition makes sense for the purpose Fujii mentioned in the mail that started this thread: determining which standby is most up-to-date. There has been a lot of good suggestions, like verifying the received WAL before overwriting existing WAL. But we're not going to start doing bigger code changes this late in the release cycle. And even if we did, this patch would still make sense - we still wouldn't want pg_last_xlog_receive_location() to move backwards. So, committed. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
wrapping up this CommitFest (was Re: [HACKERS] knngist - 0.8)
On Tue, Mar 1, 2011 at 1:35 PM, Tom Lane wrote: > Robert Haas writes: >> On Mon, Feb 28, 2011 at 2:25 PM, Tom Lane wrote: >>> Given that it is a contrib module, I personally wouldn't object to it >>> getting patched later, like during alpha or beta. But somebody's got >>> to do the work, and I've got a dozen higher-priority problems right now. > >> Well, we can argue about whether it's too late for 9.1 if and when a >> patch shows up. Right now we don't have that problem. > > We do now ... > http://archives.postgresql.org/pgsql-hackers/2011-03/msg00038.php > > Since we appear to be still holding the commitfest open for Sync Rep, > I guess this ought to get reviewed. Or else we should close the CommitFest and cut alpha4. Anyone have an opinion on which way to go? I think it's fair to say that Simon is working pretty actively on Sync Rep and that the bug count is probably dropping rapidly. It seems a shame to push sync rep out to 9.2 in that context. On the other hand, the patch wasn't done at the beginning of the CommitFest, it wasn't done at the scheduled end of the CommitFest, and it's still not done now two weeks after the scheduled end of the CommitFest. If it gets committed O(now), it's probably going to still have bugs and design problems that will take at least a few more weeks to shake out, which will directly add to the length of time that it takes to actually get the release out the door. I could go either way on this one. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] knngist - 0.8
Robert Haas writes: > On Mon, Feb 28, 2011 at 2:25 PM, Tom Lane wrote: >> Given that it is a contrib module, I personally wouldn't object to it >> getting patched later, like during alpha or beta. But somebody's got >> to do the work, and I've got a dozen higher-priority problems right now. > Well, we can argue about whether it's too late for 9.1 if and when a > patch shows up. Right now we don't have that problem. We do now ... http://archives.postgresql.org/pgsql-hackers/2011-03/msg00038.php Since we appear to be still holding the commitfest open for Sync Rep, I guess this ought to get reviewed. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Is the attribute options cache actually worth anything?
On Tue, Mar 1, 2011 at 1:15 PM, Tom Lane wrote: > Did anybody do any performance measurements to demonstrate that this > code has a reason to live? Because if I don't see some, I'm going > to rip it out. No, I have to admit I didn't do that. Might be worth doing some before you commit the rip-out, though. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
On Tue, 2011-03-01 at 10:02 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Tue, 2011-03-01 at 15:51 +0900, Fujii Masao wrote: > >> A spinlock can be used only for very short-term operation like > >> read/write of some shared-variables. The operation on the queue > >> is not short, so should be protected by LWLock, I think. > > > There's no need to sleep while holding locks and the operations are very > > short in most cases. The code around it isn't trivial, but that's no > > reason to use LWlocks. > > What does "in most cases" mean? > > > LWlocks are just spinlocks plus sem sleeps, so I don't see the need for > > that in the current code. Other views welcome. > > Simon, that is absolutely NOT acceptable. Spinlocks are to be used only > for short straight-line code segments. If the lock has any potential to > be held for more than nanoseconds, use an LWLock. The contention costs > of the shortcut you propose are too high. No problem to change. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Is the attribute options cache actually worth anything?
So while poking at a recent example from Marc Cousin (hundreds of tables each with 1000 attributes) I observed that a simple ANALYZE would bloat the backend process to the tune of several hundred megabytes. I think there is a leak in CacheMemoryContext, but haven't tracked it down yet. But I also noticed that tens of megabytes were disappearing into "Attopt cache", and after reading the code to see what the heck that was, I am wondering what the justification for having it is at all. In the presumably normal case where the attribute hasn't got options, all it's saving us is a syscache access, which is probably not noticeably more expensive than the hash lookup. In the case where there is an option, it's saving us an attribute_reloptions() call, but it's not apparent to me that that's so expensive as to justify putting a cache in front of it, especially not if we're going to do a palloc cycle anyway. Did anybody do any performance measurements to demonstrate that this code has a reason to live? Because if I don't see some, I'm going to rip it out. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] knngist - 0.8
I did a quick look at this patch. The major problem with it is of course that it needs to be fixed for the recent extension-related changes. I transposed the .sql.in changes into additions to btree_gist--1.0.sql (attached), but haven't really sanity-checked them beyond checking that the regression tests pass. The same mods would need to be made in btree_gist--unpackaged--1.0.sql. Fixed 1. oid_dist() returns oid ... really? Oid is unsigned. I'd be inclined to argue though that distance between Oids is a meaningless concept, so Hmm, oid is often used as unsigned int. you should remove this not just mess with the result type. Anybody who actually wants to form a distance between Oids should have to cast them to an arithmetic type first. Let the user figure out how wraparound cases should be handled. Distance between unsigned 32-bit integers could not be more than 2^32. 2. Beyond that, none of the distance routines have given any thought to avoiding overflow. For instance, dist_int2 had better return something wider than int2, and so on up. It looks to me like the internal gist Just like other operations: # select 32000::smallint + 32000::smallint; ERROR: smallint out of range distance functions also suffer overflow risks, in that they tend to form the difference first (in the source datatype) and only afterwards cast to float8. fixed 3. I was surprised that there wasn't a distance implementation for numeric. I suppose that this might be difficult to do without risking overflow in conversion to float8, though. Exactly 4. I didn't much care for changing the result type of gbt_num_consistent from bool to float8; that's just messy, and I don't see any compensating advantage. I suggest you leave gbt_num_consistent and its callers alone, and add a separate gbt_num_distance routine that only handles the KNNDistance case. Done -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ builtin_knngist_contrib_btree_gist-0.12.gz Description: Unix tar archive -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI bug?
Heikki Linnakangas wrote: > committed with minor changes. Thanks! > The ordering of the fields in PREDICATELOCKTAG was bizarre, so I > just expanded the offsetnumber fields to an uint32, instead of > having the padding field. I think that's a lot more readable. I can understand that, but I wonder if we shouldn't have a comment mentioning that the offsetnumber field is larger than needed, in case someone later needs to add another 16 bit field for some reason, or we go to a hash function without the same quirks. > I also added an optimization in PredicateLockTupleRowVersionLink() > to not try to transfer the page locks, if the new tuple is on the > same page as the old one. That's very cheap to check, and it's > very common for an update to stay within a page. Thanks. I had it in mind to do that, but lost track of it. Definitely worth doing. > Was there test cases for any of the issues fixed by this patch > that we should add to the suite? No, but I'm still intending to look at that some more. It makes me nervous to have an area which would be pretty easy for someone to break without any tests to catch such breakage. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI bug?
On 01.03.2011 02:03, Dan Ports wrote: An updated patch to address this issue is attached. It fixes a couple issues related to use of the backend-local lock table hint: - CheckSingleTargetForConflictsIn now correctly handles the case where a lock that's being held is not reflected in the local lock table. This fixes the assertion failure reported in this thread. - PredicateLockPageCombine now retains locks for the page that is being removed, rather than removing them. This prevents a potentially dangerous false-positive inconsistency where the local lock table believes that a lock is held, but it is actually not. - add some more comments documenting the times when the local lock table can be inconsistent with reality, as reflected in the shared memory table. This patch also incorporates Kevin's changes to copy locks when creating a new version of a tuple rather than trying to maintain a linkage between different versions. So this is a patch that should apply against HEAD and addresses all outstanding SSI bugs known to Kevin or myself. Thanks, committed with minor changes. The ordering of the fields in PREDICATELOCKTAG was bizarre, so I just expanded the offsetnumber fields to an uint32, instead of having the padding field. I think that's a lot more readable. I also added an optimization in PredicateLockTupleRowVersionLink() to not try to transfer the page locks, if the new tuple is on the same page as the old one. That's very cheap to check, and it's very common for an update to stay within a page. Was there test cases for any of the issues fixed by this patch that we should add to the suite? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python custom datatype parsers
On fre, 2011-02-11 at 16:49 +0100, Jan Urbański wrote: > I believe it's (b). But as we don't have time for that discussion that > late in the release cycle, I think we need to consider it identical to (c). As I previously mentioned, I think that there should be an SQL-level way to tie together languages and types. I previously mentioned the SQL-standard command CREATE TRANSFORM as a possibility. I've had this on my PL/Python TOTHINK list for a while. Thankfully you removed all the items ahead of this one, so I'll think of something to do in 9.2. Of course we'll be able to use the actual transform code that you already wrote. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TYPE COLLATABLE?
On fre, 2011-02-18 at 11:45 -0500, Tom Lane wrote: > While testing a fix for this, I observe that pg_dump is entirely > broken on the subject, because it fails to dump anything at all about > the typcollation property when dumping a base type. This is now fixed. > I also rather wonder > exactly what pg_dump would dump to restore a value of > pg_type.typcollation that's not either 0 or 100. It does what pg_dump does with other unrecognized or invalid type attributes: it ignores them. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] OSSP gone missing? Fate of UUID?
On Mon, Feb 28, 2011 at 4:08 PM, Michael Glaesemann wrote: > A couple of weeks ago when installing uuid-ossp on a new server, I noticed > that the ossp site is gone. I haven't found anything on the web to indicate > what happened. > > Anyone know? Maybe it's time to provide our own UUID generator. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug of the hot standby feedback
On Mon, Feb 28, 2011 at 8:54 AM, Fujii Masao wrote: > When I implemented the replication timeout patch, I found the bug on > the HS feedback feature. When wal_receiver_status_interval is zero > and hot_standby_feedback is enabled, walreceiver sends the feedback > too aggressively. I think that the attached patch should be applied to > fix the bug. Committed, with some related documentation changes. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] error order when debug postgresql step by step?
It works! Thanks Kevin and Lbrar! :) 2011/3/1 Ibrar Ahmed : > - export CFLAGS='-O0' may work for you. > > > On Tue, Mar 1, 2011 at 8:21 PM, Kevin Grittner > wrote: >> hom wrote: >> >>> How can I do to make sure the right excute order when I debug step >>> by step ? >> >> You may need to reduce the optimization level of your compile. >> >> -Kevin >> >> -- >> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) >> To make changes to your subscription: >> http://www.postgresql.org/mailpref/pgsql-hackers >> > > > > -- > Ibrar Ahmed > -- Best Wishes! hom -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] error order when debug postgresql step by step?
- export CFLAGS='-O0' may work for you. On Tue, Mar 1, 2011 at 8:21 PM, Kevin Grittner wrote: > hom wrote: > >> How can I do to make sure the right excute order when I debug step >> by step ? > > You may need to reduce the optimization level of your compile. > > -Kevin > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- Ibrar Ahmed -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
Nikhil Sontakke writes: > On Tue, Mar 1, 2011 at 10:17 PM, Heikki Linnakangas > wrote: >> Hmm, it looks like ImmediateInterruptOK is set, while we're busy running a >> query. How come? Can you debug that? Where does it get set? > Ah, this is not exactly an easily reproducible problem :( You gotta be > lucky enough to be able to interrupt inside a malloc call. No, the question is why is the ImmediateInterruptOK flag set. Whether the interrupt actually happens isn't that relevant. You could try setting a watchpoint on the flag variable. > But adding hold/resume interrrupts in mcxt.c (not aset.c, since we > want to be agnostic to the underlying layer) should be good enough to > handle this non-re-entrant issue, no? We are not doing that, because that would be only a band-aid patch for approximately 0.1% of the problems that can arise from running random code with ImmediateInterruptOK set. We need to find out what's leaving that set and fix it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] error order when debug postgresql step by step?
hom wrote: > How can I do to make sure the right excute order when I debug step > by step ? You may need to reduce the optimization level of your compile. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
Robert Haas writes: > On Tue, Mar 1, 2011 at 3:21 AM, Simon Riggs wrote: >> LWlocks are just spinlocks plus sem sleeps, so I don't see the need for >> that in the current code. Other views welcome. > An LWLock is a lot safer, in general, than a spinlock. A spinlock > mustn't do anything that could emit an error or abort (among other > things). I doubt that the performance cost of using an LWLock rather > than a spin lock here is enough to matter, and the spin lock seems > more likely to result in hard-to-find bugs. Well, stuck spinlocks aren't exactly hard to identify. But I agree that the lack of any release-on-error infrastructure is a killer reason not to use a spinlock for anything but short straight-line code segments. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] error order when debug postgresql step by step?
HI, I tried to debug PostgreSQL under Redhat with Eclipse. I follew the guide ' Working with Eclipse ', and build the DB successfully. http://wiki.postgresql.org/wiki/Working_with_Eclipse However, when I debug the souce code step by step, I found it run in the wrong order. I think the cause maybe the share library and the source code do not match. I have add MY_PG_INS_DIR/lib to LD_LIBRARY_PATH, but it does't work at all. How can I do to make sure the right excute order when I debug step by step ? Thank you for answering. -- Best Wishes! hom -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
On 01.03.2011 16:40, Nikhil Sontakke wrote: On Tue, Mar 1, 2011 at 10:17 PM, Heikki Linnakangas wrote: On 01.03.2011 12:50, Nikhil Sontakke wrote: Will try to get the call stack if needed. Yes, please. Here is the stack trace: Hmm, it looks like ImmediateInterruptOK is set, while we're busy running a query. How come? Can you debug that? Where does it get set? Ah, this is not exactly an easily reproducible problem :( You gotta be lucky enough to be able to interrupt inside a malloc call. You could put a sleep() just before the malloc(). Even if you can't reproduce a crash, we know that we shouldn't be calling malloc() in any codepath where ImmediateInterruptOK == true. Heck, you can just put an Assert(!ImmediateInterruptOK) there, although it will fire in the authentication phase because of the issue with ClientAuthentication. You can set debug_assertions=off in postgresql.conf and enable it again with SET after logging in to get around that. But adding hold/resume interrrupts in mcxt.c (not aset.c, since we want to be agnostic to the underlying layer) should be good enough to handle this non-re-entrant issue, no? We shouldn't be running with ImmediateInterruptOK == true to begin with. There are many other things beside malloc/free that are not safe to be interrupted like that. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Porting PostgreSQL to DragonFly BSD
Rumko writes: > On Sunday 27. of February 2011 23:50:17 Peter Eisentraut wrote: >> It seems to me that it would be easier to just map dragonfly to the >> freebsd template. > I didn't see a precedence for that kind of introduction of a new platform > (all > others seem to have their own templates), so thought it would've had less > chance of being accepted. When I looked at that patch, I actually couldn't see anything different at all from freebsd. We aren't interested in maintaining 99% duplicate code --- it seems much easier from a maintenance standpoint to introduce an ifdef or two into the freebsd code, rather than deal with a large cut and paste job. Please try it that way and see what you come up with. (It's possible that it'll be so ugly that we agree your original patch is better, but we need to see the results of the experiment.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
Simon Riggs writes: > On Tue, 2011-03-01 at 15:51 +0900, Fujii Masao wrote: >> A spinlock can be used only for very short-term operation like >> read/write of some shared-variables. The operation on the queue >> is not short, so should be protected by LWLock, I think. > There's no need to sleep while holding locks and the operations are very > short in most cases. The code around it isn't trivial, but that's no > reason to use LWlocks. What does "in most cases" mean? > LWlocks are just spinlocks plus sem sleeps, so I don't see the need for > that in the current code. Other views welcome. Simon, that is absolutely NOT acceptable. Spinlocks are to be used only for short straight-line code segments. If the lock has any potential to be held for more than nanoseconds, use an LWLock. The contention costs of the shortcut you propose are too high. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
On Tue, Mar 1, 2011 at 10:17 PM, Heikki Linnakangas wrote: > On 01.03.2011 12:50, Nikhil Sontakke wrote: >>> Will try to get the call stack if needed. >>> >>> Yes, please. >> >> Here is the stack trace: > > Hmm, it looks like ImmediateInterruptOK is set, while we're busy running a > query. How come? Can you debug that? Where does it get set? > Ah, this is not exactly an easily reproducible problem :( You gotta be lucky enough to be able to interrupt inside a malloc call. But adding hold/resume interrrupts in mcxt.c (not aset.c, since we want to be agnostic to the underlying layer) should be good enough to handle this non-re-entrant issue, no? Regards, Nikhils -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
On 01.03.2011 12:50, Nikhil Sontakke wrote: Will try to get the call stack if needed. Yes, please. Here is the stack trace: Hmm, it looks like ImmediateInterruptOK is set, while we're busy running a query. How come? Can you debug that? Where does it get set? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Native XML
On 03/01/2011 08:16 AM, Robert Haas wrote: On Mon, Feb 28, 2011 at 6:54 PM, Andrew Dunstan wrote: There seems to be an almost universal assumption that storing XML in its native form (i.e. a text stream) is going to produce inefficient results. Maybe it will, but I think it needs to be fairly convincingly demonstrated. And then we would have to consider the costs. For example, unless we implemented our own XPath processor to work with our own XML format (do we really want to do that?), to evaluate an XPath expression for a piece of XML we'd actually need to produce the text format from our internal format before passing it to some external library to parse into its internal format and then process the XPath expression. That means we'd actually be making things worse, not better. But this is clearly the sort of processing people want to do - see today's discussion upthread about xpath_table. Well, obviously the only point of having our own internal format is if we have our own xpath processor&c to match. One would think that this would be a lot faster than parsing the string with libxml2 every time we want to xpath it, especially for large documents. But then again, I haven't seen any benchmarks. That would be a huge body of code we'd need to maintain, complex and full of subtleties which, if we weren't deeply invested in the XML standards would bite us, I have no doubt. Now, if someone wanted to start a project that added efficient serialization/de-serialization of libxml2 (or other library) objects so we could avoid constant parsing overhead, that would make lots more sense to me. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On Mon, Feb 28, 2011 at 10:32 PM, Greg Stark wrote: > On Tue, Mar 1, 2011 at 1:43 AM, David Christensen wrote: >> Was this cluster upgraded to 8.4.4 from 8.4.0? It sounds to me like a known >> bug in 8.4.0 which was fixed by this commit: >> > > The reproduction script described was running vacuum repeatedly. A > single vacuum run out to be sufficient to clean up the problem if it > was left-over. > > I wonder if it would help to write a regression test that runs 100 or > so vacuums and see if the bulid farm turns up any examples of this > behaviour. One other thing to keep in mind here is that the warning message we've chosen can be a bit misleading. The warning is: WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation "test" page 1 ...which implies that the state of the tuples is correct, and that the page-level bit is wrong in comparison. But I recently saw a case where the infomask got clobbered, resulting in this warning. The page level bit was correct, at least relative to the intended page contents; it was the a tuple on the page that was screwed up. It might have been better to pick a more neutral phrasing, like "page is marked all-visible but some tuples are not visible". -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
On Tue, Mar 1, 2011 at 3:21 AM, Simon Riggs wrote: > On Tue, 2011-03-01 at 15:51 +0900, Fujii Masao wrote: >> Thanks for update of the patch! >> >> On Tue, Mar 1, 2011 at 3:40 AM, Simon Riggs wrote: >> >> SyncRepRemoveFromQueue seems not to be as short-term as we can >> >> use the spinlock. Instead, LW lock should be used there. >> >> You seem to have forgotten to fix the above-mentioned issue. > > Not forgotten. > >> A spinlock can be used only for very short-term operation like >> read/write of some shared-variables. The operation on the queue >> is not short, so should be protected by LWLock, I think. > > There's no need to sleep while holding locks and the operations are very > short in most cases. The code around it isn't trivial, but that's no > reason to use LWlocks. > > LWlocks are just spinlocks plus sem sleeps, so I don't see the need for > that in the current code. Other views welcome. An LWLock is a lot safer, in general, than a spinlock. A spinlock mustn't do anything that could emit an error or abort (among other things). I doubt that the performance cost of using an LWLock rather than a spin lock here is enough to matter, and the spin lock seems more likely to result in hard-to-find bugs. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Native XML
On Mon, Feb 28, 2011 at 6:54 PM, Andrew Dunstan wrote: > There seems to be an almost universal assumption that storing XML in its > native form (i.e. a text stream) is going to produce inefficient results. > Maybe it will, but I think it needs to be fairly convincingly demonstrated. > And then we would have to consider the costs. For example, unless we > implemented our own XPath processor to work with our own XML format (do we > really want to do that?), to evaluate an XPath expression for a piece of XML > we'd actually need to produce the text format from our internal format > before passing it to some external library to parse into its internal format > and then process the XPath expression. That means we'd actually be making > things worse, not better. But this is clearly the sort of processing people > want to do - see today's discussion upthread about xpath_table. Well, obviously the only point of having our own internal format is if we have our own xpath processor &c to match. One would think that this would be a lot faster than parsing the string with libxml2 every time we want to xpath it, especially for large documents. But then again, I haven't seen any benchmarks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pl/python custom exceptions for SPI
On 28/02/11 19:38, Tom Lane wrote:
> Peter Eisentraut writes:
>> On mån, 2011-02-28 at 12:08 -0500, Tom Lane wrote:
>>> I'm seeing a core dump as well as multiple inconsistencies in error
>>> message spelling in the plpython regression tests on a Fedora 13 box
>>> (python 2.6.4). Several buildfarm critters don't look too happy either.
>
>> Fixed. (Well, some of it. We'll see ...)
>
> Core dump is still there. It appears to be a python assertion failure.
> I installed python's debuginfo and got this backtrace:
>
> Program terminated with signal 6, Aborted.
> #0 0x0032a36328f5 in raise (sig=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> Missing separate debuginfos, use: debuginfo-install
> keyutils-libs-1.2-6.fc12.x86_64 krb5-libs-1.7.1-17.fc13.x86_64
> libcom_err-1.41.10-7.fc13.x86_64 libselinux-2.0.94-2.fc13.x86_64
> openssl-1.0.0c-1.fc13.x86_64 zlib-1.2.3-23.fc12.x86_64
> (gdb) bt
> #0 0x0032a36328f5 in raise (sig=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0032a36340d5 in abort () at abort.c:92
> #2 0x0032a362b8b5 in __assert_fail (assertion=0x32a5b46391
> "gc->gc.gc_refs != 0", file=, line=277, function= optimized out>)
> at assert.c:81
> #3 0x0032a5b0853e in visit_decref (op=,
> data=) at Modules/gcmodule.c:277
> #4 0x0032a5a7cbd9 in dict_traverse (op=
> {'info': , 'notice': ,
> 'Fatal': , 'log': ,
> 'prepare': , 'spiexceptions': 0x7f11c3666d38>, 'SPIError': , 'Error': remote 0x1bba300>, 'execute': , '__package__':
> None, 'quote_ident': , 'warning': function warning>, 'subtransaction': ,
> 'quote_literal': , 'quote_nullable':
> , 'error': ,
> 'debug': , '__name__': 'plpy', 'fatal': function fatal>, '__doc__': None}, visit=0x32a5b084c0 , arg=0x0)
> at Objects/dictobject.c:2003
> [...]
> #24 0x0032a5af22c4 in PyImport_ImportModuleLevel (name=0x7f11c40c2084
> "string", globals=
>
> Don't know python enough to do anything useful with this, but the
> reference to "gc_refs" sure makes it look like something is dropping the
> ball on when to do INCREF/DECREF.
That's strange, the error occurs while trying to import the "string"
module. But the error itself seems to be caused by trying to unref the
spiexceptions module (showing up here as ). Apparently adding spiexceptions as an object to the
plpy module is not done exactly right.
I'll try to reproduce it in my environment.
Cheers,
Jan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PG signal handler and non-reentrant malloc/free calls
>
>> Will try to get the call stack if needed.
>
> Yes, please.
>
Here is the stack trace:
#0 0xe410 in __kernel_vsyscall ()
#1 0xb7ee676e in __lll_mutex_lock_wait () from /lib/libc.so.6
#2 0xb7e82e41 in _L_lock_4214 () from /lib/libc.so.6
#3 0xb7e80048 in free () from /lib/libc.so.6
#4 0x082f70b1 in AllocSetDelete (context=0x84c7d68) at aset.c:503
#5 0x082f75b2 in MemoryContextDelete (context=0x84c7d68) at mcxt.c:196
#6 0x082f75e9 in MemoryContextDeleteChildren (context=0x84c7ce0) at mcxt.c:215
#7 0x082f7582 in MemoryContextDelete (context=0x84c7ce0) at mcxt.c:169
#8 0x082f75e9 in MemoryContextDeleteChildren (context=0x84c7bd0) at mcxt.c:215
#9 0x082f7582 in MemoryContextDelete (context=0x84c7bd0) at mcxt.c:169
#10 0x080b54fd in CleanupSubTransaction () at xact.c:1444
#11 0x080b5590 in AbortOutOfAnyTransaction () at xact.c:3955
#12 0x082e9b8b in ShutdownPostgres (code=1, arg=0) at postinit.c:655
#13 0x08220f95 in shmem_exit (code=1) at ipc.c:191
#14 0x08221051 in proc_exit (code=1) at ipc.c:119
#15 0x082dd0cd in errfinish (dummy=0) at elog.c:475
#16 0x08231905 in ProcessInterrupts () at postgres.c:2869
#17 0x082dd071 in errfinish (dummy=0) at elog.c:500
#18 0x08231acd in die (postgres_signal_arg=15) at postgres.c:2732
#19
#20 0xb7e8091c in _int_malloc () from /lib/libc.so.6
#21 0xb7e822c6 in malloc () from /lib/libc.so.6
#22 0x082f6bd3 in AllocSetAlloc (context=0x84c6e18, size=20263) at aset.c:533
#23 0x0829ff36 in textout (fcinfo=0xbfa9cba0) at varlena.c:491
#24 0x082e03c2 in FunctionCall1 (flinfo=0xbfa9d0dc, arg1=139801232) at
fmgr.c:1272
#25 0x082e1495 in OutputFunctionCall (flinfo=0xbfa9d0dc,
val=139801232) at fmgr.c:1905
#26 0x082e2478 in OidOutputFunctionCall (functionId=47, val=139801232)
at fmgr.c:2008
#27 0x6f3aab9e in convert_value_to_string (value=139801232,
valtype=) at pl_exec.c:5304
#28 0x6f3aac49 in exec_cast_value (value=139801232, valtype=25,
reqtype=1043, reqinput=0x84ff1b8, reqtypioparam=1043,
reqtypmod=14, isnull=0 '\0') at pl_exec.c:5346
#29 0x6f3ac0b5 in exec_assign_value (estate=0xbfa9ddc4,
target=0x84fbd50, value=139801232, valtype=25,
isNull=0xbfa9d26f "") at pl_exec.c:4130
#30 0x6f3ad7de in exec_assign_expr (estate=0xbfa9ddc4,
target=0x84fbd50, expr=0x8500cb0) at pl_exec.c:4102
#31 0x6f3afc99 in exec_stmts (estate=0xbfa9ddc4, stmts=) at pl_exec.c:1483
#32 0x6f3b09de in exec_stmt_fori (estate=0xbfa9ddc4, stmt=0x8500b48)
at pl_exec.c:1891
#33 0x6f3afc24 in exec_stmts (estate=0xbfa9ddc4, stmts=) at pl_exec.c:1381
#34 0x6f3b0dd4 in exec_stmt_loop (estate=0xbfa9ddc4, stmt=0x8500978)
at pl_exec.c:1681
---Type to continue, or q to quit---
#35 0x6f3afc0c in exec_stmts (estate=0xbfa9ddc4, stmts=) at pl_exec.c:1373
#36 0x6f3b10f2 in exec_stmt_block (estate=0xbfa9ddc4, block=0x8500f20)
at pl_exec.c:1241
#37 0x6f3af301 in exec_stmts (estate=0xbfa9ddc4, stmts=) at pl_exec.c:1349
#38 0x6f3b1635 in exec_stmt_block (estate=0xbfa9ddc4, block=0x85006c0)
at pl_exec.c:1070
#39 0x6f3af301 in exec_stmts (estate=0xbfa9ddc4, stmts=) at pl_exec.c:1349
#40 0x6f3b10f2 in exec_stmt_block (estate=0xbfa9ddc4, block=0x84fff58)
at pl_exec.c:1241
#41 0x6f3b22c5 in plpgsql_exec_function (func=0x84c6f88,
fcinfo=0xbfa9dfd0) at pl_exec.c:334
#42 0x6f3a5d8c in plpgsql_call_handler (fcinfo=0xbfa9dfd0) at pl_handler.c:112
#43 0x08185119 in ExecMakeTableFunctionResult (funcexpr=0x84f52c0,
econtext=0x84f50d0, expectedDesc=0x84f5198,
returnDesc=0xbfa9e548) at execQual.c:1651
#44 0x081924d0 in FunctionNext (node=0x84f5048) at nodeFunctionscan.c:68
#45 0x08187c34 in ExecScan (node=0x84f5048, accessMtd=0x8192460
) at execScan.c:68
#46 0x08192459 in ExecFunctionScan (node=0x84f5048) at nodeFunctionscan.c:119
#47 0x08180f57 in ExecProcNode (node=0x84f5048) at execProcnode.c:367
#48 0x08180080 in ExecutorRun (queryDesc=0x84dc640,
direction=ForwardScanDirection, count=0) at execMain.c:1335
#49 0x082369c0 in PortalRunSelect (portal=0x84d7578, forward=, count=0, dest=0x84ced18)
at pquery.c:943
#50 0x082379ba in PortalRun (portal=0x84d7578, count=2147483647,
isTopLevel=1 '\001', dest=0x84ced18,
altdest=0x84ced18, completionTag=0xbfa9e7da "") at pquery.c:797
#51 0x082323d3 in exec_simple_query (query_string=0x84cdd28 " call
testsp2('testtab_2', 1000);") at postgres.c:1074
#52 0x082345ef in PostgresMain (argc=4, argv=0x8434598,
username=0x8434550 "sys") at postgres.c:4081
#53 0x081eef68 in PostmasterMain (argc=1, argv=0x8431cc8) at postmaster.c:4191
#54 0x081a3560 in main (argc=1, argv=0x8431cc8) at main.c:188
(gdb)
Regards,
Nikhils
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
On 2011-02-28 20:32, Simon Riggs wrote: I have reworded it to see if that improves the explanation Code available at git://github.com/simon2ndQuadrant/postgres.git If a standby is removed from the list of servers then it will stop being the synchronous standby, allowing another to take it's place. How can I see at the master which one is the synchronous standby? It would be nice if it was an extra attribute in the pg_stat_replication view. As part of understanding syncrep better, we're going to work on that. regards, Yeb Havinga -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
On Tue, Mar 1, 2011 at 5:29 PM, Simon Riggs wrote: >> >> What if fast shutdown is requested while RecordTransactionCommit >> >> is waiting in SyncRepWaitForLSN? ISTM fast shutdown cannot complete >> >> until replication has been successfully done (i.e., until at least one >> >> synchronous standby has connected to the master especially if >> >> allow_standalone_primary is disabled). Is this OK? >> > >> > A "behaviour" - important, though needs further discussion. >> >> One of the scenarios which I'm concerned is: >> >> 1. The primary is running with allow_standalone_primary = on. >> 2. While some backends are waiting for replication, the user requests >> fast shutdown. >> 3. Since the timeout expires, those backends stop waiting and return the >> success >> indication to the client (but not replicated to the standby). >> 4. Since there is no backend waiting for replication, fast shutdown >> completes. >> 5. The clusterware like pacemaker detects the death of the primary and >> triggers the >> failover. >> 6. New primary doesn't have some transactions committed to the client, i.e., >> transaction lost happens!! >> >> To avoid such a transaction lost, we should prevent the primary from >> returning the >> success indication to the client while fast shutdown is being executed, even >> if >> allow_standalone_primary is enabled, I think. Thought? > > Walsenders don't shutdown until after they have sent the shutdown > checkpoint. > > We could make them wait for the reply from the standby at that point. Right. But what if the replication connection is closed before #3? In this case, obviously walsender cannot send WAL up to the shutdown checkpoint. > I'll think some more, not convinced yet. Thanks! I'll think more, too, to make sync rep more reliable! Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v17
On Tue, Mar 1, 2011 at 4:56 PM, Simon Riggs wrote: >> > If allow_standalone_primary = on then we sit and wait until we hit >> > client timeout, which occurs even after last standby has gone. >> >> In that case, why do backends need to wait until the timeout occurs? >> We can make those backends resume their transaction as soon as >> the last standby has gone. No? > > The guarantee provided is that we will wait for up to client timeout for > the sync standby to confirm. If we stop waiting right at the point that > an "event" occurs, it breaks the whole purpose of the feature. ISTM that at least people (including me) who want to use synchronous replication for high-availability don't like this behavior. Because, when there is one synchronous standby and it's shut down, the transactions would get blocked until the timeout happens. The system down time gets longer. Of course, if walsender doesn't detect the termination of replication connection, I can't complain that transactions wait until the timeout happens. But, otherwise, releasing the waiting transactions would be help for high-availability, I think. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Porting PostgreSQL to DragonFly BSD
On Sunday 27. of February 2011 23:50:17 Peter Eisentraut wrote: > On sön, 2011-02-27 at 21:19 +0100, Rumko wrote: > > The attached patch (also available at > > http://www.rumko.net/0001-DragonFly-BSD-support.patch ) applies cleanly > > to the master branch and can be cherry-picked to REL9_0_STABLE branch > > without conflicts. > > It seems to me that it would be easier to just map dragonfly to the > freebsd template. I didn't see a precedence for that kind of introduction of a new platform (all others seem to have their own templates), so thought it would've had less chance of being accepted. Is it preferable to have it linked instead of having it separate like in the current patch? -- Regards, Rumko signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] Sync Rep v17
On Tue, Mar 1, 2011 at 5:21 PM, Simon Riggs wrote: >> A spinlock can be used only for very short-term operation like >> read/write of some shared-variables. The operation on the queue >> is not short, so should be protected by LWLock, I think. > > There's no need to sleep while holding locks and the operations are very > short in most cases. The code around it isn't trivial, but that's no > reason to use LWlocks. > > LWlocks are just spinlocks plus sem sleeps, so I don't see the need for > that in the current code. Other views welcome. The following description in in src/backend/storage/lmgr/README leads me to further think that the spinlock should not be used there. Because, in the patch, while the spinlock is being held, whole of the waiting list (if there are many concurrent running transactions, this might be large) can be searched, SetLatch can be called and elog(WARNING) can be called (though this should not happen). * Spinlocks. These are intended for *very* short-term locks. If a lock is to be held more than a few dozen instructions, or across any sort of kernel call (or even a call to a nontrivial subroutine), don't use a spinlock. Spinlocks are primarily used as infrastructure for lightweight locks. They are implemented using a hardware atomic-test-and-set instruction, if available. Waiting processes busy-loop until they can get the lock. There is no provision for deadlock detection, automatic release on error, or any other nicety. There is a timeout if the lock cannot be gotten after a minute or so (which is approximately forever in comparison to the intended lock hold time, so this is certainly an error condition). Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why our counters need to be time-based WAS: WIP: cross column correlation ...
Tom Lane writes: > The ideal solution would likely be for the stats collector to expose its > data structures as shared memory, but I don't think we get to do that > under SysV shmem --- it doesn't like variable-size shmem much. Maybe > that's another argument for looking harder into mmap or POSIX shmem, > although it's not clear to me how well either of those fixes that. We could certainly use message passing style atop pgpipe.c here, right? After all we already have a protocol and know how to represent complex data structure in there, and all components of PostgreSQL should be able to alleviate this, I'd think. Or this fever ain't really gone yet :) Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
