Re: [HACKERS] write ahead logging in standby (streaming replication)
Hi, On Thu, Nov 12, 2009 at 4:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: The problem is that fsync needs to be issued too frequently, which would be harmless in asynchronous replication, but not in synchronous one. A transaction would have to wait for the primary's and standby's fsync before returning a success to a client. So I'm inclined to change the startup process and bgwriter, instead of walreceiver, so as to fsync the WAL for the WAL rule. Let's keep it simple for now. Just make the walreceiver do the fsync. We can optimize later. For now, we're only going to have async mode anyway. Okey, I'll do that; the walreceiver issues the fsync for each arrival of the WAL records, and the startup process replays only the records already fsynced. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
On Thu, 2009-11-12 at 17:03 +0900, Fujii Masao wrote: On Thu, Nov 12, 2009 at 4:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: The problem is that fsync needs to be issued too frequently, which would be harmless in asynchronous replication, but not in synchronous one. A transaction would have to wait for the primary's and standby's fsync before returning a success to a client. So I'm inclined to change the startup process and bgwriter, instead of walreceiver, so as to fsync the WAL for the WAL rule. Let's keep it simple for now. Just make the walreceiver do the fsync. We can optimize later. For now, we're only going to have async mode anyway. Okey, I'll do that; the walreceiver issues the fsync for each arrival of the WAL records, and the startup process replays only the records already fsynced. I agree with you, though it has taken some time to understand what you said and at first my reaction was to disagree. I think the responses you got on this are because you dived straight in with a question before explaining other things around this. We already have a number of options for how to handle incoming WAL. We can choose to fsync or not when WAL arrives. Choosing *not* to fsync would be the typical choice because it provides reasonable performance; fsyncing after each transaction commit would be worse. In any case, if WAL receiver does the fsyncs then we will get worse performance. If we reduce the number of fsyncs it does we just get spiky behaviour around the fsyncs. If recovery starts reading WAL records that have not been fsynced then we may need to flush a shared buffer to disk that depends upon a non-fsynced(yet) WAL record. Fsyncing WAL after *every* WAL record is going to make performance suck even worse and is completely out of the question. So implementing the fsync-WAL-before-buffer-flush rule during recovery makes much more sense. It's also only small change during XlogFlush(). Another way of doing this would be to only allow recovery to progress as far as has been fsynced. That seems a more plausible approach, but would lead to delays if we had a small number of long write transactions. The benefit of streaming is that it potentially allows us to keep as near to real-time recovery as possible. So overall, yes, we need to do as you suggested: implement WAL rule in recovery. WALreceiver smoothly does write(), Startup replays and we leave the WAL file fsyncs to be performed by the bgwriter. But I also agree with Heikki. Let's plan to do this later in this release. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Sun, 2009-11-08 at 20:52 -0500, Robert Haas wrote: I would personally prefer not to be involved in the management of the next CommitFest. Having done all of the July CommitFest and a good chunk of the September CommitFest, I am feeling a bit burned out. You did a grand job and everybody appreciates it. I feel like Simon Riggs and Fujii Masao really pulled out all the stops to get these ready in time for the September CommitFest, and while I'm not in a hurry to break the world, I think the sooner these can hit the tree, the better of we'll be in terms of releasing 8.5. Sprinting is hard and we all need to rest afterwards. How about we just slow the pace down a little? Nobody wants you to quit, we just need to set a sustainable pace. Looking at the submissions so far, it seems you've done such a grand job at clearing the backlog that there are few patches in the next fest. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Alvaro Herrera írta: Boszormenyi Zoltan escribió: I have split up (and cleaned up a little) the dynamic cursorname patch into smaller logical, easier-to-review pieces. Descriptions below. 1) 1a-unified-optfromin-fetch-ctxdiff.patch ecpg supports optional FROM/IN in FETCH and MOVE statements (mainly because it's required by Informix-compatibility). Unify core and ecpg grammar as per Tom Lane's suggestion. I have applied this patch after some tinkering. I mainly added support for fetch_args: FORWARD opt_from_in name and BACKWARD opt_from_in name in ecpg.addons which apparently you forgot. Thanks. Your fix is correct if this patch is considered standalone. This means I have to re-post the later patches to fix the reject this fix causes in them. Best regards, Zoltán Böszörményi -- Bible has answers for everything. Proof: But let your communication be, Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil. (Matthew 5:37) - basics of digital technology. May your kingdom come - superficial description of plate tectonics -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Alvaro Herrera írta: Boszormenyi Zoltan escribió: 2) 1b-cursor_name-ctxdiff.patch name - cursor_name transition in core grammar and ecpg grammar. Currently it does nothing, it's a preparation phase. Depends on patch 1. Applied this part too. I cannot apply the other ones -- they belong to the ECPG maintainer. I hope I cleared his road a bit. Thanks and best regards, Zoltán Böszörményi -- Bible has answers for everything. Proof: But let your communication be, Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil. (Matthew 5:37) - basics of digital technology. May your kingdom come - superficial description of plate tectonics -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 2:12 AM, Andrew Gierth and...@tao11.riddles.org.uk wrote: Does it cope with the case where a trigger is doing NOTIFY, and you do a whole-table update, therefore dumping potentially millions of notifications in at once? (for example a rare maintenance operation on a table which has a listen/notify arrangement triggered by single inserts or updates) The existing implementation copes with that just fine. As long as you use the existing way to send out notifications by just sending NOTIFY foo, then yes, it will cope with millions of them just fine because it will collapse them into one notification as does the current implementation (contrary to the current implementation a notification will be received from every transaction that has sent one - the current implementation even collapses notifications from different transactions). However if you decide to use the new syntax and add a distinct payload to every NOTIFY, then you also send out millions of notifications... With the proposed patch the queue has space for up to about 81,787,680 notifications. The limits are mainly imposed by slru.c which defines page size, pages per segment and the number of segments available. We could easily bump that up to a multiple of the current limits if we have decided that we need to... Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Bruce Momjian br...@momjian.us writes: Peter Eisentraut wrote: discernible benefits. But you can't expect a lot of people or employers to reserve time on top of that for handholding other people's patches and for other community stuff that has no easy to measure benefits. Totally agree. It is that zero-return work that is hard to justify for people and companies. It is clearly something that requires self-sacrifice, and personally I think a culture of self-sacrifice is what has given us such great success and such a nuturing community environment. I don't know what you're talking about here, or refering to. The easy way to explain Open Source contributing to PHB and finance people, in my mind, is the following: Open Source is not free as in free beer. At all. Rather than paying licences costs, what you have to pay for is your employees salary when they take the time to participate into this community things. And if you're using Open Source Software, you *need* employees that take part of the community processes. That also mean you have real experts now, and ones able to drive the project in a direction that will be to your profit as an employer. For example, this hairy bug in your production environment will get fixed the day you report about it, rather that when the licencing company think it matches their roadmap, if ever. Oh and some other niceties about it. Let's try the story telling style... When I worked on preprepare, I though we had an urgent need here to run it. It happened that we had yet more urgent to do, so I barely had time to have it working and commit it on some pgfoundry CVS. Less than a week after, I'm told on IRC that the product is running fine in a production environment somewhere, taking real load. Now I don't have to make tests before deploying. That has been done for me. That's Open Source. Regards, -- dim Is it 2009 and we're still having this discussion about how participating into Open Source communities is a good way to spend your money as an employer? Wow. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parsing config files in a directory
Hi, Josh Berkus j...@agliodbs.com writes: Let's NOT start that discussion again. Don't worry, no aim here to. Overall, I'm seeing this patch proposal suffer from an extreme excess of bike-shedding. Not only that. See above. My proposal is this: (1) that we support the idea of a patch which allows people to add directory includes to postgresql.conf, in the same manner that we now support file includes, with files in the included directory to be processed alphanumerically. +1 (2) that we put out a TODO for making the configuration variables which take lists able to take an accumulator as well as an assignment, syntax TBA. +1 These two above seem like nice, small incremental changes to 8.5 which require no sweeping redesigns of how people handle conf files, but do allow people who want to develop new management strategies to do so. +1 What made us have this long a thread is the premise that having the include directory facility will make any good to solving the problem of editing the configuration from a program. So everyone tried either to explain how it helps or solves it, or to explan how it does not help at all. My position is the later, as I hope to have made clear before. But you're very right when saying that this facility is worth it independantly of how much it helps solving the programatic API to configuration. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 4:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: One possible solution would be to write to the queue before committing and adding the TransactionID. Then other backends can check if our TransactionID has successfully committed or not. Not sure if this is worth the overhead however... That sounds reasonable, and it's certainly no more overhead than the tuple visibility checks that happen in the current implementation. I am not too concerned about the runtime of the visibility checks, instead I suppose that most of the overhead will come from waiting for another transaction to either commit or abort... If transaction t1 scans the queue and at some point finds notifications from t2, then it will ask for the status of t2. If it finds out that t2 is still running, then it has no other option than to stop working on the queue and wait (it will be signalled again later once t2 has finished). This also means that we cannot at the same time write notifications to the queue and read from it and if a transaction places a few million notifications into the queue, readers need to wait until it has finished and only after that they can continue and read the notifications... And it means that if the queue is full, we might run into a deadlock... A transaction adding notifications will wait for the readers to proceed and the readers wait for the transaction to commit or abort... One option could be to write new notifications to a subdirectory, and create a bunch of new segment files there. Once this is done, the segment files could be moved over and renamed, so that they continue the slru queue... If we run out of disk space while filling that temporary subdirectory, then we can just delete the subdirectory and nobody has been blocked. We could still run into errors moving and renaming the segment files (e.g. permission problems) so that we still might need to abort the transaction... 2. The payload parameter is optional. A notifying client can either call NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is currently limited to 128 characters... Not sure if we should allow longer payload strings... Might be a good idea to make the max the same as the max length for prepared transaction GUIDs? Not sure anyone would be shipping those around, but it's a pre-existing limit of about the same size. Yes, sounds reasonable to have the same limit for user-defined identifiers... Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 5:04 AM, Simon Riggs si...@2ndquadrant.com wrote: On Sun, 2009-11-08 at 20:52 -0500, Robert Haas wrote: I would personally prefer not to be involved in the management of the next CommitFest. Having done all of the July CommitFest and a good chunk of the September CommitFest, I am feeling a bit burned out. You did a grand job and everybody appreciates it. Thanks. I feel like Simon Riggs and Fujii Masao really pulled out all the stops to get these ready in time for the September CommitFest, and while I'm not in a hurry to break the world, I think the sooner these can hit the tree, the better of we'll be in terms of releasing 8.5. Sprinting is hard and we all need to rest afterwards. How about we just slow the pace down a little? Nobody wants you to quit, we just need to set a sustainable pace. I'm not sure exactly what point you're aiming at here, so I'll respond with a few thoughts that may or may not pertain. I think it would be really, really good if we could make this release come out on the schedule previously discussed. 8.4 slipped quite a bit for reasons that were, IMHO, quite preventable: and, worse, it's not clear that the slippage really bought us anything, because we still ended up with a bunch of embarassing bugs. Having said that, I'm not capable of single-handedly effecting an on-time release, and I don't particularly want to. In a community where people can disappear or change roles in the snap of a finger, it's bad to be relying on any one person to do too much. We need larger, more robust pools of committers, reviewers, commitfest managers, etc. Perhaps for next release we should consider spacing the CommitFests out a little more. I think one CommitFest every 2 months is a little too tight a schedule. As Peter and others have mentioned previously, it doesn't leave a lot of time to work on your own patches (behold the lack of any of my patches in this CommitFest). I think a CommitFest every 3 months would be too long, but maybe something in the middle. The trick is to navigate around major holidays while ending around the right time. Possibly the amount of time between CommitFests doesn't even need to be constant throughout the release cycle - maybe shorter at the beginning and longer towards the end. Looking at the submissions so far, it seems you've done such a grand job at clearing the backlog that there are few patches in the next fest. Thanks for your kind words. It does seem that most of the major patches we've seen so far landed last CommitFest, with the exception of HS and SR. That's not all me, of course - among other people, Tom did a tremendous amount of work - but I'm glad I was able to help move it along. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Hi, On Thursday 12 November 2009 12:46:46 Robert Haas wrote: Perhaps for next release we should consider spacing the CommitFests out a little more. That may lead to quite a bit frustration on the contributor side though. It can be very frustrating to have no input for a even longer timeframe... Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 6:49 AM, Andres Freund and...@anarazel.de wrote: On Thursday 12 November 2009 12:46:46 Robert Haas wrote: Perhaps for next release we should consider spacing the CommitFests out a little more. That may lead to quite a bit frustration on the contributor side though. It can be very frustrating to have no input for a even longer timeframe... Yep, that's the flip side. But you'll notice I was suggesting extending it by a pretty small amount. Anyway, it was just a thought. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
On Thu, Nov 12, 2009 at 6:27 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with you, though it has taken some time to understand what you said and at first my reaction was to disagree. I think the responses you got on this are because you dived straight in with a question before explaining other things around this. Thanks for clarifying this topic ;) If recovery starts reading WAL records that have not been fsynced then we may need to flush a shared buffer to disk that depends upon a non-fsynced(yet) WAL record. Fsyncing WAL after *every* WAL record is going to make performance suck even worse and is completely out of the question. So implementing the fsync-WAL-before-buffer-flush rule during recovery makes much more sense. It's also only small change during XlogFlush(). Agreed. This approach has lesser impact on the performance. But, as I said on my first post on this thread, even such low-frequent fsync-WAL-before-buffer-flush might cause a response time spike on the primary because the walreceiver must sleep during that fsync. I think that leaving the WAL-logging business to another process like walwriter is a good idea for reducing further the impact on the walreceiver; In typical case, * The walreceiver receives WAL records, returns the ACK to the primary, saves them in the wal_buffers, and lets the startup process know the arrival. * The walwriter writes and fsyncs the WAL records in the wal_buffers. * The startup process applies the WAL records in the wal_buffers when it receives the notice of the arrival. * The startup process and bgwriter fsyncs the WAL before the buffer flush. Of course, since this approach is too complicated, it's out of the scope of the development for v8.5. But I also agree with Heikki. Let's plan to do this later in this release. Okey. I implement nothing around this topic until the core part of asynchronous replication will have been committed. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
On Thu, 2009-11-12 at 21:45 +0900, Fujii Masao wrote: But, as I said on my first post on this thread, even such low-frequent fsync-WAL-before-buffer-flush might cause a response time spike on the primary because the walreceiver must sleep during that fsync. I think that leaving the WAL-logging business to another process like walwriter is a good idea for reducing further the impact on the walreceiver; In typical case, Agree completely. Of course, since this approach is too complicated, it's out of the scope of the development for v8.5. It's out of scope for phase 1, certainly. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New VACUUM FULL
Itagaki Takahiro wrote: Here is a patch to support rewrite version of VACUUM FULL. It was called VACUUM REWRITE in the past disucussin, but I choose the following syntax for now: VACUUM ( FULL [ INPLACE | REPLACE ] ) [ table_name ] The reason is to contrast the new REPLACE behavior with the old INPLACE behavior. I cannot find any good terms of opposite meaning of REWRITE. I thought the idea is to rip out the current implementation altogether. If that's the case, then it doesn't make sense to use a different syntax. Just rip the innards of VACUUM FULL and replace them with your new implementation. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New VACUUM FULL
Itagaki Takahiro wrote: We still need traditional VACUUM FULL behavior for system catalog because we cannot change relfilenode for them. Also, VACUUM FULL REPLACE is not always better than traditional VACUUM FULL; the new version requires additional disk space and might be slower if we have a few dead tuples. Tom was saying that we could fix the problem that relfilenode could not be changed by having a flat file filenode map. It would only be needed for nailed system catalogs (the rest of the tables grab their relfilenode from pg_class as usual) so it wouldn't have the problems that the previous flatfiles had (which was that they could grow too much). I don't recall if this got implemented (I don't think it did). As for it being slower with few dead tuples, I don't think this is a problem -- just use lazy vacuum in that case. I also remember we agreed on something about the need for extra disk space, but I can't remember what it was. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New VACUUM FULL
BTW I think the vacstmt.options change, which seems a reasonable idea to me, could be extracted from the patch and applied separately. That'd reduce the size of the patch somewhat. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
2. The payload parameter is optional. A notifying client can either call NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is currently limited to 128 characters... Not sure if we should allow longer payload strings... Might be a good idea to make the max the same as the max length for prepared transaction GUIDs? Not sure anyone would be shipping those around, but it's a pre-existing limit of about the same size. Yes, sounds reasonable to have the same limit for user-defined identifiers... [..begging..] Can this be increased significantly? I don't get it, is there any technical reason to make the limit soo small? This drastically reduces the usefulness of the payload. I've wanted this feature for quite sometime and it is quite disappointing that I could not even use it because it is unjustifiably limited. One use case I need is making the payload an absolute path, which saves us a round trip (commonly internet latency) and a query in a section of the system that's extremely performance sensitive. That sure ain't going to fit in 128 bytes. I'm sure I'm not the only one who finds this limit too small. I can almost guarentee complaints would come in if released that way. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] not logging caught exceptions
On ons, 2009-11-11 at 19:45 -0500, Tom Lane wrote: try insert catch unique_constraint_violation update end try this will end up cluttering the logs with all the constraint violation messages. Really? It's not supposed to. There might be a different bug here. This doesn't look right: CREATE LANGUAGE plpgsql; CREATE TABLE keytest (a int PRIMARY KEY, b text); CREATE OR REPLACE FUNCTION insert_or_update(new_a int, new_b text) RETURNS text LANGUAGE plpgsql AS $$ BEGIN INSERT INTO keytest VALUES (new_a, new_b); RETURN 'inserted'; EXCEPTION WHEN integrity_constraint_violation THEN UPDATE keytest SET a = new_a, b = new_b; RETURN 'updated'; END; $$; SELECT insert_or_update(1, 'one'); SELECT insert_or_update(1, 'one'); SELECT insert_or_update(2, 'two'); SELECT insert_or_update(2, 'two'); Results in: insert_or_update -- inserted (1 row) insert_or_update -- updated (1 row) insert_or_update -- inserted (1 row) ERROR: duplicate key value violates unique constraint keytest_pkey CONTEXT: SQL statement UPDATE keytest SET a = $1 , b = $2 PL/pgSQL function insert_or_update line 6 at SQL statement -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 8:25 AM, Andrew Chernow a...@esilo.com wrote: 2. The payload parameter is optional. A notifying client can either call NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is currently limited to 128 characters... Not sure if we should allow longer payload strings... Might be a good idea to make the max the same as the max length for prepared transaction GUIDs? Not sure anyone would be shipping those around, but it's a pre-existing limit of about the same size. Yes, sounds reasonable to have the same limit for user-defined identifiers... [..begging..] Can this be increased significantly? I don't get it, is there any technical reason to make the limit soo small? This drastically reduces the usefulness of the payload. I've wanted this feature for quite sometime and it is quite disappointing that I could not even use it because it is unjustifiably limited. +1 What advantage is there in limiting it to a tiny size? This is a 'payload' after all...an arbitrary data block. Looking at the patch I noticed the payload structure (AsyncQueueEntry) is fixed length and designed to lay into QUEUE_PAGESIZE (set to) BLCKSZ sized pages. Couple of questions: *) is BLCKSZ a hard requirement, that is, coming from the slru implementation, or can QUEUE_PAGESIZE be bumped independently of block size. *) why not make the AsyncQueueEntry divide evenly into BLCKSZ, that is, make the whole structure a size that is a multiple of two? (this would make the payload length 'weird') *) is there any downside you see to making the AsyncQueueEntry structure exactly BLCKSZ bytes in size? Are we worried about the downsides of spinning the notifications out to disk? *) Is a variable length AsyncQueueEntry possible? (presumably bounded by the max page size). Or does complicate the implementation too much? merlin merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] not logging caught exceptions
Peter Eisentraut pete...@gmx.net writes: There might be a different bug here. This doesn't look right: The UPDATE lacks a WHERE clause :-( regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
Fujii Masao masao.fu...@gmail.com writes: The problem is that fsync needs to be issued too frequently, which would be harmless in asynchronous replication, but not in synchronous one. A transaction would have to wait for the primary's and standby's fsync before returning a success to a client. Surely that is exactly what is *required* if the user has asked for synchronous replication. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] NULL-handling in aggregate(DISTINCT ...)
Andrew Gierth and...@tao11.riddles.org.uk writes: A followup question: currently the code uses the datum interface for tuplesort. Obviously with multiple columns the slot interface is used instead; but is there any performance advantage for staying with the datum interface for the single-column case? No idea ... measure it and see. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
What advantage is there in limiting it to a tiny size? This is a 'payload' after all...an arbitrary data block. Looking at the patch I noticed the payload structure (AsyncQueueEntry) is fixed length and designed to lay into QUEUE_PAGESIZE (set to) BLCKSZ sized pages. H. Looks like the limitation comes from slru. The true payload limit is (8K - struct members) the way this is implemented. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plperl and inline functions -- first draft
On Fri, Nov 06, 2009 at 09:53:20PM -0500, Andrew Dunstan wrote:
Joshua Tolley wrote:
I looked through the
regression tests and didn't find any that used plperl -- should we add one
for
this (or for this and all kinds of other stuff)? Is there some way to make
running the regression test conditional on having built --with-perl in the
first place?
Look in src/pl/plperl/{sql,expected}
cheers
andrew
FWIW, I've added this to the upcoming commitfest page.
https://commitfest.postgresql.org/action/patch_view?id=206
--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com
signature.asc
Description: Digital signature
Re: [HACKERS] PL/Python array support
On ons, 2009-11-04 at 16:02 +0200, Peter Eisentraut wrote:
Here is a patch to support arrays in PL/Python as parameters and
return values.
Slightly updated version with fixed reference counting.
diff --git a/src/pl/plpython/expected/plpython_types.out b/src/pl/plpython/expected/plpython_types.out
index 2dd498c..cbc93a2 100644
--- a/src/pl/plpython/expected/plpython_types.out
+++ b/src/pl/plpython/expected/plpython_types.out
@@ -477,3 +477,106 @@ CONTEXT: PL/Python function test_type_conversion_bytea10
ERROR: value for domain bytea10 violates check constraint bytea10_check
CONTEXT: while creating return value
PL/Python function test_type_conversion_bytea10
+--
+-- Arrays
+--
+CREATE FUNCTION test_type_conversion_array_int4(x int4[]) RETURNS int4[] AS $$
+plpy.info(x, type(x))
+return x
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_int4(ARRAY[0, 100]);
+INFO: ([0, 100], type 'list')
+CONTEXT: PL/Python function test_type_conversion_array_int4
+ test_type_conversion_array_int4
+-
+ {0,100}
+(1 row)
+
+SELECT * FROM test_type_conversion_array_int4(ARRAY[0,-100,55]);
+INFO: ([0, -100, 55], type 'list')
+CONTEXT: PL/Python function test_type_conversion_array_int4
+ test_type_conversion_array_int4
+-
+ {0,-100,55}
+(1 row)
+
+SELECT * FROM test_type_conversion_array_int4(ARRAY[NULL,1]);
+INFO: ([None, 1], type 'list')
+CONTEXT: PL/Python function test_type_conversion_array_int4
+ test_type_conversion_array_int4
+-
+ {NULL,1}
+(1 row)
+
+SELECT * FROM test_type_conversion_array_int4(ARRAY[]::integer[]);
+INFO: ([], type 'list')
+CONTEXT: PL/Python function test_type_conversion_array_int4
+ test_type_conversion_array_int4
+-
+ {}
+(1 row)
+
+SELECT * FROM test_type_conversion_array_int4(NULL);
+INFO: (None, type 'NoneType')
+CONTEXT: PL/Python function test_type_conversion_array_int4
+ test_type_conversion_array_int4
+-
+
+(1 row)
+
+SELECT * FROM test_type_conversion_array_int4(ARRAY[[1,2,3],[4,5,6]]);
+ERROR: cannot convert multidimensional array to Python list
+DETAIL: PL/Python only supports one-dimensional arrays.
+CONTEXT: PL/Python function test_type_conversion_array_int4
+CREATE FUNCTION test_type_conversion_array_bytea(x bytea[]) RETURNS bytea[] AS $$
+plpy.info(x, type(x))
+return x
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_bytea(ARRAY[E'\\xdeadbeef'::bytea, NULL]);
+INFO: (['\xde\xad\xbe\xef', None], type 'list')
+CONTEXT: PL/Python function test_type_conversion_array_bytea
+ test_type_conversion_array_bytea
+--
+ {\\xdeadbeef,NULL}
+(1 row)
+
+CREATE FUNCTION test_type_conversion_array_mixed1() RETURNS text[] AS $$
+return [123, 'abc']
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_mixed1();
+ test_type_conversion_array_mixed1
+---
+ {123,abc}
+(1 row)
+
+CREATE FUNCTION test_type_conversion_array_mixed2() RETURNS int[] AS $$
+return [123, 'abc']
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_mixed2();
+ERROR: invalid input syntax for integer: abc
+CONTEXT: while creating return value
+PL/Python function test_type_conversion_array_mixed2
+CREATE FUNCTION test_type_conversion_array_record() RETURNS type_record[] AS $$
+return [None]
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_record();
+ERROR: PL/Python functions cannot return type type_record[]
+DETAIL: PL/Python does not support conversion to arrays of row types.
+CREATE FUNCTION test_type_conversion_array_string() RETURNS text[] AS $$
+return 'abc'
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_string();
+ test_type_conversion_array_string
+---
+ {a,b,c}
+(1 row)
+
+CREATE FUNCTION test_type_conversion_array_tuple() RETURNS text[] AS $$
+return ('abc', 'def')
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_array_tuple();
+ test_type_conversion_array_tuple
+--
+ {abc,def}
+(1 row)
+
diff --git a/src/pl/plpython/plpython.c b/src/pl/plpython/plpython.c
index 6fd4aca..8f097b3 100644
--- a/src/pl/plpython/plpython.c
+++ b/src/pl/plpython/plpython.c
@@ -89,6 +89,9 @@ typedef struct PLyDatumToOb
Oid typoid; /* The OID of the type */
Oid typioparam;
bool typbyval;
+ int16 typlen;
+ char typalign;
+ struct PLyDatumToOb *elm;
} PLyDatumToOb;
typedef struct PLyTupleToOb
@@ -120,6 +123,9 @@ typedef struct PLyObToDatum
Oid typoid; /* The OID of the type */
Oid typioparam;
bool typbyval;
+ int16 typlen;
+ char typalign;
+ struct PLyObToDatum *elm;
} PLyObToDatum;
typedef struct PLyObToTuple
@@ -284,6 +290,7 @@ static PyObject *PLyInt_FromInt32(PLyDatumToOb *arg, Datum d);
static PyObject *PLyLong_FromInt64(PLyDatumToOb *arg, Datum d);
static PyObject
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 3:09 PM, Andrew Chernow a...@esilo.com wrote: What advantage is there in limiting it to a tiny size? This is a 'payload' after all...an arbitrary data block. Looking at the patch I noticed the payload structure (AsyncQueueEntry) is fixed length and designed to lay into QUEUE_PAGESIZE (set to) BLCKSZ sized pages. H. Looks like the limitation comes from slru. The true payload limit is (8K - struct members) the way this is implemented. So I'm beginning to think that adding a payload to NOTIFY is a bad idea altogether. The original use case NOTIFY was designed to handle was to implement condition variables. You have clients wait on some data structure and whenever someone changes that data structure they notify all waiting clients to reread the data structure and check if their condition is true. The condition might be that the data structure doesn't match their locally cached copy and they have to rebuild their cache, it might be that some work queue is non-empty, etc. So the way to build a queue would be to use a table to hold your work queue and then use NOTIFY whenever you insert into the queue and any idle workers would know to go dequeue something from the queue. It sounds like what people are trying to do is use NOTIFY as a queue directly. The problem with this is that it was specifically not designed to do this. As evidenced by the fact that it can't store arbitrary data structures (nothing else in postgres is limited to only handling text data, this would be an annoying arbitrary limitation), can't handle even moderately large data efficiently, etc. I'm beginning to think the right solution is to reject the idea of adding a payload to the NOTIFY mechanism and instead provide a queue contrib module which provides the interface people want. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] array_to_string bug?
Folks,
Here's expected behavior:
davidfet...@postgres=# SELECT array(values(1),(null));
?column?
──
{1,NULL}
(1 row)
The next one is just plain unexpected. Either it's a bug, or it needs
more documentation in the function description in the docs, \df+
output, etc.
davidfet...@postgres=# SELECT array_to_string(array(values(1),(null)),'');
array_to_string
─
1
(1 row)
I would have expected it to come out as NULL.
What say?
Cheers,
David.
--
David Fetter da...@fetter.org http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fet...@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 3:30 PM, Merlin Moncure mmonc...@gmail.com wrote: Couple of questions: *) is BLCKSZ a hard requirement, that is, coming from the slru implementation, or can QUEUE_PAGESIZE be bumped independently of block size. It's the size of slru's pages. *) why not make the AsyncQueueEntry divide evenly into BLCKSZ, that is, make the whole structure a size that is a multiple of two? (this would make the payload length 'weird') it's possible and yes, it would make the payload length weird. Also if we wanted to add further structure members later on (commit time, xid, notify counter, whatever) we'd have to announce that payload in a new release cannot have the same length as in a previous one, so I tried to keep both independent right from the beginning. *) is there any downside you see to making the AsyncQueueEntry structure exactly BLCKSZ bytes in size? Are we worried about the downsides of spinning the notifications out to disk? Yes, and also we'd need a lot more page slots to work efficiently and I fear quite some queue managing overhead (locking). Currently we are using 4 page slots and can have ~160 notifications mapped into memory. *) Is a variable length AsyncQueueEntry possible? (presumably bounded by the max page size). Or does complicate the implementation too much? That's an option to look at, I think it's technically possible, we'd just need to throw in some more logic but I didnt want to invest too much effort before we have a clear way to go. However I share Greg's concerns that people are trying to use NOTIFY as a message queue which it is not designed to be. Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] plpgsql GUC variable: custom or built-in?
So I think we're agreed on adding a variable_conflict GUC for plpgsql. The straight-and-narrow way to do it would be to make this a custom GUC that's defined only when plpgsql is dynamically loaded into the backend. However that implies a lot of notational overhead, notably having to put plpgsql into custom_variable_classes if you want to set the variable in postgresql.conf. Maybe that's okay, particularly if you are of the opinion that most people will leave it at default. But it could also be argued that plpgsql is so widely used that we should bend the rules for it, and make this a built-in GUC that just happens to only be consulted by plpgsql. I'm leaning to the custom GUC approach because it seems a little cleaner from a code point of view, but I wanted to see if anyone wishes to argue for the other way. One reason to argue for the other way is that maybe it wouldn't only be consulted by plpgsql. In particular I can easily imagine SQL functions having the same issue as soon as someone gets around to letting them use names for their parameters. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
--On 12. November 2009 15:48:44 + Greg Stark gsst...@mit.edu wrote: I'm beginning to think the right solution is to reject the idea of adding a payload to the NOTIFY mechanism and instead provide a queue contrib module which provides the interface people want. Isn't PgQ already the solution you have in mind? -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
David Fetter da...@fetter.org writes: The next one is just plain unexpected. array_to_string ignores null elements. What do you think it should do with them? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plpgsql GUC variable: custom or built-in?
On Thu, Nov 12, 2009 at 11:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: One reason to argue for the other way is that maybe it wouldn't only be consulted by plpgsql. In particular I can easily imagine SQL functions having the same issue as soon as someone gets around to letting them use names for their parameters. I don't have a strong feeling on the core issue but I don't agree with this point. AIUI, we are implementing multiple behaviors here for reasons of backward and competing-product compatibility. Presumably, if we're starting from scratch, we'll pick a sensible behavior - probably error in the case of SQL - and stick with it. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: Joachim Wieland j...@mcknight.de writes: However I share Greg's concerns that people are trying to use NOTIFY as a message queue which it is not designed to be. Yes. Particularly those complaining that they want to have very large payload strings --- that's pretty much a dead giveaway that it's not being used as a condition signal. Now you might say that yeah, that's the point, we're trying to enable using NOTIFY in a different style. The problem is that if you are trying to use NOTIFY as a queue, you will soon realize that it has the wrong semantics for that --- in particular, losing notifies across a system crash or client crash is OK for a condition notification, not so OK for a message queue. The difference is that the former style assumes that the authoritative data is in a table somewhere, so you can still find out what you need to know after reconnecting. If you are doing messaging you are likely to think that you don't need any backing store for the system state. So while a payload string for NOTIFY has been on the to-do list since forever, I have to think that Greg's got a good point questioning whether it is actually a good idea. I think there could be cases where the person writing the code can know, extrinsic to the system, that lost notifications are OK, and still want to deliver a payload. But I think the idea of enabling a huge payload is not wise, as it sounds like it will sacrifice performance for a feature that is by definition not essential to anyone who is using this now. A small payload seems like a reasonable compromise. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: Joachim Wieland j...@mcknight.de writes: However I share Greg's concerns that people are trying to use NOTIFY as a message queue which it is not designed to be. Yes. Particularly those complaining that they want to have very large payload strings --- that's pretty much a dead giveaway that it's not being used as a condition signal. Now you might say that yeah, that's the point, we're trying to enable using NOTIFY in a different style. The problem is that if you are trying to use NOTIFY as a queue, you will soon realize that it has the wrong semantics for that --- in particular, losing notifies across a system crash or client crash is OK for a condition notification, not so OK for a message queue. The difference is that the former style assumes that the authoritative data is in a table somewhere, so you can still find out what you need to know after reconnecting. If you are doing messaging you are likely to think that you don't need any backing store for the system state. So while a payload string for NOTIFY has been on the to-do list since forever, I have to think that Greg's got a good point questioning whether it is actually a good idea. You guys are assuming it's being used in a queue, which is only a fraction of cases where this feature is useful. In fact, having a payload can remove the need for a queue completely where is currently required for no other reason to deliver payload messages. I'm sorry, the 128 character limit is simply lame (other than for unsolvable implementation/performance complexity which I doubt is the case here), and if that constraint is put in by the implementation, than the implementation is busted and should be reworked until it's right. A feature that is being used for things not intended is a sign of a strong feature, not a weak one, and the idea that a payload should be length limited in order to prevent use in ways that are 'wrong' is a very weak argument IMO. People have been asking for this feature since the beginning of time, and nobody said: 'please limit it to 128 bytes'. A limit of 4k - 64k is much more appropriate if you even want a hard limit at all... merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 11:39 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane t...@sss.pgh.pa.us wrote: Joachim Wieland j...@mcknight.de writes: However I share Greg's concerns that people are trying to use NOTIFY as a message queue which it is not designed to be. Yes. Particularly those complaining that they want to have very large payload strings --- that's pretty much a dead giveaway that it's not being used as a condition signal. Now you might say that yeah, that's the point, we're trying to enable using NOTIFY in a different style. The problem is that if you are trying to use NOTIFY as a queue, you will soon realize that it has the wrong semantics for that --- in particular, losing notifies across a system crash or client crash is OK for a condition notification, not so OK for a message queue. The difference is that the former style assumes that the authoritative data is in a table somewhere, so you can still find out what you need to know after reconnecting. If you are doing messaging you are likely to think that you don't need any backing store for the system state. So while a payload string for NOTIFY has been on the to-do list since forever, I have to think that Greg's got a good point questioning whether it is actually a good idea. I think there could be cases where the person writing the code can know, extrinsic to the system, that lost notifications are OK, and still want to deliver a payload. But I think the idea of enabling a huge payload is not wise, as it sounds like it will sacrifice performance for a feature that is by definition not essential to 'premature optimization is the root of all evil' :-) merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plpgsql GUC variable: custom or built-in?
Robert Haas robertmh...@gmail.com writes: On Thu, Nov 12, 2009 at 11:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: One reason to argue for the other way is that maybe it wouldn't only be consulted by plpgsql. In particular I can easily imagine SQL functions having the same issue as soon as someone gets around to letting them use names for their parameters. I don't have a strong feeling on the core issue but I don't agree with this point. AIUI, we are implementing multiple behaviors here for reasons of backward and competing-product compatibility. Presumably, if we're starting from scratch, we'll pick a sensible behavior - probably error in the case of SQL - and stick with it. Fair enough. I'll start writing the custom GUC then. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, 2009-11-12 at 06:46 -0500, Robert Haas wrote: Having said that, I'm not capable of single-handedly effecting an on-time release You're bloody good and the task needs to fit our capability anyway. So, yes, you are. We need larger, more robust pools of committers, reviewers, commitfest managers, etc. We're living in a desert. We just need to remember it. Plan hard, focus on the important and be real. Move at a smooth pace to save resources. Don't give up when the going gets tough, just rest up and then continue. Not a new idea, but I think we should require all patch submitters to do one review per submission. There needs to be a balance between time spent on review and time spent on dev. The only real way this happens in any community is by peer review. All patch submitters need to know that they must also take their turn as patch reviewers. If it is a hard rule, then patch *sponsors* would be forced to accept that they must *also* pay for review time. It is the sponsors that need to be forced to accept that reality, though we can only get at them through controlling developer behaviour. So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 11:31 AM, Simon Riggs si...@2ndquadrant.com wrote: Not a new idea, but I think we should require all patch submitters to do one review per submission. There needs to be a balance between time spent on review and time spent on dev. The only real way this happens in any community is by peer review. All patch submitters need to know that they must also take their turn as patch reviewers. If it is a hard rule, then patch *sponsors* would be forced to accept that they must *also* pay for review time. It is the sponsors that need to be forced to accept that reality, though we can only get at them through controlling developer behaviour. So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. I agree. I would quibble only with the details. I think we should require patch authors to act as a reviewer for any CommitFest for which they have submitted patches. We don't need every patch author to review as many patches as they submit, because some people will review extras, and some non-patch-authors will review. If they review one patch each, that's probably more than enough. It's also easier for bookkeeping purposes. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Now you might say that yeah, that's the point, we're trying to enable using NOTIFY in a different style. The problem is that if you are trying to use NOTIFY as a queue, you will soon realize that it has the wrong semantics for that --- in particular, losing notifies across a system crash or client crash is OK for a condition notification, not so OK for a message queue. The difference is that the former style assumes that the authoritative data is in a table somewhere, so you can still find out what you need to know after reconnecting. If you are doing messaging you are likely to think that you don't need any backing store for the system state. I simply don't agree that the semantics have to change. You call it a queue, I call it sesison data. There is no reason why the documentation can't state that notifies may not be delivered due to crashes, so make sure to use persistent storage for any payload worth keeping post-session. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 4:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: So while a payload string for NOTIFY has been on the to-do list since forever, I have to think that Greg's got a good point questioning whether it is actually a good idea. Here's an example of why I'd like a payload (and not a queue in an add-on module). Say you have multiple users running pgAdmin. One of them creates a new table. Currently, unless the other user refreshes his view of the database, he won't see it. I'd like to be able to notify the other pgAdmin sessions that a new table has been created, so they can automatically display it, without having to poll pg_class or be manually refreshed. The payload could contain details of type of object that has been created, and it's OID/identifier to minimise the work required of the other sessions to find and display the new object. And as I'm sure you're already thinking it, yes, I know it doesn't help if the new table is created using psql, but there are lots of shops where pgAdmin is the default tool, and it could help them and just exhibit the current behaviour if someone does break out psql. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Joachim Wieland j...@mcknight.de writes: However I share Greg's concerns that people are trying to use NOTIFY as a message queue which it is not designed to be. Yes. Particularly those complaining that they want to have very large payload strings --- that's pretty much a dead giveaway that it's not being used as a condition signal. Now you might say that yeah, that's the point, we're trying to enable using NOTIFY in a different style. The problem is that if you are trying to use NOTIFY as a queue, you will soon realize that it has the wrong semantics for that --- in particular, losing notifies across a system crash or client crash is OK for a condition notification, not so OK for a message queue. The difference is that the former style assumes that the authoritative data is in a table somewhere, so you can still find out what you need to know after reconnecting. If you are doing messaging you are likely to think that you don't need any backing store for the system state. So while a payload string for NOTIFY has been on the to-do list since forever, I have to think that Greg's got a good point questioning whether it is actually a good idea. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Wed, Nov 11, 2009 at 2:03 PM, Josh Berkus j...@agliodbs.com wrote: I'd be happy to get together at some pre-appointed hour this weekend (Saturday / Sunday) to talk it over by phone / IRC. PDXPUG was already planning to get together to review some patches this Sunday from 3-5pm PST, so that is a convenient time for me. Aren't you running OpenSQL this weekend? Yes :) But we have lots of volunteers. It's an unconference, and I delegated. -selena -- http://chesnok.com/daily - me http://endpoint.com - work -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
However I share Greg's concerns that people are trying to use NOTIFY as a message queue which it is not designed to be. When you have an established libpq connection waiting for notifies it is not unreasonable to expect/desire a payload. ISTM, the problem is that the initial design was half-baked. NOTIFY is event-driven, ie. no polling! -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 8:31 AM, Simon Riggs si...@2ndquadrant.com wrote: Not a new idea, but I think we should require all patch submitters to do one review per submission. There needs to be a balance between time spent on review and time spent on dev. The only real way this happens in any community is by peer review. I like this idea. Perhaps also if a person is reviewing a patch for the first time, we could make some effort to get a more experienced person paired up with them. Pairing up was really useful for the PDXPUG folks working on review. -- http://chesnok.com/daily - me http://endpoint.com - work -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: The problem is that fsync needs to be issued too frequently, which would be harmless in asynchronous replication, but not in synchronous one. A transaction would have to wait for the primary's and standby's fsync before returning a success to a client. Surely that is exactly what is *required* if the user has asked for synchronous replication. This a distressingly common thing people get wrong about replication. You can either have synchronous replication, which as you say has to be slow: you must wait for an fsync ACK from the secondary and a return trip before you can say something is committed on the primary. Or you can get better performance by not waiting for all of those things, but the minute you do that it's *not* synchronous replication anymore. You can't get high-performance and true synchronous behavior; you have to pick one. The best you can do if you need both is work on accelerating fsync everywhere using the standard battery-backed write cache technique. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com
[HACKERS] CommitFest 2009-11 Call for Reviewers
In a few days the 3rd 8.5 development CommitFest, 2009-11, is going to kick off, with the end goal being an alpha3 prerelease. If you have a patch in progress, you'll need to submit it before the deadline of 2009-11-15 00:00:00 GMT for it to be considered during this round: http://wiki.postgresql.org/wiki/Submitting_a_Patch The actual process of the CommitFest itself is fairly well documented at this point: http://wiki.postgresql.org/wiki/Reviewing_a_Patch http://wiki.postgresql.org/wiki/RRReviewers http://wiki.postgresql.org/wiki/Running_a_CommitFest For lack of a more qualified volunteer, I'll be handling the initial round of patch assignments and reviewer organization. I suspect we'll reorganize on the fly as things proceed based on who has time; I'd certainly welcome patch-chasing help in addition to reviewing. Since the backlog for this CommitFest is so far lighter than we've seen recently, the small patches that don't already have an active reviewer shouldn't be too difficult to get through. Please send me an email (without copying the list) if you are available to help with review. Include any information that might be helpful in assigning you an appropriate patch. If there's a specific one you want to claim, by all means let me know that. All reviewers will need to be subscribed to the RRR mailing list, so when you write me please also follow the subscription link at http://archives.postgresql.org/pgsql-rrreviewers/ to add yourself to that list, too, if you're not already there. The set of patches I have the least feel for are the five ECPG submissions, some of which were reviewed already. I would particularly appreciate any early information reviewers might provide about their capability/willingness to work on that set. Those are not so easy to just split among multiple people due to how they relate to one another. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Simon Riggs escreveu: So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. Is it really impolite for a first-contributor, no? -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
On Thu, Nov 12, 2009 at 11:20:26AM -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: The next one is just plain unexpected. array_to_string ignores null elements. What do you think it should do with them? It should concatenate them, i.e. return a NULL if the array includes any NULLs. That, or it should explain that it doesn't do that because there's nothing in the docs that would indicate this behavior. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RRR] CommitFest 2009-11 Call for Reviewers
On 11/12/09 9:45 AM, Greg Smith wrote: For lack of a more qualified volunteer, I'll be handling the initial round of patch assignments and reviewer organization. Hmmm? Who's more qualified than you, exactly? --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
That's basically just it: Assume bashing is part of the process. Don't think of it as bashing. Take the constructive criticism from it, ignore the rest. Assume only one out of three feature ideas will make it. Apply the prerequisite amount of gamesmanship to the system and tune your bikeshedding detectors. Don't take anything personally. Live and learn. I think what Emmanuel is complaining about here ... with some justification ... is not the revision requirements of our patch process but the extremely confusing and frustrating nature of it for new contributors. For example, how exactly is a new contributor supposed to know the difference between bikeshedding and constructive criticism, when (a) they don't know the people involved, and (b) even our more dedicated committers engage in bikeshedding periodically? This isn't just Emmanuel; I've heard this complaint from numerous first-time contributors, and have seen several talented people walk away from our project because of it. Even people who have stuck with us, such as Josh Tolley, have remarked on the hazing ritual inherent in getting a first-time contribution in. Probably, the only possible solution is for each new contributor to have a mentor who sticks with them throughout the process of getting their first contribution accepted, explaining the process, pre-reviewing their submissions, and explaining to them which criticism they should be paying attention to and which they shouldn't. I *thought* that Bruce was doing that for AsterData, but apparently not. This would require a pool of experienced contributors volunteering to be mentors, which I'm not sure we have. It would also help if people on this list *in general*, were to be a bit more consistent about phrasing criticism as constructive criticism. I've seen far too much how stupid are you? on this list for the good health of our developer community. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
Tom Lane wrote: David Fetter da...@fetter.org writes: The next one is just plain unexpected. array_to_string ignores null elements. What do you think it should do with them? regards, tom lane This seems somewhat related to the long-running discussion from back in February-April regarding string_to_array with empty input which faded away somewhere around here: http://archives.postgresql.org/pgsql-hackers/2009-04/msg00363.php. At the time the decision was to defer any decision to after 8.4. Perhaps there is a solution which can address both cases - ideally one which would, to the extent practical, allow string_to_array to be the inverse of array_to_string. This could be particularly useful when dealing with clients that don't know how to directly deal with PostgreSQL arrays but which can generally easily deal with strings. Although it might cause a fair amount of backward-compatibility trouble, the string representation could either use NULL to represent a null element as is allowed in other contexts or require that empty-string elements be represented as to differentiate ,, (empty-string element) from ,, (null element). Cheers, Steve
Re: [HACKERS] next CommitFest
I like this idea. Perhaps also if a person is reviewing a patch for the first time, we could make some effort to get a more experienced person paired up with them. When I was CFM last year, I assigned patches for review first if the patch author was doing a review themselves. Patches whose authors were not doing reviews often got reviewed last, which means that sometimes they waited a commitfest. Of course, Simon submitted so many patches that despite him doing reviews, it was hard to get them all assigned. Simon: slow down a little! ;-) --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 1:25 PM, Albert Cervera i Areny alb...@nan-tic.com wrote: A Dijous, 12 de novembre de 2009, Euler Taveira de Oliveira va escriure: Simon Riggs escreveu: So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. Is it really impolite for a first-contributor, no? I don't think so, as long as it's properly explained. Personally, I would not propose to impose this rule of first-time contributors, or even second-time contributors. But by about patch #3 I think everyone should be pitching in. Just MHO, of course. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
A Dijous, 12 de novembre de 2009, Euler Taveira de Oliveira va escriure: Simon Riggs escreveu: So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. Is it really impolite for a first-contributor, no? I don't think so, as long as it's properly explained. -- Albert Cervera i Areny http://www.NaN-tic.com Mòbil: +34 669 40 40 18 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
On Thu, Nov 12, 2009 at 1:28 PM, Steve Crawford scrawf...@pinpointresearch.com wrote: Although it might cause a fair amount of backward-compatibility trouble, the string representation could either use NULL to represent a null element as is allowed in other contexts or require that empty-string elements be represented as to differentiate ,, (empty-string element) from ,, (null element). That would cause a substantial amount of grief to people who might not want that behavior, though. I use these functions for creating human-readable output, not for serialization. Simple, predictable behavior is very important. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
2009/11/13 Euler Taveira de Oliveira eu...@timbira.com: Simon Riggs escreveu: So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. Is it really impolite for a first-contributor, no? I support Simon's proposal, but I think we would certainly need to make an exception for first-time contributors. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 11:40 AM, Merlin Moncure mmonc...@gmail.com wrote: I'm sorry, the 128 character limit is simply lame (other than for unsolvable implementation/performance complexity which I doubt is the case here), and if that constraint is put in by the implementation, than the implementation is busted and should be reworked until it's right. After some reflection, I realized this was an overly strong statement and impolite to the OP. It's easy to yarp from the gallery with the other peanuts :-). It's not the implementation I have an issue with, just the _idea_ that we should be restricted to small payloads for religious reasons...until that came upI was already scheming on how to both extend the patch to be more flexible in terms of payload size, and to backpatch and test it on 8.4 (no point if the community has no interest however). In any event, sorry for the strong words. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, 2009-11-12 at 15:52 -0200, Euler Taveira de Oliveira wrote: Simon Riggs escreveu: So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. Is it really impolite for a first-contributor, no? I believe the community has always given extra help to first-time contributors. If anyone sees a new contributor being ignored, they should make an effort to help. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Robert Haas escribió: On Thu, Nov 12, 2009 at 1:25 PM, Albert Cervera i Areny alb...@nan-tic.com wrote: A Dijous, 12 de novembre de 2009, Euler Taveira de Oliveira va escriure: Simon Riggs escreveu: So, I propose that we simply ignore patches from developers until they have done sufficient review to be allowed to develop again. Is it really impolite for a first-contributor, no? I don't think so, as long as it's properly explained. Personally, I would not propose to impose this rule of first-time contributors, or even second-time contributors. But by about patch #3 I think everyone should be pitching in. +1. It's OK to not review anything if you're just doing a one-off. If you submit a bunch of patches you start becoming part of the community. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plpgsql GUC variable: custom or built-in?
I wrote: Fair enough. I'll start writing the custom GUC then. While testing that, I noticed a pre-existing GUC limitation that maybe we should do something about now: it does not work very nicely for custom SUSET GUC variables. Normally, if a variable is SUSET, an ordinary user can't do ALTER USER SET or ALTER DATABASE SET to modify it. However, a superuser can do that on his behalf, and then the variable will default in the desired way for that user or database. But this logic doesn't work for custom variables: the system doesn't record whether the option was set by a superuser or not, so it's afraid to allow the value to be applied when the defining module gets loaded. (Instead you get a WARNING and the value reverts to default.) I think we discussed this once before and came up with the idea that it wouldn't be a problem if ALTER USER/DATABASE SET disallowed setting of variables not currently known to the system. Right now, if plpgsql is in custom_variable_classes, you can do ALTER USER foouser SET plpgsql.variable_conflict = variable_first; and the system will take it even if plpgsql isn't loaded. If we required plpgsql to be loaded then we could be sure that the appropriate permissions check had been made when the ALTER was done, and then in subsequent sessions it would be safe to apply the variable value while loading plpgsql. One possible objection is that a loadable module couldn't safely upgrade a USERSET variable to SUSET (except across a major version boundary), since the permissions check would already have been made implicitly for any ALTER settings. This doesn't seem like a big problem compared to the current situation of not being able to use SUSET effectively at all, though. Another issue is that pg_dumpall output would fail to reload with such a restriction, since the dump script would most likely be executed in a session that hadn't loaded the relevant loadable module. We could get around that by still allowing superusers to issue ALTER SET for unknown variables, but that seems a tad weird. OTOH the current rule (must be in custom_variable_classes) is pretty hazardous for dump reloading, too. Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
On Thu, Nov 12, 2009 at 01:33:41PM -0500, Robert Haas wrote: On Thu, Nov 12, 2009 at 1:28 PM, Steve Crawford scrawf...@pinpointresearch.com wrote: Although it might cause a fair amount of backward-compatibility trouble, the string representation could either use NULL to represent a null element as is allowed in other contexts or require that empty-string elements be represented as to differentiate ,, (empty-string element) from ,, (null element). That would cause a substantial amount of grief to people who might not want that behavior, though. I use these functions for creating human-readable output, not for serialization. Simple, predictable behavior is very important. My question boils down to, why is this string concatenation different from all other string concatenations? For now, the answer can be, it behaves differently with respect to NULLs, and we just document this. We can later decide whether this behavior should change. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
Robert Haas robertmh...@gmail.com writes: On Thu, Nov 12, 2009 at 1:28 PM, Steve Crawford scrawf...@pinpointresearch.com wrote: Although it might cause a fair amount of backward-compatibility trouble, the string representation could either use NULL to represent a null element as is allowed in other contexts or require that empty-string elements be represented as to differentiate ,, (empty-string element) from ,, (null element). That would cause a substantial amount of grief to people who might not want that behavior, though. I use these functions for creating human-readable output, not for serialization. Simple, predictable behavior is very important. I agree --- we don't want to start introducing quoting rules into array_to_string. I think the viable alternatives are the current behavior, or treating a NULL element as if it were an empty string. David's idea that the entire output should go to NULL might be sane from a strict semantics point of view, but it also seems to make the function just about entirely useless in practice. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, 2009-11-12 at 11:36 -0500, Robert Haas wrote: I agree. I would quibble only with the details. I think we should require patch authors to act as a reviewer for any CommitFest for which they have submitted patches. We don't need every patch author to review as many patches as they submit, because some people will review extras, and some non-patch-authors will review. If they review one patch each, that's probably more than enough. It's also easier for bookkeeping purposes. Not all contributors are fluent English readers and writers. I certainly do not discourage those people from reviewing, but I can understand how it might be more frustrating and less productive for them. An important part of review is to read the relevant mailing list threads and try to tie up loose ends and find a consensus. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On Thu, Nov 12, 2009 at 1:27 PM, Josh Berkus j...@agliodbs.com wrote: That's basically just it: Assume bashing is part of the process. Don't think of it as bashing. Take the constructive criticism from it, ignore the rest. Assume only one out of three feature ideas will make it. Apply the prerequisite amount of gamesmanship to the system and tune your bikeshedding detectors. Don't take anything personally. Live and learn. I think what Emmanuel is complaining about here ... with some justification ... is not the revision requirements of our patch process but the extremely confusing and frustrating nature of it for new contributors. For example, how exactly is a new contributor supposed to know the difference between bikeshedding and constructive criticism, when (a) they don't know the people involved, and (b) even our more dedicated committers engage in bikeshedding periodically? This isn't just Emmanuel; I've heard this complaint from numerous first-time contributors, and have seen several talented people walk away from our project because of it. Even people who have stuck with us, such as Josh Tolley, have remarked on the hazing ritual inherent in getting a first-time contribution in. I used to feel this way, too. I'm not sure whether it's really worse at first, or whether it just seems worse a first until you get used to it. There is no getting around the fact that this is a community of very smart people. I work at a company where I'm the only developer. Guess what - my development ideas are all brilliant - or at least there's no contrary evidence! Then I come here, where there are plenty of other developers just as smart as I am, or possibly smarter, and here some of my ideas crash and burn. On the other hand, sometimes someone objects to my idea and the *objections* crash and burn. You can't expect to win them all when you're dealing with people who are as smart as you and more knowledgeable about the code. I try to be gentle with newcomers but it's not always easy to figure out what that means. When someone submits a bad patch and I review it, they sometimes get two or three pages of feedback detailing inadequacies of various types. I understand that can be hard on the ego, but I don't think it does anyone any favors to give them a PARTIAL list of things they need to fix - in fact I think breaking up the pain into multiple rounds is quite a bit worse, because then you create the illusion of progress without the actuality. I don't give people feedback because I want to hurt them or cause them pain, but I know from experience that until those things are fixed, it *will* *not* *get* *committed*. If I say nothing now, someone else will say it later. I am afraid that the difference between bikeshedding and legitimate criticism is something that contributors have to learn for themselves. I can tell you which things *I* think are bikeshedding, but someone else may have a longer, shorter, or same-size non-overlapping list. No one can speak with authority on this topic because no one is the boss. All that having been said, I feel your pain. I don't want to scare off new contributors, but I don't necessarily know how to improve the situation. Probably, the only possible solution is for each new contributor to have a mentor who sticks with them throughout the process of getting their first contribution accepted, explaining the process, pre-reviewing their submissions, and explaining to them which criticism they should be paying attention to and which they shouldn't. I *thought* that Bruce was doing that for AsterData, but apparently not. Well, this is a good idea, although I'm not sure how well it will play out in practice. I'm always game if anyone wants to email me off-list to discuss issues of this type, or... This would require a pool of experienced contributors volunteering to be mentors, which I'm not sure we have. ...I'm willing to do this. It would also help if people on this list *in general*, were to be a bit more consistent about phrasing criticism as constructive criticism. I've seen far too much how stupid are you? on this list for the good health of our developer community. I don't see very much of this at all, so maybe this is another area where it boils down to perspective. I find this list to be quite civilized most of the time; I wouldn't hang around if I didn't. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE...ALTER COLUMN vs inheritance
--On 4. November 2009 09:57:27 -0500 Tom Lane t...@sss.pgh.pa.us wrote: I think the consensus was that the way to fix this (along with some other problems) is to start representing NOT NULL constraints in pg_constraint, turning attnotnull into just a bit of denormalization for performance. I've just started looking into this and wonder how this should look like. My first idea is to just introduce a special contype in pg_constraint representing a NOT NULL constraint on a column, which holds all required information to do the mentioned maintenance stuff on them and to keep most of the current infrastructure. Utility commands need to track all changes in pg_constraint and keep pg_attribute.attnotnull up to date. Another possibility is to change the representation of NOT NULL to be a CHECK constraint (e.g. CHECK(col IS NOT NULL)) internally and leave all the responsibility up to the current existing check constraint infrastructure (which already does the right thing for inheritance, e.g. it's not possible to drop such a constraint if it was inherited). ALTER TABLE ... SET NOT NULL and DROP NOT NULL will be just syntactic sugar then. I don't know the original design decisions for the current representation, but it seems it wasn't essential? Of course, there's still the requirement to special case those check constraints in various places, since pg_dump and psql have to do the right thing. -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RRR] CommitFest 2009-11 Call for Reviewers
Josh Berkus wrote: On 11/12/09 9:45 AM, Greg Smith wrote: For lack of a more qualified volunteer, I'll be handling the initial round of patch assignments and reviewer organization. Hmmm? Who's more qualified than you, exactly? I was alluding to the fact that Robert isn't available to handle this one. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com
Re: [HACKERS] Patch committers
I'm not addressing the complaints Emmanuel has about patch submission, but was inspired by what he said. I hesitated to post this to -hackers, but my hope is that my comments are taken in the spirit of reflection. I've been thinking about the Postgres community and things that differentiate it from other FOSS projects for a few years. We lack both a benevolent dictator, or a single-corporate backer... We have an incredible reputation for technical expertise -- most knowledgeable FOSS developers fear and are in awe of the quality of code our community produces. I hear criticism for the way in which our development community operates, and how we communicate with outsiders. But few people are able to offer concrete suggestions on /how/ we might change, as opposed to the /particular things/ that they think should change. So on this point: On Wed, Nov 11, 2009 at 7:30 PM, Emmanuel Cecchet m...@frogthinker.org wrote: The commitfest started to address some of the most obvious issues but there is probably much more that can be done by looking at what is happening in the other open source communities. Speaking from a cultural, and organizational standpoint: * Distributed revision control as standard for the project Using a distributed revision control system, and working from developer branches which are pulled pushed directly between developers is consistently reported to have a positive impact on solving problems and sharing information. Summer of Code mentors consistently mention DVCS as having a huge positive impact on productivity for their students - both because it's easy and cheap for students to commit to their local repos, and because the merge/pull tools are intuitive for the mentors and students. This is how people work and share code out there. I realize that Postgres has an established cultural norm of posting patches to the mailing list, and I'm not suggesting that people stop doing that. I think more developers should be sharing branches as well. git.postgresql.org should probably be reorganized to clearly divide repos between postgresql-core and other projects (contrib, add-ons, whatever). We should also publish more information about how our developers are sharing code with each other through repos. * Decision making Is there a project you'd point to as a model for decision making? (serious question) This is a volunteer organization, and as such, we're subject to who has the expertise + time/energy to devote to the patches and projects. Going back to what Bruce has said about skill drain -- if more individuals were able to get their time sponsored to devote to the technical challenges posed by the patches, then it might be possible to have more coordinated/streamlined responses and decision making. My experience in FOSS projects is that Postgres does fairly well on execution compared to other projects (getting a new release out the door every year, and now the alphas every couple months), but has some issues with a couple large, important patches. Part of the problem there appears to be people being stretched too thin, and inconsistent communication practices. More involvement from people really focusing on improving communication between developers would be helpful to make those issues easier to work around. * Volunteers, funding and burnout It's challenging to keep up with the technical discussions on -hackers even if you are a very knowledgeable developer. If you're only doing that on your free time, and say, you have a family? Most of the developers that I know in other projects in leadership/commit manager roles don't have young children, and very few are women. So, another thing that we could use are more people like Bruce that can devote a great deal of *work* time to thinking about and acting on the issues around community management and recruitment. Robert Haas' time spent on commitfest, for example, if sponsored by a foundation would be very helpful. Maybe he could have bought a new dishwasher. :) Also, we could use an individual dedicated to raising funds and consistently executing on distributing the money out for non-code infrastructure projects. Like: * release management, * marketing, * dedicated public relations (driving things like new software release articles, organizing interviews with press, and getting new case studies written), and * keeping track of our relationships with other organizations (Summer of Code, conferences, commercial forks). A lot of this work falls on Josh Berkus to coordinate. There are many more tasks, and some are delegated to others coordinated on -advocacy. Again, this is work that could be funded through a foundation, or as part of positions in particular companies. IMO, it's about providing incentives that encourage people to use their time wisely, and to help prevent burnout on the difficult tasks around people management. And to say definitively that the people tasks require organization and management, and we as a
[HACKERS] Check constraint on domain over an array not executed for array literals
Hi While trying to create a domain over an array type to enforce a certain shape or certain contents of an array (like the array being only one-dimensional or not containing NULLs), I've stumbled over what I believe to be a bug in postgresql 8.4 It seems that check constraints on domains are *not* executed for literals of the domain-over-array-type - in other words, for expressions like: array[...]::my-domain-over-array-type. They are, however, executed if I first force the array to be of the base type, and then cast it to the array type. Here is an example that reproduces the problem: create domain myintarray as int[] check ( -- Check that the array is neither null, nor empty, -- nor multi-dimensional (value is not null) and (array_length(value,1) is not null) and (array_length(value,1) 0) and (array_length(value,2) is null) ); select null::myintarray; -- Fails (Right) select array[]::myintarray; -- Succeeds (Wrong) select array[]::int[]::myintarray; -- Fails (Right) select array[1]::myintarray; -- Succeeds (Right) select array[1]::int[]::myintarray; -- Succeeds (Right) select array[array[1]]::myintarray; -- Succeeds (Wrong) select array[array[1]]::int[][]::myintarray; -- Fails (Right) I guess the reason is that the ::arraytype part of array[...]::arraytype isn't really a cast at all, but instead part of the array literal syntax. Hence, array[]::myintarray probably creates an empty myintarray instance, and then adds the elements between the square brackets (none) - with none of this steps triggering a run of the check constraint. I still have the feeling that this a bug, though. First, because it leaves you with no way at guarantee that values of a given domain always fulfill certain constraints. And second because array[...]::arraytype at least *looks* like a cast, and hence should behave like one too. best regards, Florian Pflug smime.p7s Description: S/MIME Cryptographic Signature
Re: [HACKERS] CommitFest 2009-11 Call for Reviewers
Greg Smith g...@2ndquadrant.com writes: The set of patches I have the least feel for are the five ECPG submissions, some of which were reviewed already. I would particularly appreciate any early information reviewers might provide about their capability/willingness to work on that set. Those are not so easy to just split among multiple people due to how they relate to one another. AFAIK the ecpg patches are all waiting on Michael Meskes to have time to review/commit them. ecpg is pretty much his turf and no other committers are likely to touch these patches. If anyone really wants to review them, of course, that's fine ... but I don't think you should assign them to someone just because the CF process says to. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_dump enhancement proposal
Hello Everyone, I recently spent some time comparing the various techniques used to backup/export MySQL, SQLite, and PostgreSQL databases. So far, pg_dump is the best tool I've used. I was well pleased with its efficient syntax and functionality. However, I have two feature requests that I think can make it even better: 1. View Based Exports In addition to being able to dump specific tables, it seems useful to be able to quickly specify and dump entire views. 2. Custom Query Exports In my use of mysqldump, I found one feature very useful: the ability to execute a custom SELECT. . .WHERE statement and then dump only the results. This feature currently provides MySQL users with the ability to quickly and easily export very granular data subsets, and I see no reason why PostgreSQL users wouldn't benefit from the same capability. While it is true that this functionality can already be achieved in PostgreSQL using Copy, it seems to me that it would logically fit well as an extension to pg_dump, especially since many beginning and even some intermediate PostgreSQL users aren't aware of the alternatives. If the community finds value in the above features, I would be glad to begin developing them for possible inclusion in the next PostgreSQL release. Best Regards, Mark Hammonds Web Engineer OmniTI Computer Consulting technology.markhammonds.ws -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 1:45 PM, Jeff Davis pg...@j-davis.com wrote: On Thu, 2009-11-12 at 11:36 -0500, Robert Haas wrote: I agree. I would quibble only with the details. I think we should require patch authors to act as a reviewer for any CommitFest for which they have submitted patches. We don't need every patch author to review as many patches as they submit, because some people will review extras, and some non-patch-authors will review. If they review one patch each, that's probably more than enough. It's also easier for bookkeeping purposes. Not all contributors are fluent English readers and writers. I certainly do not discourage those people from reviewing, but I can understand how it might be more frustrating and less productive for them. An important part of review is to read the relevant mailing list threads and try to tie up loose ends and find a consensus. Unfortunately, those people's patches also typically require more work from other community members. Discussions are longer and more confused, and someone has to rewrite the documentation and comments before commit. If anything, people whose patches require more help need to find ways to contribute MORE to the community, not less. I understand that's not easy, but it's necessary. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
On Nov 12, 2009, at 10:46 AM, David Fetter wrote: My question boils down to, why is this string concatenation different from all other string concatenations? Does it have something to do with afikoman? David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Boszormenyi Zoltan escribió: Alvaro Herrera írta: I have applied this patch after some tinkering. I mainly added support for fetch_args: FORWARD opt_from_in name and BACKWARD opt_from_in name in ecpg.addons which apparently you forgot. Thanks. Your fix is correct if this patch is considered standalone. This means I have to re-post the later patches to fix the reject this fix causes in them. Yeah. BTW I don't think the rest of the pieces in this series make sense to apply separately, because they don't do anything useful by themselves (one of them introduces an unused function, what good is in that?). I think they should be submitted as a single patch. If you want to submit patches in a series like this one, they need to be considered standalone, I think. The Linux kernel devs work differently than us here. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-11 Call for Reviewers
Tom Lane wrote: Greg Smith g...@2ndquadrant.com writes: The set of patches I have the least feel for are the five ECPG submissions, some of which were reviewed already. I would particularly appreciate any early information reviewers might provide about their capability/willingness to work on that set. Those are not so easy to just split among multiple people due to how they relate to one another. AFAIK the ecpg patches are all waiting on Michael Meskes to have time to review/commit them. ecpg is pretty much his turf and no other committers are likely to touch these patches. If anyone really wants to review them, of course, that's fine ... but I don't think you should assign them to someone just because the CF process says to. FWIW I committed the parts of one of these patches that touched the core grammar mostly, because I think those might have been holding Michael back a bit. Hopefully that'll make it easier for him to review the rest. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] array_to_string bug?
On Thu, Nov 12, 2009 at 11:46:54AM -0800, David Wheeler wrote: On Nov 12, 2009, at 10:46 AM, David Fetter wrote: My question boils down to, why is this string concatenation different from all other string concatenations? Does it have something to do with afikoman? I was wondering who'd comment on that first ;) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Python 3.1 support
Here's the patch to support Python =3.1 with PL/Python. The
compatibility code is mostly in line with the usual 2-3 C porting
practice and is documented inline.
I needed to create an arguably weird hack to manage the regression
tests. Instead of creating a new expected file for pretty much every
test file and also for some input files (where Python syntax had
changed), a sed script creates a complete Python 3 compatible set of
input and output files. Doesn't look pretty but works quite well. If
anyone has a better idea, please let me know.
diff --git a/src/pl/plpython/Makefile b/src/pl/plpython/Makefile
index 373bc79..840b874 100644
--- a/src/pl/plpython/Makefile
+++ b/src/pl/plpython/Makefile
@@ -88,8 +88,32 @@ installdirs: installdirs-lib
uninstall: uninstall-lib
+ifneq (,$(findstring 3.,$(python_version)))
+# Adjust regression tests for Python 3 compatibility
+prep3:
+ mkdir -p 3 3/sql 3/expected
+ for file in $(srcdir)/sql/* $(srcdir)/expected/*; do \
+ sed -r -e 's/except ([[:alpha:].]+), ?(\w+):/except \1 as \2:/g' \
+ -e s/type 'exceptions\.(\w+)'/class '\1'/g \
+ -e s/type 'long'/class 'int'/g \
+ -e s/\b([0-9]+)L\b/\1/g \
+ -e 's/\bu//g' \
+ -e s/\bu'/'/g \
+ -e s/def next\b/def __next__/g \
+ $$file `echo $$file | sed 's,$(srcdir),3,'`; \
+ done
+
+clean3:
+ rm -rf 3/
+
+installcheck: submake prep3
+ cd 3 $(top_builddir)/../src/test/regress/pg_regress --inputdir=. --psqldir=$(PSQLDIR) $(REGRESS_OPTS) $(REGRESS)
+
+clean: clean3
+else
installcheck: submake
$(top_builddir)/src/test/regress/pg_regress --inputdir=$(srcdir) --psqldir=$(PSQLDIR) $(REGRESS_OPTS) $(REGRESS)
+endif
.PHONY: submake
submake:
diff --git a/src/pl/plpython/expected/README b/src/pl/plpython/expected/README
index 47f31e8..a187937 100644
--- a/src/pl/plpython/expected/README
+++ b/src/pl/plpython/expected/README
@@ -8,3 +8,5 @@ plpython_unicode_0.out any version, when server encoding != SQL_ASCII and clien
plpython_unicode_2.out Python 2.2
plpython_unicode_3.out Python 2.3, 2.4
plpython_unicode_5.out Python 2.5, 2.6
+
+plpython_types_3.out Python 3.1
diff --git a/src/pl/plpython/expected/plpython_types_3.out b/src/pl/plpython/expected/plpython_types_3.out
new file mode 100644
index 000..3fcb0f4
--- /dev/null
+++ b/src/pl/plpython/expected/plpython_types_3.out
@@ -0,0 +1,479 @@
+--
+-- Test data type behavior
+--
+--
+-- Base/common types
+--
+CREATE FUNCTION test_type_conversion_bool(x bool) RETURNS bool AS $$
+plpy.info(x, type(x))
+return x
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_bool(true);
+INFO: (True, class 'bool')
+CONTEXT: PL/Python function test_type_conversion_bool
+ test_type_conversion_bool
+---
+ t
+(1 row)
+
+SELECT * FROM test_type_conversion_bool(false);
+INFO: (False, class 'bool')
+CONTEXT: PL/Python function test_type_conversion_bool
+ test_type_conversion_bool
+---
+ f
+(1 row)
+
+SELECT * FROM test_type_conversion_bool(null);
+INFO: (None, class 'NoneType')
+CONTEXT: PL/Python function test_type_conversion_bool
+ test_type_conversion_bool
+---
+
+(1 row)
+
+-- test various other ways to express Booleans in Python
+CREATE FUNCTION test_type_conversion_bool_other(n int) RETURNS bool AS $$
+# numbers
+if n == 0:
+ ret = 0
+elif n == 1:
+ ret = 5
+# strings
+elif n == 2:
+ ret = ''
+elif n == 3:
+ ret = 'fa' # true in Python, false in PostgreSQL
+# containers
+elif n == 4:
+ ret = []
+elif n == 5:
+ ret = [0]
+plpy.info(ret, not not ret)
+return ret
+$$ LANGUAGE plpythonu;
+SELECT * FROM test_type_conversion_bool_other(0);
+INFO: (0, False)
+CONTEXT: PL/Python function test_type_conversion_bool_other
+ test_type_conversion_bool_other
+-
+ f
+(1 row)
+
+SELECT * FROM test_type_conversion_bool_other(1);
+INFO: (5, True)
+CONTEXT: PL/Python function test_type_conversion_bool_other
+ test_type_conversion_bool_other
+-
+ t
+(1 row)
+
+SELECT * FROM test_type_conversion_bool_other(2);
+INFO: ('', False)
+CONTEXT: PL/Python function test_type_conversion_bool_other
+ test_type_conversion_bool_other
+-
+ f
+(1 row)
+
+SELECT * FROM test_type_conversion_bool_other(3);
+INFO: ('fa', True)
+CONTEXT: PL/Python function test_type_conversion_bool_other
+ test_type_conversion_bool_other
+-
+ t
+(1 row)
+
+SELECT * FROM test_type_conversion_bool_other(4);
+INFO: ([], False)
+CONTEXT: PL/Python function test_type_conversion_bool_other
+ test_type_conversion_bool_other
+-
+ f
+(1 row)
+
+SELECT * FROM test_type_conversion_bool_other(5);
+INFO: ([0], True)
+CONTEXT: PL/Python function test_type_conversion_bool_other
+ test_type_conversion_bool_other
+-
+ t
+(1 row)
+
+CREATE FUNCTION test_type_conversion_char(x char) RETURNS
Re: [HACKERS] Patch committers
Robert Haas escribió: I used to feel this way, too. I'm not sure whether it's really worse at first, or whether it just seems worse a first until you get used to it. There is no getting around the fact that this is a community of very smart people. I work at a company where I'm the only developer. Guess what - my development ideas are all brilliant - or at least there's no contrary evidence! Then I come here, where there are plenty of other developers just as smart as I am, or possibly smarter, and here some of my ideas crash and burn. On the other hand, sometimes someone objects to my idea and the *objections* crash and burn. You can't expect to win them all when you're dealing with people who are as smart as you and more knowledgeable about the code. Maybe this paragraph (and some others from this email and Josh's) should be part of the how to submit a patch wiki page. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
On Tue, 03 Nov 2009 10:32:17 -0800 Josh Berkus wrote: The same goes for other OSS projects. There's quite a few random OSS apps which were created on PG 7.4 and have never offered their users an upgrade path (Gnuworld comes to mind). They need an EOL announcement to get them motivated to upgrade. I know several customers who decided to move from 7.3 only after the EOL was announced. If 7.3 would not has see an EOL, they would never ever have moved to a newer version. We'd want to do a full publicity around this, including a how do I upgrade page and an what does EOL mean for an OSS project page. If this goes well, we could EOL 8.0 after 8.5 comes out, and thus decrease our maintenance burden. +1 -- Andreas 'ads' Scherbaum German PostgreSQL User Group European PostgreSQL User Group - Board of Directors Volunteer Regional Contact, Germany - PostgreSQL Project -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
On Thu, Nov 12, 2009 at 2:49 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Boszormenyi Zoltan escribió: Alvaro Herrera írta: I have applied this patch after some tinkering. I mainly added support for fetch_args: FORWARD opt_from_in name and BACKWARD opt_from_in name in ecpg.addons which apparently you forgot. Thanks. Your fix is correct if this patch is considered standalone. This means I have to re-post the later patches to fix the reject this fix causes in them. Yeah. BTW I don't think the rest of the pieces in this series make sense to apply separately, because they don't do anything useful by themselves (one of them introduces an unused function, what good is in that?). I think they should be submitted as a single patch. If you want to submit patches in a series like this one, they need to be considered standalone, I think. The Linux kernel devs work differently than us here. Zoltan broke them up because Michael asked him to do so. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, 2009-11-12 at 14:43 -0500, Robert Haas wrote: Not all contributors are fluent English readers and writers. I certainly do not discourage those people from reviewing, but I can understand how it might be more frustrating and less productive for them. An important part of review is to read the relevant mailing list threads and try to tie up loose ends and find a consensus. Unfortunately, those people's patches also typically require more work from other community members. Discussions are longer and more confused, and someone has to rewrite the documentation and comments before commit. If anything, people whose patches require more help need to find ways to contribute MORE to the community, not less. I understand that's not easy, but it's necessary. Keep in mind that many of these people may also be regional contacts, translators, local conference (or user group) organizers, and provide support to users in languages other than English. They may even organize development efforts in languages other than English. I think the best policies encourage people to help in the ways that they are most effective, and/or enjoy the most. I know that's a general statement, and it doesn't translate (excuse the pun) easily into policy. I don't have a good answer for that. But sometimes the policy discussions here seem a little abstract, and I have a hard time seeing how they apply in real situations. It seems like you have a few freeloaders in mind, but I have a hard time imagining it's more than a couple people. Maybe they just need a nice reminder to be a more helpful community member if they want timely feedback on their work? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
Andreas 'ads' Scherbaum wrote: On Tue, 03 Nov 2009 10:32:17 -0800 Josh Berkus wrote: The same goes for other OSS projects. There's quite a few random OSS apps which were created on PG 7.4 and have never offered their users an upgrade path (Gnuworld comes to mind). They need an EOL announcement to get them motivated to upgrade. I know several customers who decided to move from 7.3 only after the EOL was announced. If 7.3 would not has see an EOL, they would never ever have moved to a newer version. Nobody that I have seen is arguing against EOLing 7.4. We'd want to do a full publicity around this, including a how do I upgrade page and an what does EOL mean for an OSS project page. If this goes well, we could EOL 8.0 after 8.5 comes out, and thus decrease our maintenance burden. +1 The only burden of significance I have seen actually mentioned, as opposed to supposed, is from Dave Page. What I and others have been arguing is necessary to do EOL right is a serious amount of notice, by way of press releases etc. We can't expect users to keep polling our web site to see if there's an EOL. That means we need to prepare for an EOL months or a year in advance, ISTM. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
I think that's the key argument here. We have several customers, which need a very careful and time consuming evaluation before they can go into production with a new platform, which is quite time consuming and needs significant preparation for them. Announcing an EOL early in time would give them the required time before the version used disappears. So, should we announce it for June? --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] cvs head doesn't pass make check on one of the machines here
consistently fails when compiled on ubuntu 9.10 here (on mini 10v). regression.diffs.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE...ALTER COLUMN vs inheritance
Bernd Helmle maili...@oopsware.de writes: My first idea is to just introduce a special contype in pg_constraint representing a NOT NULL constraint on a column, which holds all required information to do the mentioned maintenance stuff on them and to keep most of the current infrastructure. Utility commands need to track all changes in pg_constraint and keep pg_attribute.attnotnull up to date. Another possibility is to change the representation of NOT NULL to be a CHECK constraint (e.g. CHECK(col IS NOT NULL)) internally and leave all the responsibility up to the current existing check constraint infrastructure (which already does the right thing for inheritance, e.g. it's not possible to drop such a constraint if it was inherited). I'd go for the first of those, for sure. Testing attnotnull is significantly cheaper than enforcing a generic constraint expression, and NOT NULL is a sufficiently common case to be worth worrying about optimizing it. Furthermore, removing attnotnull would break an unknown but probably not-negligible amount of client-side code that cares whether columns are known not null (I think both JDBC and ODBC track that). And it would significantly complicate code in the backend that wants to determine whether a column is known not null --- there are several proposals for planner optimizations that would depend on knowing that, for example. You will find yourself copying-and-pasting a fair amount of the check-constraint code if you implement this as a completely separate code path, and so it might be worthwhile to be creative about exactly what the pg_constraint representations look like --- maybe you can design it to share some of that code. But I recommend strongly that attnotnull stay there. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
Peter Eisentraut pete...@gmx.net writes: Here's the patch to support Python =3.1 with PL/Python. The compatibility code is mostly in line with the usual 2-3 C porting practice and is documented inline. There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Thu, Nov 12, 2009 at 3:12 PM, Jeff Davis pg...@j-davis.com wrote: On Thu, 2009-11-12 at 14:43 -0500, Robert Haas wrote: Not all contributors are fluent English readers and writers. I certainly do not discourage those people from reviewing, but I can understand how it might be more frustrating and less productive for them. An important part of review is to read the relevant mailing list threads and try to tie up loose ends and find a consensus. Unfortunately, those people's patches also typically require more work from other community members. Discussions are longer and more confused, and someone has to rewrite the documentation and comments before commit. If anything, people whose patches require more help need to find ways to contribute MORE to the community, not less. I understand that's not easy, but it's necessary. Keep in mind that many of these people may also be regional contacts, translators, local conference (or user group) organizers, and provide support to users in languages other than English. They may even organize development efforts in languages other than English. They may, but it's beyond my power to keep track of everything that a person may or may not do. If we're going to have any chance of enforcing a policy, it has to be simple and clear. I don't have a good answer for that. But sometimes the policy discussions here seem a little abstract, and I have a hard time seeing how they apply in real situations. It seems like you have a few freeloaders in mind, but I have a hard time imagining it's more than a couple people. Maybe they just need a nice reminder to be a more helpful community member if they want timely feedback on their work? Well, you can go back and look at the list of people who had patches reviewed last CF (see commitfest.postgresql.org), and then go back and look at the reviewers (see pgsql-rrreviewers archives). It's all public data. It's not 1 or 2 people - I count at least 8. I could cross-reference for you and list the names, but that seems like embarassing those people without accomplishing anything useful. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs head doesn't pass make check on one of the machines here
Grzegorz Jaskiewicz wrote: consistently fails when compiled on ubuntu 9.10 here (on mini 10v). + ERROR: incompatible library /home/kgrittn/postgresql-8.4.0/src/test/regress/refint.so: version mismatch + DETAIL: Server is version 8.4, library is version 8.3. You might take this as a clue as to what you're doing wrong ... cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs head doesn't pass make check on one of the machines here
Grzegorz Jaskiewicz g...@pointblue.com.pl writes: consistently fails when compiled on ubuntu 9.10 here (on mini 10v). Locale issue maybe? Looks like it has some odd ideas about text sort order. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-11 Call for Reviewers
Tom Lane wrote: AFAIK the ecpg patches are all waiting on Michael Meskes to have time to review/commit them. ecpg is pretty much his turf and no other committers are likely to touch these patches. Great to know, and since some of the regular reviewers already made a pass through them there's probably not too much general feedback left anyway. I just marked all of those as having Michael as the reviewer. If it gets to where those are the main remaining hold-up I guess we'll revisit who else might help out then. Would rather get the patches it's more obvious how to handle out of the way first. Not considering those, HS/SR, or other patches with an already assigned reviewer, we're at 16 patches in the queue, and I've got 9 reviewer volunteers just so far today. Barring a flood of last-minute entries, if I can get each reviewer to handle one patch and a moderate percentage of them to handle two, that should be all it takes for this round. Will move the rest of the discussion here to just rrreviewers. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump enhancement proposal
Mark Hammonds mhammo...@omniti.com writes: 1. View Based Exports In addition to being able to dump specific tables, it seems useful to be able to quickly specify and dump entire views. Isn't this pretty much the same thing as your #2? A view in PG isn't materialized, it's nothing but a custom SELECT. 2. Custom Query Exports In my use of mysqldump, I found one feature very useful: the ability to execute a custom SELECT. . .WHERE statement and then dump only the results. This feature currently provides MySQL users with the ability to quickly and easily export very granular data subsets, and I see no reason why PostgreSQL users wouldn't benefit from the same capability. While it is true that this functionality can already be achieved in PostgreSQL using Copy, it seems to me that it would logically fit well as an extension to pg_dump, especially since many beginning and even some intermediate PostgreSQL users aren't aware of the alternatives. As you say, we already have this using COPY, and I don't agree that it would be a good idea to plaster it into pg_dump as well. pg_dump is intended for dumping and restoring data, not for ETL-type tasks. Furthermore, pg_dump is a overly complex beast already --- much more so than one could wish, for a tool that is absolutely fundamental to database reliability. Putting requirements on it that are well outside its charter seems like a short route to maintenance disaster. There has been some occasional chatter about developing one or more tools focused on ETL rather than dump/restore, and my thought is that this idea would fit better there. An ETL tool would not have the kind of requirements pg_dump has for coping with multiple server versions and knowing everything there is to know about database contents, so it seems like it could address new areas of functionality without a complexity explosion. You might want to check the archives for previous discussions --- I think the last go-round started with someone wanting to add an arbitrary WHERE filter to pg_dump dumps. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
On Thu, 12 Nov 2009 15:23:06 -0500 Andrew Dunstan wrote: Andreas 'ads' Scherbaum wrote: On Tue, 03 Nov 2009 10:32:17 -0800 Josh Berkus wrote: The same goes for other OSS projects. There's quite a few random OSS apps which were created on PG 7.4 and have never offered their users an upgrade path (Gnuworld comes to mind). They need an EOL announcement to get them motivated to upgrade. I know several customers who decided to move from 7.3 only after the EOL was announced. If 7.3 would not has see an EOL, they would never ever have moved to a newer version. Nobody that I have seen is arguing against EOLing 7.4. True. But as Josh pointed out: some people/projects/companies need more motivation to actually consider an upgrade at all. What I and others have been arguing is necessary to do EOL right is a serious amount of notice, by way of press releases etc. We can't expect users to keep polling our web site to see if there's an EOL. That means we need to prepare for an EOL months or a year in advance, ISTM. Months. The software will not stop working once we announced the EOL. And yes, i'm +1 for having a rule for EOL, like 5 versions are supported. Bye -- Andreas 'ads' Scherbaum German PostgreSQL User Group European PostgreSQL User Group - Board of Directors Volunteer Regional Contact, Germany - PostgreSQL Project -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plpgsql GUC variable: custom or built-in?
Tom Lane escribió: I wrote: Fair enough. I'll start writing the custom GUC then. While testing that, I noticed [...] With all this fallout, I think it would be cleaner to step back and make it a regular GUC variable, not custom. Pretending that plpgsql is somehow not an integral part of the system is not completely true anyway. Yes, we're playing favorites in the PL camp here, but so what? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
On Thu, 2009-11-12 at 23:09 +0100, Andreas 'ads' Scherbaum wrote: On Thu, 12 Nov 2009 15:23:06 -0500 Andrew Dunstan wrote: Andreas 'ads' Scherbaum wrote: On Tue, 03 Nov 2009 10:32:17 -0800 Josh Berkus wrote: The same goes for other OSS projects. There's quite a few random OSS apps which were created on PG 7.4 and have never offered their users an upgrade path (Gnuworld comes to mind). They need an EOL announcement to get them motivated to upgrade. I know several customers who decided to move from 7.3 only after the EOL was announced. If 7.3 would not has see an EOL, they would never ever have moved to a newer version. Nobody that I have seen is arguing against EOLing 7.4. True. But as Josh pointed out: some people/projects/companies need more motivation to actually consider an upgrade at all. We have discussed in the past EOLing 7.4 I thought at the end of this year. IMO 7.4 and 8.0 both need to be EOL. Can we just set a date and call it good? March 31st sounds good. Let's write up a quick announcement, add the letters EOL to the download pages and call it good. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] plpgsql GUC variable: custom or built-in?
Alvaro Herrera alvhe...@commandprompt.com writes: With all this fallout, I think it would be cleaner to step back and make it a regular GUC variable, not custom. Pretending that plpgsql is somehow not an integral part of the system is not completely true anyway. Yes, we're playing favorites in the PL camp here, but so what? True, but on the other hand, if plpgsql needs this to work nicely, then $YOURPL probably needs it too. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE...ALTER COLUMN vs inheritance
On Thu, Nov 12, 2009 at 13:55, Tom Lane t...@sss.pgh.pa.us wrote: I'd go for the first of those, for sure. Testing attnotnull is significantly cheaper than enforcing a generic constraint expression, and NOT NULL is a sufficiently common case to be worth worrying about optimizing it. When I looked at doing this, I thought about just using check constraints just for the book keeping and leaving attnotnull as it is. If would be easier, but it seemed quite ugly. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On the subject topic, I have to say that I don't see where Robert hasn't met the qualifications mentioned so far on this thread as required to promote someone to the committer level; are there some requirements which exist but haven't been mentioned? Regarding the specific issues below, which seem to be more about the related topic of drawing in new contributors... Robert Haas robertmh...@gmail.com wrote: Josh Berkus j...@agliodbs.com wrote: That's basically just it: Assume bashing is part of the process. I've heard this complaint from numerous first-time contributors, and have seen several talented people walk away from our project because of it. Even people who have stuck with us, such as Josh Tolley, have remarked on the hazing ritual inherent in getting a first-time contribution in. I used to feel this way, too. I'm not sure whether it's really worse at first, or whether it just seems worse a first until you get used to it. I would say that I don't often see anything I would consider as bashing, but that I have seen newcomers who appeared to want to help contribute to the project handled with less finesse than one could perhaps expect. It seems to me that those needing assistance (especially if they are in dire straights or are trying to convert to PostgreSQL) are treated very gently. Someone with a question, if it isn't something too obviously documented in the FAQ or the online manual, is given a polite and informative response. The responses get pretty blunt if the person offers suggestions which show that the person has (or assumes) technical competence, when those suggestions aren't acceptable to the community. At that point it takes a pretty thick skin to find some other way to contribute, rather than just walking away. Again, I wouldn't call it bashing, but a level of bluntness which could be disconcerting if the person hasn't followed these lists long enough to understand how it is intended. When someone submits a bad patch and I review it, they sometimes get two or three pages of feedback detailing inadequacies of various types. When done in a professional manner, which is typical of this list, I don't think there's any reasonable alternative. I don't see that as a problem. Well, there is one thing which must be frustrating. If I was seated near others who read this list, I would start a pool every time a newcomer submitted a large patch -- how long until someone tells them to break it into smaller patches, and how long until someone else tells them it makes no sense separated like that and should be put back together? the only possible solution is for each new contributor to have a mentor who sticks with them throughout the process of getting their first contribution accepted I don't know how common it is, but I felt that Bruce did that for me when I worked on the standard_conforming_strings TODO item for 8.2. (Perhaps part of the reason was that it was urgently needed to address problems we found in production after we converted a critical database to PostgreSQL; but whatever the reason, I was grateful for the help.) Having gone through that experience, I feel confident that such assistance for first-time contributors can make a difference both in the success of their efforts (which will then, presumably, benefit the community in general) and in their willingness to contribute in other ways. It would also help if people on this list *in general*, were to be a bit more consistent about phrasing criticism as constructive criticism. I've seen far too much how stupid are you? on this list for the good health of our developer community. I don't see very much of this at all, so maybe this is another area where it boils down to perspective. I don't see much of that either, although describing another's statement as nonsense or something similar isn't unheard of. It's not usually an unfair characterization, but some could take such bluntness as hostility. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE...ALTER COLUMN vs inheritance
Alex Hunsaker bada...@gmail.com writes: On Thu, Nov 12, 2009 at 13:55, Tom Lane t...@sss.pgh.pa.us wrote: I'd go for the first of those, for sure. Â Testing attnotnull is significantly cheaper than enforcing a generic constraint expression, and NOT NULL is a sufficiently common case to be worth worrying about optimizing it. When I looked at doing this, I thought about just using check constraints just for the book keeping and leaving attnotnull as it is. Yeah, you could definitely attack it like that. The code that fixes up attnotnull would have to look for check constraints that look like foo NOT NULL rather than something more instantly recognizable, but presumably ALTER TABLE is not a performance-critical path. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] actualised funcs typmod patch
Pavel Stehule escribió: Hello actualised version: enhance check inside sql function What is this against? It has a few suspicious chunks, such as *** *** 358,364 funcexpr-funcid = funcid; funcexpr-funcresulttype = rettype; ! funcexpr-funcresulttypmod = -1; // TODO STEHULE funcexpr-funcretset = retset; funcexpr-funcformat = COERCE_EXPLICIT_CALL; funcexpr-args = fargs; --- 365,371 funcexpr-funcid = funcid; funcexpr-funcresulttype = rettype; ! funcexpr-funcresulttypmod = rettypmod; funcexpr-funcretset = retset; funcexpr-funcformat = COERCE_EXPLICIT_CALL; funcexpr-args = fargs; This ended up in ./src/backend/parser/parse_func.c.rej There are other rejects and I'm not sure if I should fix them (at least the first couple look trivial, though it's against a bunch of code that last changed in Aug. last year) or ask for a better version ... -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 needs significant preparation for them. Announcing an EOL early in time would give them the required time before the version used disappears. So, should we announce it for June? No, it should be longer. June is practically around the corner as far as business planning is concerned. Make it a year. Since it's mid November, why not just say 2011? And yes, i'm +1 for having a rule for EOL, like 5 versions are supported. If we released on a consistent schedule, this *might* be possible. But we don't, so we can't say something like this. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 200911121815 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkr8lykACgkQvJuQZxSWSsi6xACdHU7xKgsfG+/zE2StXp97mdjC XGoAn3LvIjzh1RKmD9K0Zyrg9W3LuHxt =jCVo -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
