Re: [HACKERS] Synchronous replication - patch status inquiry
On 06/09/10 17:14, Simon Riggs wrote: On Mon, 2010-09-06 at 16:14 +0300, Heikki Linnakangas wrote: The standby is sending a stream of messages to the master with current LSN positions at the time the message is sent. Given a synchronous transaction, the master would wait until the feedback stream reports that the current transaction is in the past compared to the streamed last known synced one (or the same). That doesn't really answer the question: *when* does standby send back the acknowledgment? I think you should explain when you think this happens in your proposal. Are you saying that you think the standby should send back one message for every transaction? That you do not think we should buffer the return messages? For the sake of argument, yes that's what I was thinking. Now please explain how *you're* thinking it should work. You seem to be proposing a design for responsiveness to a single transaction, not for overall throughput. That's certainly a design choice, but it wouldn't be my recommendation that we did that. Sure, if there's more traffic, you can combine things. For example, if one fsync in the standby flushes more than one commit record, you only need one acknowledgment for all of them. But don't dodge the question! -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Tom Lane wrote: Michael Haggerty mhag...@alum.mit.edu writes: No, it is also possible to use cvs tag -b REL8_4_STABLE filename. In this case the file as it appears on the current branch is added to the specified branch, but CVS records no commit, author, or timestamp. So, if we're prepared to assert that we've never done that, could we have an option to cvs2git that is willing to use the first commit on a branch to represent the act of adding the file to the branch? I'm afraid this would be pretty far down on my long todo list. Somebody could use git filter-branch to make this change after the conversion, but I can't estimate how much work it would be. Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)
On 06/09/10 23:10, Markus Wanner wrote: Good. How about syscall overhead? One more write operation to the self-pipe per signal from within the signal handler and one read to actually clear the 'ready' state of the pipe from the waiter portion of the code, right? Right. Do we plan to replace all (or most) existing internal signals with these latches to circumvent the interruption problem? Or just the ones we need to wait for using pg_usleep()? At least the poll loops in bgwriter and walwriter need to be replaced if we want to fix the issue Tom mentioned earlier that the server currently wakes up periodically, waking up the CPU which wastes electricity. There's no hurry to replace other code. For Postgres-R, I'd probably have to extend it to call select() not only on the self-pipe, but on at least one other socket as well (to talk to the GCS). As long as that's possible, it looks like a more portable replacement for the pselect() variant that's currently in place there. Yeah, that would be a straightforward extension. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)
On 06/09/10 20:24, Markus Wanner wrote: On 09/06/2010 06:27 PM, Heikki Linnakangas wrote: + * It's important to reset the latch*before* checking if there's work to + * do. Otherwise, if someone sets the latch between the check and the + * ResetLatch call, you will miss it and Wait will block. Why doesn't WaitLatch() clear it? What's the use case for waiting for a latch and *not* wanting to reset it? Setting a latch that's already set is very fast, so you want to keep it set until the last moment. See the coding in walsender for example, it goes to some lengths to avoid clearing the latch until it's very sure there's no more work for it to do. That helps to keep the overhead in backends committing transactions low. (no-one has tried to measure that yet, though) -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] patch: tsearch - some memory diet
Hello this simple patch reduce a persistent allocated memory for tsearch ispell dictionaries. on 32bit from 27MB (3399 blocks) to 13MB (1564 blocks) on 64bit from 55MB to cca 27MB. Regards Pavel Stehule tsearch_group_alloc.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Triggers on VIEWs
On 7 September 2010 02:03, David Christensen da...@endpoint.com wrote: On Sep 5, 2010, at 3:09 AM, Dean Rasheed wrote: On 15 August 2010 18:38, Dean Rasheed dean.a.rash...@gmail.com wrote: Here is a WIP patch implementing triggers on VIEWs ... snip There are still a number of things left todo: - extend ALTER VIEW with enable/disable trigger commands - much more testing - documentation Attached is an updated patch with more tests and docs, and a few minor At least for me, there are some portions of the docs which could use some formatting changes in order to not be confusing grammatically; e.g., this was confusing to me on the first read: - trigger name. In the case of before triggers, the + trigger name. In the case of before and instead of triggers, the I realize this lack of formatting was inherited from the existing docs, but this would make more sense to me if this (and presumably the other related instances of before and after) were set apart with literal/ or similar. This is already in use in some places in this patch, so seems like the correct markup. Yeah, I think you're right. That chapter is the first place in the manual where the concepts of before/after/instead of are introduced. The first time it mentions them it uses firstterm, but then it doesn't use any markup for them for the remainder of the chapter. It looks like the rest of the manual uses literal+uppercase wherever they're mentioned, which does help readability, so it would make sense to bring the rest of that chapter into line with that convention. Thanks for looking at this. Cheers, Dean -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
Hi, On 05/27/2010 01:28 PM, Robert Haas wrote: How do you propose to guarantee that? ISTM that you have to either commit locally first, or send the commit to the remote first. Either way, the two events won't occur exactly simultaneously. I'm not getting the point of this discussion. As long as the database confirms the commit to the client only *after* having an ack from the standby and *after* committing locally, there's no problem. In any case, a server failure in between the commit request of the client and the commit confirmation for the client results in a client that can't tell if its transaction committed or not. So why do we care what to do first internally? Ideally, these two tasks should happen concurrently, IMO. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Interruptible sleeps (was Re: [HACKERS] CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)
On 09/07/2010 09:06 AM, Heikki Linnakangas wrote: Setting a latch that's already set is very fast, so you want to keep it set until the last moment. See the coding in walsender for example, it goes to some lengths to avoid clearing the latch until it's very sure there's no more work for it to do. That helps to keep the overhead in backends committing transactions low. (no-one has tried to measure that yet, though) Understood, thanks. Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] can we publish a aset interface?
Hello I would to use a special memory context for shared data (based on mmap) and I like impementation of aset. There is only one difference - aset is based on malloc and I would to use a mmap. malloc() is used in AllocSetContextCreate and AllocSetAlloc. These procedures should be overwritten, but other code and data structures can be used. This step can be useful for previous discuss about some more comfortable maintaining of shared memory. What do you think about? Regards Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, 2010-09-07 at 09:27 +0300, Heikki Linnakangas wrote: On 06/09/10 17:14, Simon Riggs wrote: On Mon, 2010-09-06 at 16:14 +0300, Heikki Linnakangas wrote: The standby is sending a stream of messages to the master with current LSN positions at the time the message is sent. Given a synchronous transaction, the master would wait until the feedback stream reports that the current transaction is in the past compared to the streamed last known synced one (or the same). That doesn't really answer the question: *when* does standby send back the acknowledgment? I think you should explain when you think this happens in your proposal. Are you saying that you think the standby should send back one message for every transaction? That you do not think we should buffer the return messages? For the sake of argument, yes that's what I was thinking. Now please explain how *you're* thinking it should work. The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. You seem to be proposing a design for responsiveness to a single transaction, not for overall throughput. That's certainly a design choice, but it wouldn't be my recommendation that we did that. Sure, if there's more traffic, you can combine things. For example, if one fsync in the standby flushes more than one commit record, you only need one acknowledgment for all of them. But don't dodge the question! Given that I've previously outlined the size and contents of request packets, their role and frequency I don't think I've dodged anything; in fact, I've almost outlined the whole design for you. I am coding something to demonstrate the important aspects I've espoused, just as you have done in the past when I didn't appreciate and/or understand your ideas. That seems like the best way forwards rather than wrangle through all the that can't work responses, which actually takes longer. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On 07/09/10 12:47, Simon Riggs wrote: The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. Ok, thank you. The obvious performance problem is that even if you define a transaction to use synchronization level 'recv', and there's no other concurrent transactions running, you actually need to wait until it's applied. If you have only one client, there is no difference between the levels, you always get the same performance hit you get with 'apply'. With more clients, you get some benefit, but there's still plenty of delays compared to the optimum. Also remember that there can be a very big gap between when a record is fsync'd and when it's applied, if the recovery needs to wait for a hot standby transaction to finish. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, 2010-09-07 at 13:11 +0300, Heikki Linnakangas wrote: The obvious performance problem Is not obvious at all, and you misunderstand again. This emphasises the need for me to show code. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
On Tue, Sep 7, 2010 at 4:53 AM, Pavel Stehule pavel.steh...@gmail.com wrote: I would to use a special memory context for shared data (based on mmap) and I like impementation of aset. There is only one difference - aset is based on malloc and I would to use a mmap. malloc() is used in AllocSetContextCreate and AllocSetAlloc. These procedures should be overwritten, but other code and data structures can be used. This step can be useful for previous discuss about some more comfortable maintaining of shared memory. What do you think about? What would this be good for? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, Sep 7, 2010 at 4:01 AM, Markus Wanner mar...@bluegap.ch wrote: In any case, a server failure in between the commit request of the client and the commit confirmation for the client results in a client that can't tell if its transaction committed or not. So why do we care what to do first internally? Ideally, these two tasks should happen concurrently, IMO. Right, definitely. The trouble is that if they happen concurrently, and there's a crash, you have to be prepared for the possibility that either one of the two has completed and the other is not. In practice, this means that the master and standby need to compare notes on the ending WAL location and whichever one is further advanced needs to stream the intervening records to the other. This would be an awesome feature, but it's hard, so for a first version, it makes sense to commit on the master first and then on the standby after the master is known done. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] returning multiple result sets from a stored procedure
On Sun, Sep 5, 2010 at 2:26 PM, Peter Eisentraut pete...@gmx.net wrote: On fre, 2010-09-03 at 16:18 -0400, Tom Lane wrote: Part of the reason it's sat on TODO is lack of consensus about how such a feature ought to look/work; particularly since most of the discussion about it has considered that it'd go along with stored procedures executing outside of transactions. I would probably be a mistake to tie these features together. They are tricky enough separately. Hm, do you think it would be possible to request manual transaction state when setting up the procedure (or reserve that ability for the future)? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
2010/9/7 Robert Haas robertmh...@gmail.com: On Tue, Sep 7, 2010 at 4:53 AM, Pavel Stehule pavel.steh...@gmail.com wrote: I would to use a special memory context for shared data (based on mmap) and I like impementation of aset. There is only one difference - aset is based on malloc and I would to use a mmap. malloc() is used in AllocSetContextCreate and AllocSetAlloc. These procedures should be overwritten, but other code and data structures can be used. This step can be useful for previous discuss about some more comfortable maintaining of shared memory. What do you think about? What would this be good for? I try to solve performance problems with czech tsearch. I checked serialization and deserialization, but this decrease load time only to 100ms (from 500) that is too much for us. After some gaming with mmap I thinking so there some chance to preallocate mmap memory, and then use a special memory context based on mmap instead of malloc. Teoretically I can copy aset interface - this module probably never be in core (this problem is probably local - only Czech), but it isn't nice. So I asking. Regards Pavel Stehule -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On 09/07/2010 02:16 PM, Robert Haas wrote: Right, definitely. The trouble is that if they happen concurrently, and there's a crash, you have to be prepared for the possibility that either one of the two has completed and the other is not. Understood. In practice, this means that the master and standby need to compare notes on the ending WAL location and whichever one is further advanced needs to stream the intervening records to the other. Not necessarily, no. Remember that the client didn't get a commit confirmation. So reverting might also be a correct solution (i.e. not violating the durability constraint). This would be an awesome feature, but it's hard, so for a first version, it makes sense to commit on the master first and then on the standby after the master is known done. The obvious downside of that is that latency adds up, instead of just being the max of the two operations. And that for normal operation. While at best it saves an un-confirmed transaction in the failure case. It might be harder to implement, yes. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Michael Haggerty mhag...@alum.mit.edu writes: Tom Lane wrote: So, if we're prepared to assert that we've never done that, could we have an option to cvs2git that is willing to use the first commit on a branch to represent the act of adding the file to the branch? I'm afraid this would be pretty far down on my long todo list. Fair enough. Somebody could use git filter-branch to make this change after the conversion, but I can't estimate how much work it would be. The conversion is already far better than I expected it would be when we were first discussing this switch, so my inclination is to just live with this one wart. I spent more time over the weekend comparing various branches' histories between cvs2cl and Max's repository. I found a lot of places where cvs2cl had problems :-(, but none where the git history could be blamed. I'm ready to sign off on this conversion process as being Good Enough, modulo two points: * Change the committer name assigned to manufactured commits, as already mentioned. * Please make the manufactured commits read cvs2git not cvs2svn. I don't want people wondering in future when it was we used SVN. AFAIK both of these are trivial configuration fixes. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 15:53, Tom Lane t...@sss.pgh.pa.us wrote: Michael Haggerty mhag...@alum.mit.edu writes: Tom Lane wrote: So, if we're prepared to assert that we've never done that, could we have an option to cvs2git that is willing to use the first commit on a branch to represent the act of adding the file to the branch? I'm afraid this would be pretty far down on my long todo list. Fair enough. Somebody could use git filter-branch to make this change after the conversion, but I can't estimate how much work it would be. The conversion is already far better than I expected it would be when we were first discussing this switch, so my inclination is to just live with this one wart. I spent more time over the weekend comparing various branches' histories between cvs2cl and Max's repository. I found a lot of places where cvs2cl had problems :-(, but none where the git history could be blamed. I'm ready to sign off on this conversion process as being Good Enough, modulo two points: * Change the committer name assigned to manufactured commits, as already mentioned. * Please make the manufactured commits read cvs2git not cvs2svn. I don't want people wondering in future when it was we used SVN. AFAIK both of these are trivial configuration fixes. I'm actually re-running a migration right now with this - and with the change to use rcs instead of cvs, to see if I can reproduce Max's proper repository. You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 9:56 AM, Magnus Hagander mag...@hagander.net wrote: You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? I think that the latest issue here is the issue of how files get added to branches, which we discussed before with pretty much the same set of conclusions. I'm not wild about the way that's getting converted, but I'm not sure I care enough about it to argue with Tom. However, I want to convince myself that the deletes we've done over the years have been properly handled. I need to look at Max's latest conversion and I'll look at yours as well. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fix for pg_upgrade's forcing pg_controldata into English
Bruce Momjian wrote: Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Tom Lane wrote: I certainly hope that pg_regress isn't freeing the strings it passes to putenv() ... pg_regress does not restore these settings (it says with C/English) so the code is different. That's not what I'm on about. You're trashing strings that are part of the live environment. It might accidentally fail to fail for you, if your version of free() doesn't immediately clobber the released storage, but it's still broken. Read the putenv() man page. + #ifndef WIN32 + char *envstr = (char *) pg_malloc(ctx, strlen(var) + + strlen(val) + 1); + + sprintf(envstr, %s=%s, var, val); + putenv(envstr); + pg_free(envstr); + #else + SetEnvironmentVariableA(var, val); + #endif The fact that there is no such free() in pg_regress is not an oversight or shortcut. Interesting. I did not know this and it was not clear from my manual page or FreeBSD's manual page, but Linux clearly does this. Updated patch attached. Applied to HEAD and 9.0.X. Thanks for the ideas/review. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
On Tue, Sep 7, 2010 at 9:27 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2010/9/7 Robert Haas robertmh...@gmail.com: On Tue, Sep 7, 2010 at 4:53 AM, Pavel Stehule pavel.steh...@gmail.com wrote: I would to use a special memory context for shared data (based on mmap) and I like impementation of aset. There is only one difference - aset is based on malloc and I would to use a mmap. malloc() is used in AllocSetContextCreate and AllocSetAlloc. These procedures should be overwritten, but other code and data structures can be used. This step can be useful for previous discuss about some more comfortable maintaining of shared memory. What do you think about? What would this be good for? I try to solve performance problems with czech tsearch. I checked serialization and deserialization, but this decrease load time only to 100ms (from 500) that is too much for us. After some gaming with mmap I thinking so there some chance to preallocate mmap memory, and then use a special memory context based on mmap instead of malloc. Teoretically I can copy aset interface - this module probably never be in core (this problem is probably local - only Czech), but it isn't nice. So I asking. I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, Sep 7, 2010 at 9:45 AM, Markus Wanner mar...@bluegap.ch wrote: On 09/07/2010 02:16 PM, Robert Haas wrote: Right, definitely. The trouble is that if they happen concurrently, and there's a crash, you have to be prepared for the possibility that either one of the two has completed and the other is not. Understood. In practice, this means that the master and standby need to compare notes on the ending WAL location and whichever one is further advanced needs to stream the intervening records to the other. Not necessarily, no. Remember that the client didn't get a commit confirmation. So reverting might also be a correct solution (i.e. not violating the durability constraint). In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. This would be an awesome feature, but it's hard, so for a first version, it makes sense to commit on the master first and then on the standby after the master is known done. The obvious downside of that is that latency adds up, instead of just being the max of the two operations. And that for normal operation. While at best it saves an un-confirmed transaction in the failure case. It might be harder to implement, yes. Yeah, I hope we'll get there eventually. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 15:53, Tom Lane t...@sss.pgh.pa.us wrote: Michael Haggerty mhag...@alum.mit.edu writes: Somebody could use git filter-branch to make this change after the conversion, but I can't estimate how much work it would be. The conversion is already far better than I expected it would be when we were first discussing this switch, so my inclination is to just live with this one wart. You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? If you want to try, and it doesn't take much time, go for it. I was just saying I wouldn't complain if we decide to live with it as-is. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would suffice to late abort committed transactions the master doesn't know about. However, I realize that undoing of WAL isn't something that's implemented (nor planned). So it's probably easier to forward the master in such a case. Yeah, I hope we'll get there eventually. Understood. Thanks. Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 16:16, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 15:53, Tom Lane t...@sss.pgh.pa.us wrote: Michael Haggerty mhag...@alum.mit.edu writes: Somebody could use git filter-branch to make this change after the conversion, but I can't estimate how much work it would be. The conversion is already far better than I expected it would be when we were first discussing this switch, so my inclination is to just live with this one wart. You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? If you want to try, and it doesn't take much time, go for it. I was just saying I wouldn't complain if we decide to live with it as-is. Ok. Do we have a way of identifying them - e.g. is it all the commits with a certain commit msg? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
Pavel Stehule pavel.steh...@gmail.com writes: I would to use a special memory context for shared data (based on mmap) and I like impementation of aset. There is only one difference - aset is based on malloc and I would to use a mmap. malloc() is used in AllocSetContextCreate and AllocSetAlloc. These procedures should be overwritten, but other code and data structures can be used. This step can be useful for previous discuss about some more comfortable maintaining of shared memory. What do you think about? If you're proposing factoring aset.c into two levels, I don't think so. That code is already a tremendous performance hot-spot and introducing any more inefficiency into it doesn't seem like a good idea. Especially not for shared memory allocation, which is a feature that still has no buy-in. Also, you'd need to do more than just replace malloc: you'd need to add locking capability. That would make the code even uglier, and slower, if it has to support locking or no locking dynamically. Use the mcxt.c switch. That's what it's there for. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
Markus Wanner wrote: On 09/07/2010 02:16 PM, Robert Haas wrote: practice, this means that the master and standby need to compare notes on the ending WAL location and whichever one is further advanced needs to stream the intervening records to the other. Not necessarily, no. Remember that the client didn't get a commit confirmation. So reverting might also be a correct solution (i.e. not violating the durability constraint). In that situation, wouldn't it be possible that a different client queried the slave and already saw the result of that transaction which would later be rolled back? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 09:27 +0300, Heikki Linnakangas wrote: For the sake of argument, yes that's what I was thinking. Now please explain how *you're* thinking it should work. The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. That premise is completely false. SR does not send WAL in page units. If it did, it would have the same performance problems as the old WAL-file-at-a-time implementation, just with slightly smaller granularity. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
Robert Haas robertmh...@gmail.com writes: On Tue, Sep 7, 2010 at 9:27 AM, Pavel Stehule pavel.steh...@gmail.com wrote: I try to solve performance problems with czech tsearch. I checked serialization and deserialization, but this decrease load time only to 100ms (from 500) that is too much for us. After some gaming with mmap I thinking so there some chance to preallocate mmap memory, and then use a special memory context based on mmap instead of malloc. Teoretically I can copy aset interface - this module probably never be in core (this problem is probably local - only Czech), but it isn't nice. So I asking. I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. More to the point, this entire approach to speeding up dictionary loading has already been proposed and rejected, and it'll get rejected again if it's submitted. The conclusion of the previous discussion was that we should build precompiled dictionaries, using some pointer-free representation, which would be stored in files that could be either mmap'd in or just read in if running on a platform lacking mmap. There is no need for any shmem allocator in that implementation. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: tsearch dictionary initialization hook
Hm, what is aim of this hook? It looks like a wrapper of dictionary init method. I propose a new hook type - that helps with controlling a life cycle of some tsearch dictionaries. This hook has minimal impact on performance - it's called once per session for one tsearch configuration. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] knngist - 0.8
http://www.sigaev.ru/misc/builtin_knngist_core-0.8.gz http://www.sigaev.ru/misc/builtin_knngist_itself-0.8.gz http://www.sigaev.ru/misc/builtin_knngist_proc-0.8.gz http://www.sigaev.ru/misc/builtin_knngist_contrib_pg_trgm-0.8.gz http://www.sigaev.ru/misc/builtin_knngist_contrib_btree_gist-0.8.gz New version, synced with CVS HEAD http://www.sigaev.ru/misc/builtin_knngist_itself-0.8.2.gz http://www.sigaev.ru/misc/builtin_knngist_contrib_btree_gist-0.8.1.gz AFAICS, these patches include no documentation. That's pretty much a fatal flaw for a feature of this magnitude. At an absolute minimum, you need to update the system catalog documentation and the documentation on CREATE / ALTER OPERATOR CLASS. There might be some other places that need to be touched, also. Oleg promised to do that + if (opform-oprresult == BOOLOID) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), +errmsg(index ordering operators must not return boolean))); My first thought was that this code was there to prevent people from doing the wrong thing by accident. But I have a niggling feeling that you're actually relying on this for the correctness of the system. I hope I'm wrong, because I don't think that would be a very good idea. This play is around do we really want to have support of boolean-distance in GiST? I think no, because it's a strange idea to measure distance in true/false measurement units. I can't imagine such real-life distance definition and never heard about that. Next, pg_amop_opr_fam_index requires uniqueness of operation in operation family and a lot of places in planner believes in that. Suppose, changing that requires a lot of work which has the single aim to support boolean distance in ORDER BY clause. The GIST code code use more comments; and perhaps the names of some of the functions and structures could be chosen to be more descriptive. I think that what used to be called GISTSearchStack has apparently been replaced with DataPointer; it's not obvious to me that it's good to change the name, but if it is I don't think DataPointer is a good GISTSearchStack is replaced by RBTree (GISTScanOpaqueData-stack), tree's nodes contain a StackElem struct which represents list of pointers at the same distance. Each pointer could be a pointer to the inner index's page or to the heap's tuple and this struct is a DataPointer. Note, list of DataPointer in StackElem struct is organized by non-obvious way: we keep pointer to the head of list and pointer to the middle of list. New pointer-to-heap is inserted in the beginning of list, pointers-to-index-page - in the middle. That's done because we would like to: 1) pop pointers-to-heap as fast as possible, before any pointers-to-index-page 2) pop pointers-to-index-page to deep page (which is closer to leaf pages) first. That's good for KNN performance and emulates classical first-depth search in ordinary search. choice. gistindex_keytest has been replaced (sort of) by processIndexTuple, which again seems more generic than what it replaced. Renamed, comments are improved Minor nit: the word shoould is mis-spelled. fixed BTW, now consistentFn is able to manage tree traversal - even for for ordinary search, GiST will choose child page with minimal distance. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, 2010-09-07 at 16:31 +0200, Markus Wanner wrote: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would suffice to late abort committed transactions the master doesn't know about. The slave *might* be ahead of the master. And if it is, the case we're discussing is where the master just crashed and *might* not even be coming back at all, at least for a while. The standby does differ from master, but with the master down I don't regard that as a useful statement. If we wait for fsync on master and then transfer to standby the times are additive. If we do them concurrently the response times will be the maximum response time of fsync/transfer, as Markus observes. ISTM that most people would be more interested in reducing response times by ~50% rather than in being exactly correct in an edge case. So we should be planning that as a robustness option, not it cannot be done, which seems to be echoing around to much for my liking. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 16:16, Tom Lane t...@sss.pgh.pa.us wrote: If you want to try, and it doesn't take much time, go for it. I was just saying I wouldn't complain if we decide to live with it as-is. Ok. Do we have a way of identifying them - e.g. is it all the commits with a certain commit msg? Look for This commit was manufactured by cvs2svn to create branch ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
Markus Wanner mar...@bluegap.ch writes: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would suffice to late abort committed transactions the master doesn't know about. Oh yes it is. If the slave replays WAL that didn't happen on the master, it might for instance have heap tuples in TID slots that are empty on the master, or index pages laid out differently from the master. Trying to apply additional WAL from the master will fail badly. We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. If you feel that decoupling WAL application is absolutely essential to have a credible feature, then you'd better bite the bullet and start working on the ship-WAL-back code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
Excerpts from Robert Haas's message of mar sep 07 10:13:12 -0400 2010: I try to solve performance problems with czech tsearch. I checked serialization and deserialization, but this decrease load time only to 100ms (from 500) that is too much for us. After some gaming with mmap I thinking so there some chance to preallocate mmap memory, and then use a special memory context based on mmap instead of malloc. Teoretically I can copy aset interface - this module probably never be in core (this problem is probably local - only Czech), but it isn't nice. So I asking. I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. Hmm, surely you could store offsets instead of absolute pointers. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 17:07, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 16:16, Tom Lane t...@sss.pgh.pa.us wrote: If you want to try, and it doesn't take much time, go for it. I was just saying I wouldn't complain if we decide to live with it as-is. Ok. Do we have a way of identifying them - e.g. is it all the commits with a certain commit msg? Look for This commit was manufactured by cvs2svn to create branch ... Ok, found a bunch of those (78 to be exact). And the issue with them is we want to change the commit author on them to be whomever made the first commit on the branch *after* that? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, 2010-09-07 at 10:47 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 09:27 +0300, Heikki Linnakangas wrote: For the sake of argument, yes that's what I was thinking. Now please explain how *you're* thinking it should work. The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. That premise is completely false. SR does not send WAL in page units. If it did, it would have the same performance problems as the old WAL-file-at-a-time implementation, just with slightly smaller granularity. There's no dependence on pages in that proposal, so don't understand. What aspect of the above would you change? and to what? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, 2010-09-07 at 11:17 -0400, Tom Lane wrote: Markus Wanner mar...@bluegap.ch writes: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would suffice to late abort committed transactions the master doesn't know about. Oh yes it is. If the slave replays WAL that didn't happen on the master, it might for instance have heap tuples in TID slots that are empty on the master, or index pages laid out differently from the master. Trying to apply additional WAL from the master will fail badly. We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. If you feel that decoupling WAL application is absolutely essential to have a credible feature, then you'd better bite the bullet and start working on the ship-WAL-back code. Why not just failover? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 17:07, Tom Lane t...@sss.pgh.pa.us wrote: Look for This commit was manufactured by cvs2svn to create branch ... Ok, found a bunch of those (78 to be exact). And the issue with them is we want to change the commit author on them to be whomever made the first commit on the branch *after* that? What I'd like is for those commits to vanish from the git log entirely. In a practical sense, what you should probably do is for each file mentioned in such a commit, cause the file's addition to the branch to become part of the first regular commit on the branch that touched that file. In the CVS history, at least, there always is such a commit (since we never did the cvs tag -b thing). I am not sure though whether the converted git history includes a touch of the file in that commit, if the version committed into the branch is identical to what was on HEAD. Michael, can you comment on that point? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On 09/07/2010 04:47 PM, Ron Mayer wrote: In that situation, wouldn't it be possible that a different client queried the slave and already saw the result of that transaction which would later be rolled back? Good point, yes. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On 07/09/10 16:21, Magnus Hagander wrote: On Tue, Sep 7, 2010 at 17:07, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 16:16, Tom Lane t...@sss.pgh.pa.us wrote: If you want to try, and it doesn't take much time, go for it. I was just saying I wouldn't complain if we decide to live with it as-is. Ok. Do we have a way of identifying them - e.g. is it all the commits with a certain commit msg? Look for This commit was manufactured by cvs2svn to create branch ... Ok, found a bunch of those (78 to be exact). And the issue with them is we want to change the commit author on them to be whomever made the first commit on the branch *after* that? I would say you emphatically don't want to do that, because they can contain more changes that were unrelated to that author. The logic, as I understand it from Michael's explanation of cvs2git's guts, is to flush out any pending add to branch because of implicit appearance of a branch tag operations when something other change is about to occur on the destination branch. So unrelated stuff can get batched together. Personally, the idea of trying to use git-filter-branch to make what cvs2git currently gives you more sensible scares me silly. I think the approach should be to use it as is, or improve cvs2git. Another glitch that might be worth fixing before you convert is the way that cvs2git says This commit was manufactured by cvs2svn to create branch, when it actually means manufactured to incrementally create the branch state as it appears in CVS - i.e. many of these commits actually update an existing branch. Just as soon as I can figure out how to cleanly fit that into cvs2git's structure, I want it to change the word create to update in most of those commits. Max. signature.asc Description: OpenPGP digital signature
[HACKERS] PgWest 2010 CFP extended (one week)
For one week guys... https://www.postgresqlconference.org/2010/west/cfp/ -- PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 10:47 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. That premise is completely false. SR does not send WAL in page units. If it did, it would have the same performance problems as the old WAL-file-at-a-time implementation, just with slightly smaller granularity. There's no dependence on pages in that proposal, so don't understand. Oh, well you certainly didn't explain it well then. What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to sleep waiting for the next chunk. That's fine as far as the protocol goes, but I'm not convinced that it really does all that much in terms of improving performance. You still have the problem that the master has to fsync its WAL before it can send it to the slave. Also, the slave won't know whether it ought to fsync its own WAL before replying. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 11:17 -0400, Tom Lane wrote: We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. If you feel that decoupling WAL application is absolutely essential to have a credible feature, then you'd better bite the bullet and start working on the ship-WAL-back code. Why not just failover? Guaranteed failover is another large piece we don't have. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Max Bowsher m...@f2s.com writes: Personally, the idea of trying to use git-filter-branch to make what cvs2git currently gives you more sensible scares me silly. I'm not excited about it either --- but if Magnus wants to experiment, no harm trying. Another glitch that might be worth fixing before you convert is the way that cvs2git says This commit was manufactured by cvs2svn to create branch, when it actually means manufactured to incrementally create the branch state as it appears in CVS - i.e. many of these commits actually update an existing branch. Just as soon as I can figure out how to cleanly fit that into cvs2git's structure, I want it to change the word create to update in most of those commits. I thought all of those message texts were taken from the configuration file. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
On Tue, Sep 7, 2010 at 11:18 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar sep 07 10:13:12 -0400 2010: I try to solve performance problems with czech tsearch. I checked serialization and deserialization, but this decrease load time only to 100ms (from 500) that is too much for us. After some gaming with mmap I thinking so there some chance to preallocate mmap memory, and then use a special memory context based on mmap instead of malloc. Teoretically I can copy aset interface - this module probably never be in core (this problem is probably local - only Czech), but it isn't nice. So I asking. I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. Hmm, surely you could store offsets instead of absolute pointers. Surely you could. But then where does palloc come in? As Tom said upthread, the right thing to do here is to create a pre-compiler that outputs a pointer-free representation which you can then mmap(). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
I wrote: Magnus Hagander mag...@hagander.net writes: Ok, found a bunch of those (78 to be exact). What I'd like is for those commits to vanish from the git log entirely. In a practical sense, what you should probably do is for each file mentioned in such a commit, cause the file's addition to the branch to become part of the first regular commit on the branch that touched that file. In the CVS history, at least, there always is such a commit (since we never did the cvs tag -b thing). I am not sure though whether the converted git history includes a touch of the file in that commit, Given that there are only 78 such commits, it would not take too long to manually prepare a list of which commit each file addition should get moved into. Would that be a more sensible approach than trying to extract the information from the git log? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, Sep 7, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: Oh, well you certainly didn't explain it well then. What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to sleep waiting for the next chunk. That's fine as far as the protocol goes, but I'm not convinced that it really does all that much in terms of improving performance. You still have the problem that the master has to fsync its WAL before it can send it to the slave. We have that problem in all of these proposals, don't we? We certainly have no infrastructure to handle the slave getting ahead of the master in the WAL stream. Also, the slave won't know whether it ought to fsync its own WAL before replying. Right. And whether it ought to replay it before replying. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
Hi, On 09/07/2010 05:17 PM, Tom Lane wrote: Oh yes it is. If the slave replays WAL that didn't happen on the master, it might for instance have heap tuples in TID slots that are empty on the master, or index pages laid out differently from the master. Trying to apply additional WAL from the master will fail badly. Sure. Reverting to the master's state would be required to be able to safely proceed. Granted, that's far from simple. Robert's argument about read queries on the standby convinced me, that you always need to recover to the node with the newest transactions applied (i.e. better advance rather than revert). Making sure the standby can't ever be ahead of the master node certainly is the simplest way to guarantee that. At its cost for normal operation, though. How about a master failure which leads to a fail-over, immediately followed by a failure of that former standby (and now a master)? The old master might then be in the very same situation: having WAL applied that the new master doesn't. Do we require former masters to fetch a base backup? How does it know the difference, once it gets back up? We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. Agreed. (And it's worth pointing out that this approach has a pretty nasty requirement for a full-cluster crash: all nodes that were synchronously replicated to need to come back up after such a crash, so as to be able to reliably determine which has the newest transaction). If you feel that decoupling WAL application is absolutely essential to have a credible feature, then you'd better bite the bullet and start working on the ship-WAL-back code. My feeling is that WAL is the wrong format to do replication. But that's a another story. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, 2010-09-07 at 11:41 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 10:47 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. That premise is completely false. SR does not send WAL in page units. If it did, it would have the same performance problems as the old WAL-file-at-a-time implementation, just with slightly smaller granularity. There's no dependence on pages in that proposal, so don't understand. Oh, well you certainly didn't explain it well then. What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to sleep waiting for the next chunk. That's fine as far as the protocol goes, but I'm not convinced that it really does all that much in terms of improving performance. You still have the problem that the master has to fsync its WAL before it can send it to the slave. Also, the slave won't know whether it ought to fsync its own WAL before replying. Yes, apart from last sentence. Please wait for the code. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, Sep 7, 2010 at 11:06 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-09-07 at 16:31 +0200, Markus Wanner wrote: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would suffice to late abort committed transactions the master doesn't know about. The slave *might* be ahead of the master. And if it is, the case we're discussing is where the master just crashed and *might* not even be coming back at all, at least for a while. The standby does differ from master, but with the master down I don't regard that as a useful statement. If we wait for fsync on master and then transfer to standby the times are additive. If we do them concurrently the response times will be the maximum response time of fsync/transfer, as Markus observes. ISTM that most people would be more interested in reducing response times by ~50% rather than in being exactly correct in an edge case. So we should be planning that as a robustness option, not it cannot be done, which seems to be echoing around to much for my liking. People who are more concerned about performance than robustness aren't going to use sync rep in the first place. They're going to run it in async, which will improve performance by FAR more than you'll ever be able to manage by deciding that you don't care about handling some of the failure cases correctly. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] gincostestimate
I also dropped the use of rd_amcache, instead having ginGetStats() Ok, I'm agree I didn't do anything about the questionable equations in gincostestimate. Those need to either be fixed, or documented as to why they're correct. Other than that I think this could be committed. Fixed, and slightly reworked to be more clear. Attached patch is based on your patch. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ gincostestimate-0.24.gz Description: Unix tar archive -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, Sep 7, 2010 at 11:59 AM, Simon Riggs si...@2ndquadrant.com wrote: What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to sleep waiting for the next chunk. That's fine as far as the protocol goes, but I'm not convinced that it really does all that much in terms of improving performance. You still have the problem that the master has to fsync its WAL before it can send it to the slave. Also, the slave won't know whether it ought to fsync its own WAL before replying. Yes, apart from last sentence. Please wait for the code. So, we're going around and around in circles here because you're repeatedly refusing to explain how the slave will know WHEN to send acknowledgments back to the master without knowing which sync rep level is in use. It seems to be perfectly evident to everyone else here that there are only two ways for this to work: either the value is configured on the standby, or there's a registration system on the master and the master tells the standby its wishes. Instead of asking the entire community to wait for an unspecified period of time for you to write code that will handle this in an unspecified way, how about answering the question? We've wasted far too much time arguing about this already. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] function_name.parameter_name
Howdy, Anyone ever thought to try to add $subject to PL/pgSQL? Someone left a [comment][] on the PGXN blog about how this is a supported syntax for using named parameters on Oracle. The context is to avoid conflicts between variable names and column names by function-qualifyin the former and table-qualifying the latter. [comment]: http://blog.pgxn.org/post/1053165383/alias-in-vogue#dsq-comment-75687336 Would this be do-able in PL/pgSQL? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: tsearch - some memory diet
on 32bit from 27MB (3399 blocks) to 13MB (1564 blocks) on 64bit from 55MB to cca 27MB. Good results. But, I think, there are more places in ispell to use hold_memory(): - affixes and affix tree - regis (REGex for ISpell, regis.c) -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] function_name.parameter_name
David E. Wheeler david.whee...@pgexperts.com writes: Anyone ever thought to try to add $subject to PL/pgSQL? How does $subject differ from what we already do? See http://www.postgresql.org/docs/9.0/static/plpgsql-structure.html particularly this: Note: There is actually a hidden outer block surrounding the body of any PL/pgSQL function. This block provides the declarations of the function's parameters (if any), as well as some special variables such as FOUND (see Section 39.5.5). The outer block is labeled with the function's name, meaning that parameters and special variables can be qualified with the function's name. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] function_name.parameter_name
On Sep 7, 2010, at 9:35 AM, Tom Lane wrote: How does $subject differ from what we already do? See http://www.postgresql.org/docs/9.0/static/plpgsql-structure.html particularly this: Note: There is actually a hidden outer block surrounding the body of any PL/pgSQL function. This block provides the declarations of the function's parameters (if any), as well as some special variables such as FOUND (see Section 39.5.5). The outer block is labeled with the function's name, meaning that parameters and special variables can be qualified with the function's name. Well I'll be damned. I never knew about this! So I can get rid of those aliases! http://github.com/theory/pgxn-manager/commit/e5add190ff5358a0b2ede64b62616491be454c50 Thanks Tom, I had *no idea* about this. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: tsearch - some memory diet
On 07/09/10 19:27, Teodor Sigaev wrote: on 32bit from 27MB (3399 blocks) to 13MB (1564 blocks) on 64bit from 55MB to cca 27MB. Good results. But, I think, there are more places in ispell to use hold_memory(): - affixes and affix tree - regis (REGex for ISpell, regis.c) A more general solution would be to have a new MemoryContext implementation that does the same your patch does. Ie. instead of tracking each allocation, just allocate a big chunk, and have palloc() return the next n free bytes from it, like a stack. pfree() would obviously not work, but wholesale MemoryContextDelete of the whole memory context would. I remember I actually tried this years ago, trying to reduce the overhead of parsing IIRC. The parser also does a lot of small allocations that are not individually pfree'd. And I think it helped a tiny bit, but I didn't pursue it further. But if there's many places where it would help, then it might well be worth it. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: tsearch - some memory diet
2010/9/7 Teodor Sigaev teo...@sigaev.ru: on 32bit from 27MB (3399 blocks) to 13MB (1564 blocks) on 64bit from 55MB to cca 27MB. Good results. But, I think, there are more places in ispell to use hold_memory(): - affixes and affix tree - regis (REGex for ISpell, regis.c) yes, but minimally for Czech dictionary other places are not important. It's decrease 1MB more. Last month I moved all these unreleased parts and it's not important. But can be interesting check it for other languages than Czech. Regards Pavel Stehule -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On 07/09/10 16:47, Tom Lane wrote: Max Bowsher m...@f2s.com writes: Personally, the idea of trying to use git-filter-branch to make what cvs2git currently gives you more sensible scares me silly. I'm not excited about it either --- but if Magnus wants to experiment, no harm trying. Another glitch that might be worth fixing before you convert is the way that cvs2git says This commit was manufactured by cvs2svn to create branch, when it actually means manufactured to incrementally create the branch state as it appears in CVS - i.e. many of these commits actually update an existing branch. Just as soon as I can figure out how to cleanly fit that into cvs2git's structure, I want it to change the word create to update in most of those commits. I thought all of those message texts were taken from the configuration file. Yes, but currently these two cases both reference the same entry in the configuration file. Max. signature.asc Description: OpenPGP digital signature
Re: [HACKERS] patch: tsearch - some memory diet
2010/9/7 Heikki Linnakangas heikki.linnakan...@enterprisedb.com: On 07/09/10 19:27, Teodor Sigaev wrote: on 32bit from 27MB (3399 blocks) to 13MB (1564 blocks) on 64bit from 55MB to cca 27MB. Good results. But, I think, there are more places in ispell to use hold_memory(): - affixes and affix tree - regis (REGex for ISpell, regis.c) A more general solution would be to have a new MemoryContext implementation that does the same your patch does. Ie. instead of tracking each allocation, just allocate a big chunk, and have palloc() return the next n free bytes from it, like a stack. pfree() would obviously not work, but wholesale MemoryContextDelete of the whole memory context would. I remember I actually tried this years ago, trying to reduce the overhead of parsing IIRC. The parser also does a lot of small allocations that are not individually pfree'd. And I think it helped a tiny bit, but I didn't pursue it further. But if there's many places where it would help, then it might well be worth it. I sent patch last year - simpleAllocator, and this idea was rejected. But now I dislike this idea too. This is unclear, and some forgotten free calling can do problems. This is more simple, more readable and secure. Regards Pavel Stehule -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: tsearch - some memory diet
A more general solution would be to have a new MemoryContext implementation that does the same your patch does. Ie. instead of tracking each allocation, just allocate a big chunk, and have palloc() return the next n free bytes from it, like a stack. pfree() would obviously not work, but wholesale MemoryContextDelete of the whole memory context would. repalloc() will not work too. Such implementation should have possibility to debug memory allocation/management by using some kind of red-zones or CLOBBER_FREED_MEMORY/MEMORY_CONTEXT_CHECKING -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] proposal: tsearch dictionary initialization hook
Hello 2010/9/7 Teodor Sigaev teo...@sigaev.ru: Hm, what is aim of this hook? It looks like a wrapper of dictionary init method. If I use a mmap for shared dictionary, then I have to prealloc and maybe preread dictionary - it can be done in external module. But I have to join preloaded dictionary to requested dictionary. This hook allows this relation - and it's general - I don't need any special support in ispell dictionary. Regards Pavel I propose a new hook type - that helps with controlling a life cycle of some tsearch dictionaries. This hook has minimal impact on performance - it's called once per session for one tsearch configuration. -- Teodor Sigaev E-mail: teo...@sigaev.ru WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
2010/9/7 Robert Haas robertmh...@gmail.com: On Tue, Sep 7, 2010 at 9:27 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2010/9/7 Robert Haas robertmh...@gmail.com: On Tue, Sep 7, 2010 at 4:53 AM, Pavel Stehule pavel.steh...@gmail.com wrote: I would to use a special memory context for shared data (based on mmap) and I like impementation of aset. There is only one difference - aset is based on malloc and I would to use a mmap. malloc() is used in AllocSetContextCreate and AllocSetAlloc. These procedures should be overwritten, but other code and data structures can be used. This step can be useful for previous discuss about some more comfortable maintaining of shared memory. What do you think about? What would this be good for? I try to solve performance problems with czech tsearch. I checked serialization and deserialization, but this decrease load time only to 100ms (from 500) that is too much for us. After some gaming with mmap I thinking so there some chance to preallocate mmap memory, and then use a special memory context based on mmap instead of malloc. Teoretically I can copy aset interface - this module probably never be in core (this problem is probably local - only Czech), but it isn't nice. So I asking. I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. you can, but you have to do preallocation and you have to use a FIXED flag. Pavel -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Tue, Sep 7, 2010 at 17:07, Tom Lane t...@sss.pgh.pa.us wrote: Look for This commit was manufactured by cvs2svn to create branch ... Ok, found a bunch of those (78 to be exact). And the issue with them is we want to change the commit author on them to be whomever made the first commit on the branch *after* that? What I'd like is for those commits to vanish from the git log entirely. In a practical sense, what you should probably do is for each file mentioned in such a commit, cause the file's addition to the branch to become part of the first regular commit on the branch that touched that file. In the CVS history, at least, there always is such a commit (since we never did the cvs tag -b thing). I am not sure though whether the converted git history includes a touch of the file in that commit, if the version committed into the branch is identical to what was on HEAD. Michael, can you comment on that point? If the situation is a file that had a branch tag added to it after the branch was first created, then there is a git commit corresponding to that event that consists of the addition of that file with no history. This commit might also include the addition of other files to the branch, but should not include any file content changes. It seems to me that in your case such commits could be grafted over: *---*---*---* \ A---B---C---D E.g., if C is one of these special manufactured commits, then you could use git grafts to change the parent of D from C to B, then bake in the change with git filter-branch. This would make C inaccessible and subject to garbage collection. But please check by hand to make sure that this makes sense; for example, it could be that other branches in the neighborhood make the excision impossible. Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Max Bowsher m...@f2s.com writes: On 07/09/10 16:47, Tom Lane wrote: Max Bowsher m...@f2s.com writes: ... Just as soon as I can figure out how to cleanly fit that into cvs2git's structure, I want it to change the word create to update in most of those commits. I thought all of those message texts were taken from the configuration file. Yes, but currently these two cases both reference the same entry in the configuration file. Oh, I misunderstood the most bit ;-) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Michael Haggerty mhag...@alum.mit.edu writes: Tom Lane wrote: What I'd like is for those commits to vanish from the git log entirely. It seems to me that in your case such commits could be grafted over: *---*---*---* \ A---B---C---D E.g., if C is one of these special manufactured commits, then you could use git grafts to change the parent of D from C to B, then bake in the change with git filter-branch. This would make C inaccessible and subject to garbage collection. Hmm, I see. This depends on the fact that git commits reference filesystem states and not deltas, correct? So it does actually make sense to just delete that commit from the history. I was concerned that it'd invalidate later commits, but I guess it doesn't. But please check by hand to make sure that this makes sense; for example, it could be that other branches in the neighborhood make the excision impossible. Since we weren't doing merging, nor branching off from back branches, I'm having a hard time seeing how there'd be any risk there. Is there a case I'm missing? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: tsearch - some memory diet
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: A more general solution would be to have a new MemoryContext implementation that does the same your patch does. Ie. instead of tracking each allocation, just allocate a big chunk, and have palloc() return the next n free bytes from it, like a stack. pfree() would obviously not work, but wholesale MemoryContextDelete of the whole memory context would. The trick with that is to not crash horribly if pfree or GetMemoryChunkSpace or GetMemoryChunkContext is applied to such a chunk. Perhaps we can live without that requirement, but it greatly limits the safe usage of such a context type. In the particular case here, the dictionary structures could probably safely use such a context type, but I'm not sure it's worth bothering if the long-term plan is to implement a precompiler. There would be no need for this after the precompiled representation is installed, because that'd just be one big hunk of memory anyway. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
Hi, On 09/07/2010 06:00 PM, Robert Haas wrote: People who are more concerned about performance than robustness aren't going to use sync rep in the first place. I'm advocating sync (or eager, FWIW) replication for years, now. One of the hardest preconception I'm always confronted with is: this must perform poorly! Whether or not that's true depends, but my point is: people who need that level of robustness certainly care about performance as well. Telling them to use async replication instead is not an option. (The ability to mix sync and async replication per transaction is one, BTW). They're going to run it in async, which will improve performance by FAR more than you'll ever be able to manage by deciding that you don't care about handling some of the failure cases correctly. Running in async and then trying to achieve the required level of robustness in the application layer pretty certainly performs worse than a good sync replication implementation. Async only wins if you really don't care about the loss of transactions in the case of a failure. In every other case, robustness is better taken care of by the database system itself, IMO. That being said, I certainly agree to do things step by step. And the ability to write to WAL and wait for ack from a standby concurrently can (and probably should) be considered an optimization, yes. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On 09/07/2010 05:55 PM, Markus Wanner wrote: Robert's argument Sorry, I meant Ron. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, 2010-09-07 at 12:07 -0400, Robert Haas wrote: On Tue, Sep 7, 2010 at 11:59 AM, Simon Riggs si...@2ndquadrant.com wrote: What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to sleep waiting for the next chunk. That's fine as far as the protocol goes, but I'm not convinced that it really does all that much in terms of improving performance. You still have the problem that the master has to fsync its WAL before it can send it to the slave. Also, the slave won't know whether it ought to fsync its own WAL before replying. Yes, apart from last sentence. Please wait for the code. So, we're going around and around in circles here because you're repeatedly refusing to explain how the slave will know WHEN to send acknowledgments back to the master without knowing which sync rep level is in use. It seems to be perfectly evident to everyone else here that there are only two ways for this to work: either the value is configured on the standby, or there's a registration system on the master and the master tells the standby its wishes. Instead of asking the entire community to wait for an unspecified period of time for you to write code that will handle this in an unspecified way, how about answering the question? We've wasted far too much time arguing about this already. Every time I explain anything, I get someone run around shouting but that can't work!. I'm sorry, but again your logic is poor and the bias against properly considering viable alternatives is the only thing perfectly evident. So yes, I agree, it is a waste of time discussing it until I show working code. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On 07/09/10 18:16, Tom Lane wrote: Michael Haggerty mhag...@alum.mit.edu writes: Tom Lane wrote: What I'd like is for those commits to vanish from the git log entirely. It seems to me that in your case such commits could be grafted over: *---*---*---* \ A---B---C---D E.g., if C is one of these special manufactured commits, then you could use git grafts to change the parent of D from C to B, then bake in the change with git filter-branch. This would make C inaccessible and subject to garbage collection. Hmm, I see. This depends on the fact that git commits reference filesystem states and not deltas, correct? So it does actually make sense to just delete that commit from the history. I was concerned that it'd invalidate later commits, but I guess it doesn't. It wouldn't - except for the fact that cvs2git batches such manufactured commits such that there is no guarantee that a single manufactured commit pertains only to files in the commit immediately afterwards. For example, consider the it.po file in the commit referenced in this thread yesterday: commit b36518cb880bb236496ec3e505ede4001ce56157 Author: PostgreSQL Daemon webmas...@postgresql.org Date: Sun Feb 28 21:32:02 2010 + This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'. Cherrypick from master 2010-02-28 21:31:57 UTC Tom Lane t...@sss.pgh.pa.us 'Fix up memory management problems in contrib/xml2.': contrib/xml2/expected/xml2.out contrib/xml2/sql/xml2.sql src/bin/pg_dump/po/it.po Max. signature.asc Description: OpenPGP digital signature
Re: [HACKERS] Synchronous replication - patch status inquiry
On Tue, Sep 7, 2010 at 2:15 PM, Simon Riggs si...@2ndquadrant.com wrote: Every time I explain anything, I get someone run around shouting but that can't work!. I'm sorry, but again your logic is poor and the bias against properly considering viable alternatives is the only thing perfectly evident. So yes, I agree, it is a waste of time discussing it until I show working code. Obviously you don't agree, because that's the exact opposite of what I just said. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
On Tue, Sep 7, 2010 at 12:44 PM, Pavel Stehule pavel.steh...@gmail.com wrote: I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. you can, but you have to do preallocation and you have to use a FIXED flag. MAP_FIXED? As TFM says: Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] can we publish a aset interface?
2010/9/7 Robert Haas robertmh...@gmail.com: On Tue, Sep 7, 2010 at 12:44 PM, Pavel Stehule pavel.steh...@gmail.com wrote: I don't see how you could do anything with this that you can't do with the existing implementation. It's not as if you can store pointers into an mmap'd block and then count on them being valid the next time you map the file... it might not end up at the same offset. you can, but you have to do preallocation and you have to use a FIXED flag. MAP_FIXED? As TFM says: Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged. yes, I know. This will be used for proprietary Czech language - 95% of postgresql instalations are on Linux, 10% on MS Windows (in Czech Republic) I don't plan to try to move this module to core. And it's useless - other languages has not our problems. Regards Pavel -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Max Bowsher m...@f2s.com writes: On 07/09/10 18:16, Tom Lane wrote: Hmm, I see. This depends on the fact that git commits reference filesystem states and not deltas, correct? So it does actually make sense to just delete that commit from the history. I was concerned that it'd invalidate later commits, but I guess it doesn't. It wouldn't - except for the fact that cvs2git batches such manufactured commits such that there is no guarantee that a single manufactured commit pertains only to files in the commit immediately afterwards. Hmm ... so the consequence of that would be that (in this example) it.po would show up as being part of the REL8_4_STABLE file set as of that commit, rather than as of the later commit where it really got added. That's kind of annoying, but it is not a showstopper I think. Recall that the goals we set for this conversion in the first place were (1) duplicate the file set as of any back release tag and (2) duplicate the CVS log history as nearly as practical. We know we have met (1), because Magnus explicitly tested that. IMO we have met (2) adequately as well, with or without any fix for the manufactured-commit issue. On reflection it might be better to leave well enough alone, though. Anybody looking at the real commit in future might be confused by the fact that it added a seemingly unrelated file. It would be less confusing to have an obviously made-up commit adding some files, probably. A compromise might be to excise only those manufactured commits that added files directly related to the following real commit. I haven't looked to see how many there are that grouped unrelated files. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Freezing per-role settings
Folks, I noticed a little unimplemented feature which I suspect a lot of people would find useful, namely the ability to freeze certain settings for a role. Example: We'd like to create a role called read_only, with eponymous capability. At the moment, we can't do what's below, but I'd like to be able to make it possible. First, we'd issue the following, which doesn't work yet: ALTER ROLE read_only SET transaction_isolation read_only; Then, there's one way via DCL (Data Control Language) REVOKE SET transaction_isolation FROM read_only; Another would be via DDL: ALTER ROLE read_only FREEZE transaction_isolation; I'd think of the reverse of each of these as GRANT and ALTER ... THAW, respectively. Is anyone else interested in such a feature? If so, is it more DCL-ish, or more DDL-ish? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] function_name.parameter_name
Hi, On 7 September 2010 20:35, Tom Lane t...@sss.pgh.pa.us wrote: How does $subject differ from what we already do? See http://www.postgresql.org/docs/9.0/static/plpgsql-structure.html So will it be possible to do things like this? 1. CREATE FUNCTION func_name(arg_name text) RETURNS integer AS $$ BEGIN RAISE INFO '%', func_name.arg_name; ... 2. CREATE FUNCTION func_name() RETURNS integer AS $$ DECLARE var_name text := 'bla'; BEGIN RAISE INFO '%', func_name.var_name; ... 3. CREATE FUNCTION func_very_very_very_very_long_name() RETURNS integer AS $$ func_alias DECLARE var_name text := 'bla'; BEGIN RAISE INFO '%', func_alias.var_name; ... -- Sergey Konoplev Blog: http://gray-hemp.blogspot.com / Linkedin: http://ru.linkedin.com/in/grayhemp / JID/GTalk: gray...@gmail.com / Skype: gray-hemp / ICQ: 29353802 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] UTF16 surrogate pairs in UTF8 encoding
On sön, 2010-08-22 at 15:15 -0400, Tom Lane wrote: We combine the surrogate pair components to a single code point and encode that in UTF-8. We don't encode the components separately; that would be wrong. Oh, OK. Should the docs make that a bit clearer? Done. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Freezing per-role settings
On Tue, 2010-09-07 at 11:39 -0700, David Fetter wrote: We'd like to create a role called read_only, with eponymous capability. Seems useful. If so, is it more DCL-ish, or more DDL-ish? I don't like the idea of a security model relying on the ability (or lack thereof) to set GUCs. Imagine the effects of adding new GUCs, removing old ones, changing a GUC name, or tweaking the behavior slightly. It makes more sense to tie it to the role directly, so DDL. Also, you should put this in the context of previous discussions, which lead to the ON ALL TABLES IN SCHEMA feature in 9.0. In particular, that feature only affects existing objects, and you are trying to create some kind of permissions mask which will affect new objects, as well. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, Sep 7, 2010 at 5:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. If you feel that decoupling WAL application is absolutely essential to have a credible feature, then you'd better bite the bullet and start working on the ship-WAL-back code. In the mode where it is not required that the WAL is applied (only sent to the slave / synced to slave disk) one alternative is to have a separate pointer to the last WAL record that can be safely applied on the slave. Then You can send the un-synced WAL to the slave (while concurrently syncing it on the master). When both the slave an the master sync complete, one can give the client a commit notification, increase the pointer, and send it to the slave (it would be a separate WAL record type I guess). In case of master failure, the slave can discard the un-applied WAL after the pointer. Greetings marcin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 10:08 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 7, 2010 at 9:56 AM, Magnus Hagander mag...@hagander.net wrote: You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? I think that the latest issue here is the issue of how files get added to branches, which we discussed before with pretty much the same set of conclusions. I'm not wild about the way that's getting converted, but I'm not sure I care enough about it to argue with Tom. However, I want to convince myself that the deletes we've done over the years have been properly handled. I need to look at Max's latest conversion and I'll look at yours as well. Magnus - I just looked at your latest conversion (based on what Max did) and it looks a lot better. I think, though, that we should re-remove these branches: origin/unlabeled-1.44.2 origin/unlabeled-1.51.2 origin/unlabeled-1.59.2 origin/unlabeled-1.87.2 origin/unlabeled-1.90.2 -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronization levels in SR
On Tue, Sep 7, 2010 at 4:06 PM, marcin mank marcin.m...@gmail.com wrote: On Tue, Sep 7, 2010 at 5:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. If you feel that decoupling WAL application is absolutely essential to have a credible feature, then you'd better bite the bullet and start working on the ship-WAL-back code. In the mode where it is not required that the WAL is applied (only sent to the slave / synced to slave disk) one alternative is to have a separate pointer to the last WAL record that can be safely applied on the slave. Then You can send the un-synced WAL to the slave (while concurrently syncing it on the master). When both the slave an the master sync complete, one can give the client a commit notification, increase the pointer, and send it to the slave (it would be a separate WAL record type I guess). In case of master failure, the slave can discard the un-applied WAL after the pointer. But the pointer on the slave has to be fsync'd to make it persistent, which likely takes roughly the same amount of time as fsync-ing the WAL itself. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Robert Haas robertmh...@gmail.com writes: I just looked at your latest conversion (based on what Max did) and it looks a lot better. I think, though, that we should re-remove these branches: origin/unlabeled-1.44.2 origin/unlabeled-1.51.2 origin/unlabeled-1.59.2 origin/unlabeled-1.87.2 origin/unlabeled-1.90.2 I haven't looked at Magnus' latest iteration, but in Max's version this was showing as a branch: remotes/origin/REL8_0_0 AFAIK that was simply a mistake: it was intended to be a tag not a branch. If it's feasible to downgrade it to a tag during the conversion, that would be a good thing to do. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 22:06, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 7, 2010 at 10:08 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 7, 2010 at 9:56 AM, Magnus Hagander mag...@hagander.net wrote: You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? I think that the latest issue here is the issue of how files get added to branches, which we discussed before with pretty much the same set of conclusions. I'm not wild about the way that's getting converted, but I'm not sure I care enough about it to argue with Tom. However, I want to convince myself that the deletes we've done over the years have been properly handled. I need to look at Max's latest conversion and I'll look at yours as well. Magnus - I just looked at your latest conversion (based on what Max did) and it looks a lot better. I think, though, that we should re-remove these branches: origin/unlabeled-1.44.2 origin/unlabeled-1.51.2 origin/unlabeled-1.59.2 origin/unlabeled-1.87.2 origin/unlabeled-1.90.2 Oh yeah, I did the push before I ran that step of my script. Oops, sorry. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Freezing per-role settings
On Tue, Sep 07, 2010 at 12:41:51PM -0700, Jeff Davis wrote: On Tue, 2010-09-07 at 11:39 -0700, David Fetter wrote: We'd like to create a role called read_only, with eponymous capability. Seems useful. Great to hear :) If so, is it more DCL-ish, or more DDL-ish? I don't like the idea of a security model relying on the ability (or lack thereof) to set GUCs. Imagine the effects of adding new GUCs, removing old ones, changing a GUC name, or tweaking the behavior slightly. Offhand, I'm not thinking of past examples of mutating/disappearing GUC that people would want to freeze, nor of a new GUC that would negate or substantially alter such freezing. What have I missed? It makes more sense to tie it to the role directly, so DDL. There are still arguments for making it DCL-ish, in the sense that it is, at least in this case, viewable as a data control issue. Also, you should put this in the context of previous discussions, which lead to the ON ALL TABLES IN SCHEMA feature in 9.0. In particular, that feature only affects existing objects, and you are trying to create some kind of permissions mask which will affect new objects, as well. I guess I can see a case for making read-only non-global, but I think a good first try at it would be to make such freezes global. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 22:16, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I just looked at your latest conversion (based on what Max did) and it looks a lot better. I think, though, that we should re-remove these branches: origin/unlabeled-1.44.2 origin/unlabeled-1.51.2 origin/unlabeled-1.59.2 origin/unlabeled-1.87.2 origin/unlabeled-1.90.2 I haven't looked at Magnus' latest iteration, but in Max's version this was showing as a branch: remotes/origin/REL8_0_0 AFAIK that was simply a mistake: it was intended to be a tag not a branch. If it's feasible to downgrade it to a tag during the conversion, that would be a good thing to do. Shold be doable with a simple: git tag REL8_0_0 REL8_0_0 git branch -D REL8_0_0 I'll try that and re-run my content-verification script on top of that. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] function_name.parameter_name
I think so. Try it! David On Sep 7, 2010, at 11:39 AM, Sergey Konoplev wrote: Hi, On 7 September 2010 20:35, Tom Lane t...@sss.pgh.pa.us wrote: How does $subject differ from what we already do? See http://www.postgresql.org/docs/9.0/static/plpgsql-structure.html So will it be possible to do things like this? 1. CREATE FUNCTION func_name(arg_name text) RETURNS integer AS $$ BEGIN RAISE INFO '%', func_name.arg_name; ... 2. CREATE FUNCTION func_name() RETURNS integer AS $$ DECLARE var_name text := 'bla'; BEGIN RAISE INFO '%', func_name.var_name; ... 3. CREATE FUNCTION func_very_very_very_very_long_name() RETURNS integer AS $$ func_alias DECLARE var_name text := 'bla'; BEGIN RAISE INFO '%', func_alias.var_name; -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Freezing per-role settings
On Tue, 2010-09-07 at 13:30 -0700, David Fetter wrote: Offhand, I'm not thinking of past examples of mutating/disappearing GUC that people would want to freeze, nor of a new GUC that would negate or substantially alter such freezing. What have I missed? If you'll allow me to change my argument slightly, it just seems chaotic. We'd be introducing the 100+ GUCs all as potential security features, and it would (presumably) be up to the user whether they considered it a security feature or not. I think, in practice, that would confuse users about the security of the system, and we'd be more reluctant to change GUC behavior because someone, somewhere, might have considered it a part of their system's security. Perhaps someone will assume that they can prevent a user from performing joins by disabling and freezing enable_hashjoin/nestloop/mergejoin. Or perhaps someone will try to contain a user to a few schemas by freezing the search_path. Maybe this is a little far-fetched, but the point is that we are quite a ways away from blessing all GUCs with a word like security. It just seems like the wrong mechanism. It makes more sense to tie it to the role directly, so DDL. There are still arguments for making it DCL-ish, in the sense that it is, at least in this case, viewable as a data control issue. I would be more open to it if it didn't rely on GUCs at all. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Max Bowsher m...@f2s.com writes: It wouldn't - except for the fact that cvs2git batches such manufactured commits such that there is no guarantee that a single manufactured commit pertains only to files in the commit immediately afterwards. For example, consider the it.po file in the commit referenced in this thread yesterday: OK, I looked at this example, and I'm confused again. The actual 8.4 history of src/bin/pg_dump/po/it.po is that it was removed from HEAD on 2009-06-26, before the 8.4 branch was split off; and then re-added to the 8.4 branch on 2010-05-13, just before 8.4.4 was tagged. See http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/pg_dump/po/it.po Looking at Max's conversion with git log --all --source --name-status, this file is shown as modified in the latter commit: commit 575981a2fd6da5ccbf75c57580bf2d98b41f936e refs/tags/REL8_4_4 Author: Peter Eisentraut pete...@gmx.net Date: Thu May 13 10:50:20 2010 + Translation update ... M src/bin/pg_dump/po/it.po ... The deletion is correctly shown here: commit 4ade8dc6f7030b14306916b787fa8f75e4d49b2e refs/tags/REL8_4_0 Author: Peter Eisentraut pete...@gmx.net Date: Fri Jun 26 19:33:52 2009 + Translation updates for 8.4 release. File that are translated less than 80% have been removed, as per new translation team policy. ... D src/bin/pg_dump/po/it.po ... Now I can find two intermediate commits that touched this file: commit b78e79ec74fd4fac0c24753bbf8fa69fe7e6feb9 refs/tags/REL8_4_3 Author: PostgreSQL Daemon webmas...@postgresql.org Date: Fri Mar 12 03:23:24 2010 + This commit was manufactured by cvs2svn to create tag 'REL8_4_3'. Sprout from REL8_4_STABLE 2010-03-12 03:23:23 UTC Marc G. Fournier scra...@hub.org '' Delete: src/bin/pg_dump/po/it.po D src/bin/pg_dump/po/it.po commit b36518cb880bb236496ec3e505ede4001ce56157 refs/tags/REL8_4_4 Author: PostgreSQL Daemon webmas...@postgresql.org Date: Sun Feb 28 21:32:02 2010 + This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'. Cherrypick from master 2010-02-28 21:31:57 UTC Tom Lane t...@sss.pgh.pa.us 'Fix up memory management problems in contrib/xml2.': contrib/xml2/expected/xml2.out contrib/xml2/sql/xml2.sql src/bin/pg_dump/po/it.po A contrib/xml2/expected/xml2.out A contrib/xml2/sql/xml2.sql A src/bin/pg_dump/po/it.po Now it seems to me that this is just totally wacko. In the first place, the commit manufactured by cvs2svn to create tag 'REL8_4_3' postdates the commit where Marc actually tagged 8.4.3: commit 3aa54912637319c516f59d3a0265cb7826ed125f refs/tags/REL8_4_4 Author: Marc G. Fournier scra...@hub.org Date: Fri Mar 12 03:23:23 2010 + tag 8.4.3 M configure M configure.in M doc/bug.template M src/include/pg_config.h.win32 M src/interfaces/libpq/libpq.rc.in M src/port/win32ver.rc BTW, why is this commit shown as being a predecessor of refs/tags/REL8_4_4 and not refs/tags/REL8_4_3? That's nothing to do with it.po, perhaps, but it sure looks wrong. (Magnus, did you check against the 8.4.3 tarball?) But the main gripe is: how can it be claimed to be sane to represent the revision history as being that it.po was added to 8.4.4 two weeks before it was deleted from 8.4.3? There is definitely *something* not kosher about the manufactured-commit logic. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Freezing per-role settings
On Tue, Sep 07, 2010 at 02:43:12PM -0700, Jeff Davis wrote: On Tue, 2010-09-07 at 13:30 -0700, David Fetter wrote: Offhand, I'm not thinking of past examples of mutating/disappearing GUC that people would want to freeze, nor of a new GUC that would negate or substantially alter such freezing. What have I missed? If you'll allow me to change my argument slightly, it just seems chaotic. We'd be introducing the 100+ GUCs all as potential security features, and it would (presumably) be up to the user whether they considered it a security feature or not. I think, in practice, that would confuse users about the security of the system, and we'd be more reluctant to change GUC behavior because someone, somewhere, might have considered it a part of their system's security. Perhaps someone will assume that they can prevent a user from performing joins by disabling and freezing enable_hashjoin/nestloop/mergejoin. Or perhaps someone will try to contain a user to a few schemas by freezing the search_path. Maybe this is a little far-fetched, but the point is that we are quite a ways away from blessing all GUCs with a word like security. It just seems like the wrong mechanism. OK :) It makes more sense to tie it to the role directly, so DDL. There are still arguments for making it DCL-ish, in the sense that it is, at least in this case, viewable as a data control issue. I would be more open to it if it didn't rely on GUCs at all. There are two problems at hand here, as I see it: the more general problem of freezing settings for a given role, and the very specific capability of guaranteeing read-only-ness, which could have large implications in, for example, data warehousing and replication systems. Should we just call them separate problems, look into how to approach the latter one, and table the former? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On 07/09/10 21:25, Magnus Hagander wrote: On Tue, Sep 7, 2010 at 22:06, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 7, 2010 at 10:08 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 7, 2010 at 9:56 AM, Magnus Hagander mag...@hagander.net wrote: You're saying you don't require a fix on the latest issue here? Or should we spend some time trying to figure out if we can fix it with git-filter-branch? I think that the latest issue here is the issue of how files get added to branches, which we discussed before with pretty much the same set of conclusions. I'm not wild about the way that's getting converted, but I'm not sure I care enough about it to argue with Tom. However, I want to convince myself that the deletes we've done over the years have been properly handled. I need to look at Max's latest conversion and I'll look at yours as well. Magnus - I just looked at your latest conversion (based on what Max did) and it looks a lot better. I think, though, that we should re-remove these branches: origin/unlabeled-1.44.2 origin/unlabeled-1.51.2 origin/unlabeled-1.59.2 origin/unlabeled-1.87.2 origin/unlabeled-1.90.2 Oh yeah, I did the push before I ran that step of my script. Oops, sorry. Speaking of which, could you update the public copy of all the conversion documentation / machinery? Thanks, Max. signature.asc Description: OpenPGP digital signature
Re: [HACKERS] git: uh-oh
On Tue, Sep 7, 2010 at 5:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, why is this commit shown as being a predecessor of refs/tags/REL8_4_4 and not refs/tags/REL8_4_3? That's nothing to do with it.po, perhaps, but it sure looks wrong. (Magnus, did you check against the 8.4.3 tarball?) I think this is another result of the same basic problem. Since cvs2git thinks it.po was added to REL8_4_STABLE on 2010-02-28 rather than 2010-05-13,the REL8_4_STABLE version that existed on to 2010-03-12, when 8.4.3 was tagged, includes that file. But cvs2git also knows that 8.4.3 does NOT include that file, so it picks the commit on the 8.4.3 branch that most closely matches the contents of the tag (namely, Marc's tag 8.4.3 commit) and then shoves a manufactured commit on top of that to make the contents of the 8.4.3 tag match what actually got tagged. But that manufactured commit is only there to make the tag contents match; it's not actually part of the branch. If the conversion correctly made it.po get added on 2010-05-13 rather than 2010-02-28 then Marc's tag 8.4.3 commit would match the tag contents exactly and no manufactured commit would be created. The effect of all of this is that if someone checks out a git commit between 2010-02-28 and 2010-05-13, it.po will be there, even though file didn't exist on that CVS branch at that time. Max's contention seems to be that this is a CVS problem rather than a cvs2git problem. Perhaps we can do something like cvs update -r REL8_4_STABLE -d SOME_INTERMEDIATE_DATE and see whether that file is there or not. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On 07/09/10 23:15, Robert Haas wrote: On Tue, Sep 7, 2010 at 5:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, why is this commit shown as being a predecessor of refs/tags/REL8_4_4 and not refs/tags/REL8_4_3? That's nothing to do with it.po, perhaps, but it sure looks wrong. (Magnus, did you check against the 8.4.3 tarball?) I think this is another result of the same basic problem. Since cvs2git thinks it.po was added to REL8_4_STABLE on 2010-02-28 rather than 2010-05-13,the REL8_4_STABLE version that existed on to 2010-03-12, when 8.4.3 was tagged, includes that file. But cvs2git also knows that 8.4.3 does NOT include that file, so it picks the commit on the 8.4.3 branch that most closely matches the contents of the tag (namely, Marc's tag 8.4.3 commit) and then shoves a manufactured commit on top of that to make the contents of the 8.4.3 tag match what actually got tagged. But that manufactured commit is only there to make the tag contents match; it's not actually part of the branch. If the conversion correctly made it.po get added on 2010-05-13 rather than 2010-02-28 then Marc's tag 8.4.3 commit would match the tag contents exactly and no manufactured commit would be created. Yes, this is the correct analysis. The effect of all of this is that if someone checks out a git commit between 2010-02-28 and 2010-05-13, it.po will be there, even though file didn't exist on that CVS branch at that time. Max's contention seems to be that this is a CVS problem rather than a cvs2git problem. Perhaps we can do something like cvs update -r REL8_4_STABLE -d SOME_INTERMEDIATE_DATE and see whether that file is there or not. $ cvs co -r REL8_4_STABLE -D 2010-04-01 pgsql ... $ ls -la pgsql/src/bin/pg_dump/po/it.po -rw-r--r-- 1 maxb maxb 67871 2010-02-19 00:40 pgsql/src/bin/pg_dump/po/it.po It's there. Max. signature.asc Description: OpenPGP digital signature
Re: [HACKERS] Freezing per-role settings
On Tue, 2010-09-07 at 14:49 -0700, David Fetter wrote: There are two problems at hand here, as I see it: the more general problem of freezing settings for a given role, and the very specific capability of guaranteeing read-only-ness, which could have large implications in, for example, data warehousing and replication systems. Should we just call them separate problems, look into how to approach the latter one, and table the former? That sounds like a good plan. Right now, the only solid use case we have is read only role, and it's difficult to build a (good) general mechanism from a single use case. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
Robert Haas robertmh...@gmail.com writes: On Tue, Sep 7, 2010 at 5:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, why is this commit shown as being a predecessor of refs/tags/REL8_4_4 and not refs/tags/REL8_4_3? I think this is another result of the same basic problem. Since cvs2git thinks it.po was added to REL8_4_STABLE on 2010-02-28 rather than 2010-05-13,the REL8_4_STABLE version that existed on to 2010-03-12, when 8.4.3 was tagged, includes that file. But cvs2git also knows that 8.4.3 does NOT include that file, so it picks the commit on the 8.4.3 branch that most closely matches the contents of the tag (namely, Marc's tag 8.4.3 commit) and then shoves a manufactured commit on top of that to make the contents of the 8.4.3 tag match what actually got tagged. But that manufactured commit is only there to make the tag contents match; it's not actually part of the branch. If the conversion correctly made it.po get added on 2010-05-13 rather than 2010-02-28 then Marc's tag 8.4.3 commit would match the tag contents exactly and no manufactured commit would be created. Hmm. Some further looking in the git log output shows that that manufactured commit is actually the ONLY commit shown as being a predecessor of REL8_4_3. Everything else after 8.4.2 was tagged is shown as reached from refs/tags/REL8_4_4. This is at the least pretty weird, and I have to suppose it's the manufactured commit causing it. It does appear to agree with your explanation: the 8.4.3 state is not part of the branch's main evolution, but is a little side branch all by itself. The effect of all of this is that if someone checks out a git commit between 2010-02-28 and 2010-05-13, it.po will be there, even though file didn't exist on that CVS branch at that time. Yeah, that's what it's doing for me. Max's contention seems to be that this is a CVS problem rather than a cvs2git problem. No doubt. However, the facts on the ground are that it.po is provably not there in REL8_4_0, REL8_4_1, REL8_4_2, or REL8_4_3, and is there in REL8_4_4, and that no commit on the branch touched it before 2010-05-13 (just before 8.4.4). I will be interested to see the argument why cvs2git should consider the sanest translation of these facts to involve adding it.po to the branch after 8.4.2 and removing it again before 8.4.3. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] git: uh-oh
On 07/09/10 23:20, Max Bowsher wrote: On 07/09/10 23:15, Robert Haas wrote: On Tue, Sep 7, 2010 at 5:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, why is this commit shown as being a predecessor of refs/tags/REL8_4_4 and not refs/tags/REL8_4_3? That's nothing to do with it.po, perhaps, but it sure looks wrong. (Magnus, did you check against the 8.4.3 tarball?) I think this is another result of the same basic problem. Since cvs2git thinks it.po was added to REL8_4_STABLE on 2010-02-28 rather than 2010-05-13,the REL8_4_STABLE version that existed on to 2010-03-12, when 8.4.3 was tagged, includes that file. But cvs2git also knows that 8.4.3 does NOT include that file, so it picks the commit on the 8.4.3 branch that most closely matches the contents of the tag (namely, Marc's tag 8.4.3 commit) and then shoves a manufactured commit on top of that to make the contents of the 8.4.3 tag match what actually got tagged. But that manufactured commit is only there to make the tag contents match; it's not actually part of the branch. If the conversion correctly made it.po get added on 2010-05-13 rather than 2010-02-28 then Marc's tag 8.4.3 commit would match the tag contents exactly and no manufactured commit would be created. Yes, this is the correct analysis. The effect of all of this is that if someone checks out a git commit between 2010-02-28 and 2010-05-13, it.po will be there, even though file didn't exist on that CVS branch at that time. Max's contention seems to be that this is a CVS problem rather than a cvs2git problem. Perhaps we can do something like cvs update -r REL8_4_STABLE -d SOME_INTERMEDIATE_DATE and see whether that file is there or not. $ cvs co -r REL8_4_STABLE -D 2010-04-01 pgsql ... $ ls -la pgsql/src/bin/pg_dump/po/it.po -rw-r--r-- 1 maxb maxb 67871 2010-02-19 00:40 pgsql/src/bin/pg_dump/po/it.po It's there. And, I've just tracked down that this bug was apparently fixed in CVS 1.11.18, released November 2004. Max. signature.asc Description: OpenPGP digital signature
Re: [HACKERS] git: uh-oh
I wrote: Hmm. Some further looking in the git log output shows that that manufactured commit is actually the ONLY commit shown as being a predecessor of REL8_4_3. Everything else after 8.4.2 was tagged is shown as reached from refs/tags/REL8_4_4. This is at the least pretty weird, and I have to suppose it's the manufactured commit causing it. It does appear to agree with your explanation: the 8.4.3 state is not part of the branch's main evolution, but is a little side branch all by itself. This same pattern can be found repeated in at least ten earlier places in our project history, btw --- just look for commits using the phrase manufactured by cvs2svn to create tag instead of to create branch. The worst example is probably the one for tag REL7_1_BETA, which deletes 70-odd files. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication - patch status inquiry
Robert Haas wrote: On Tue, Sep 7, 2010 at 11:59 AM, Simon Riggs si...@2ndquadrant.com wrote: What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to sleep waiting for the next chunk. ?That's fine as far as the protocol goes, but I'm not convinced that it really does all that much in terms of improving performance. ?You still have the problem that the master has to fsync its WAL before it can send it to the slave. ?Also, the slave won't know whether it ought to fsync its own WAL before replying. Yes, apart from last sentence. Please wait for the code. So, we're going around and around in circles here because you're repeatedly refusing to explain how the slave will know WHEN to send acknowledgments back to the master without knowing which sync rep level is in use. It seems to be perfectly evident to everyone else here that there are only two ways for this to work: either the value is configured on the standby, or there's a registration system on the master and the master tells the standby its wishes. Instead of asking the entire community to wait for an unspecified period of time for you to write code that will handle this in an unspecified way, how about answering the question? We've wasted far too much time arguing about this already. Ideally I would like the sync method to be set on each slave, and have some method for the master to query the sync mode of all the slaves, e.g. appname. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
