Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On 08.03.2011 04:07, Greg Stark wrote: Well from that log you definitely have OldestXmin going backwards. And not by a little bit either. at 6:33 it set the all_visible flag and then at 7:01 it was almost 1.3 million transactions earlier. In fact to precisely the same value that was in use for a transaction at 1:38. That seems like a bit of a coincidence though it's not repeated earlier. Yep. After staring at GetOldestXmin() again, it finally struck me how OldestXmin can move backwards. You need two databases for it, which probably explains why this has been so elusive. Here's how to reproduce that: CREATE DATABASE foodb; CREATE DATABASE bardb; session 1, in foodb: foodb=# begin isolation level serializable; BEGIN foodb=# CREATE TABLE foo (a int4); -- just something to force this xact to have an xid CREATE TABLE foodb=# (leave the transaction open) session 2, in bardb: bardb=# CREATE TABLE foo AS SELECT 1; SELECT bardb=# vacuum foo; -- to set the PD_ALL_VISIBLE flag VACUUM session 3, in bardb: bardb=# begin isolation level serializable; BEGIN bardb=# SELECT 1; ?column? -- 1 (1 row) (leave transaction open) session 2, in bardb: bardb=# vacuum foo; WARNING: PD_ALL_VISIBLE flag was incorrectly set in relation foo page 0 (OldestXmin 803) VACUUM bardb=# What there are no other transactions active in the same database, GetOldestXmin() returns just latestCompletedXid. When you open a transaction in the same database after that, its xid will be above latestCompletedXid, but its xmin includes transactions from all databases, and there might be a transaction in some other database with an xid that precedes the value that GetOldestXmin() returned earlier. I'm not sure what to do about that. One idea is track two xmin values in proc-array, one that includes transactions in all databases, and another that only includes transactions in the same database. GetOldestXmin() (when allDbs is false) would only pay attention to the latter. It would add a few instructions to GetSnapshotData(), though. Another idea is to give up on the warning when it appears that oldestxmin has moved backwards, and assume that it's actually fine. We could still warn in other cases where the flag appears to be incorrectly set, like if there is a deleted tuple on the page. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
On Mon, Mar 07, 2011 at 07:15:28PM -0500, Tom Lane wrote: Only if the expression-to-be-sorted does not already fully specify the collation, which so far as I can tell (either from the code or your description above) it does. I think that the explicit representation of collation as part of the PathKey node is unnecessary, inefficient, and bug-inducing --- the latter because it promotes fuzzy thinking about where the collation information is coming from. (And this isn't just hypothetical: IMO the bugs I exhibited upthread are *directly* due to fuzzy thinking about what defines an index's sort order.) Well, collation processing happens in two phases. Initially collate information is provided by the columns in the query, explicit clauses, etc. These are indeed attached to the values. From here the collations of expressions are determined. The SQL committee thought up a bunch of actually quite logical rules here. Explicit overrides implicit, implicit overrides default. Both explicit or both implicit is an error, etc. Note error state is only a problem if you use it for sorting or comparison, otherwise it is ignored. This phase of processing happens in the parse analysis, the end result being that every expression should have a collation set, and every operator where it matters has consistant collation information for its arguments. So at this point the collation should be attached to the expression. In the planning phase however, all collation information is ignored except where it matters, which is the comparison operators and ORDER BY and similar. By the previous phase all comparison operators should have a defined collation for its arguments and thus for itself, this allows it to construct appropriate pathkeys for index scans, etc. I havn't looked at the patch, perhaps it confuses the information in the two phases but the basic idea seems to me: - parse analysis - collation in expressions - planning - collation in path key Or, to put it another way: the properties that define a sort order are the sort comparison operator, the collation, the ASC/DESC bit, and the NULLS FIRST/LAST bit. Given the way that the SQL committee has constructed the language, the operator and the two flag bits are attached to the ORDER BY clause, but the collation is a property of the expression-to-be-sorted. If we fail to preserve that distinction in the internal representation, we're just creating problems for ourselves. The assigning of the collation to expressions is simply the method for getting the collation information to the operators. Mainly because 99% of the time there will be no explicit clauses in the queries, so the information has to get there by other means. Hope this helps, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Patriotism is when love of your own people comes first; nationalism, when hate for people other than your own comes first. - Charles de Gaulle signature.asc Description: Digital signature
Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On 08.03.2011 10:00, Heikki Linnakangas wrote: Another idea is to give up on the warning when it appears that oldestxmin has moved backwards, and assume that it's actually fine. We could still warn in other cases where the flag appears to be incorrectly set, like if there is a deleted tuple on the page. This is probably a better idea at least in back-branches. It also handles the case of twiddling vacuum_defer_cleanup_age, which tracking two xmins per transactions would not handle. Here's a patch. I also changed the warning per Robert's suggestion. Anyone see a hole in this? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c index 67e0be9..dbd2e8d 100644 --- a/src/backend/commands/vacuumlazy.c +++ b/src/backend/commands/vacuumlazy.c @@ -332,6 +332,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, Size freespace; bool all_visible_according_to_vm = false; bool all_visible; + bool has_dead_tuples; /* * Skip pages that don't require vacuuming according to the visibility @@ -475,6 +476,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, * requiring freezing. */ all_visible = true; + has_dead_tuples = false; nfrozen = 0; hastup = false; prev_dead_count = vacrelstats-num_dead_tuples; @@ -613,6 +615,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, { lazy_record_dead_tuple(vacrelstats, (tuple.t_self)); tups_vacuumed += 1; +has_dead_tuples = true; } else { @@ -671,9 +674,22 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, PageSetAllVisible(page); SetBufferCommitInfoNeedsSave(buf); } - else if (PageIsAllVisible(page) !all_visible) + /* + * It's possible for the value returned by GetOldestXmin() to move + * backwards, so it's not wrong for us to see tuples that appear to + * not be visible to everyone yet, while PD_ALL_VISIBLE is already + * set. The real safe xmin value never moves backwards, but + * GetOldestXmin() is conservative and sometimes returns a value + * that's unnecessarily small, so if we see that contradiction it + * just means that the tuples that we think are not visible to + * everyone yet actually are, and the PD_ALL_VISIBLE flag is correct. + * + * There shouldn't be any dead tuples on a page with PD_ALL_VISIBLE + * set, however. + */ + else if (PageIsAllVisible(page) has_dead_tuples) { - elog(WARNING, PD_ALL_VISIBLE flag was incorrectly set in relation \%s\ page %u, + elog(WARNING, page containing dead tuples is marked as all-visible in relation \%s\ page %u, relname, blkno); PageClearAllVisible(page); SetBufferCommitInfoNeedsSave(buf); -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On Tue, Mar 08, 2011 at 10:00:01AM +0200, Heikki Linnakangas wrote: On 08.03.2011 04:07, Greg Stark wrote: Well from that log you definitely have OldestXmin going backwards. And not by a little bit either. at 6:33 it set the all_visible flag and then at 7:01 it was almost 1.3 million transactions earlier. In fact to precisely the same value that was in use for a transaction at 1:38. That seems like a bit of a coincidence though it's not repeated earlier. Yep. After staring at GetOldestXmin() again, it finally struck me how OldestXmin can move backwards. You need two databases for it, which probably explains why this has been so elusive. ... What there are no other transactions active in the same database, GetOldestXmin() returns just latestCompletedXid. When you open a transaction in the same database after that, its xid will be above latestCompletedXid, but its xmin includes transactions from all databases, and there might be a transaction in some other database with an xid that precedes the value that GetOldestXmin() returned earlier. I'm not sure what to do about that. One idea is track two xmin values in proc-array, one that includes transactions in all databases, and another that only includes transactions in the same database. GetOldestXmin() (when allDbs is false) would only pay attention to the latter. It would add a few instructions to GetSnapshotData(), though. Another idea is to give up on the warning when it appears that oldestxmin has moved backwards, and assume that it's actually fine. We could still warn in other cases where the flag appears to be incorrectly set, like if there is a deleted tuple on the page. I read this to mean that it is safe to ignore this warning and that these databases are not at risk for data corruption or wrong results so long as the warning is due to oldestxmin. Please correct me if I have misunderstood. Thanks -dg -- David Gould da...@sonic.net 510 536 1443510 282 0869 If simplicity worked, the world would be overrun with insects. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On Tue, Mar 08, 2011 at 10:37:24AM +0200, Heikki Linnakangas wrote: On 08.03.2011 10:00, Heikki Linnakangas wrote: Another idea is to give up on the warning when it appears that oldestxmin has moved backwards, and assume that it's actually fine. We could still warn in other cases where the flag appears to be incorrectly set, like if there is a deleted tuple on the page. This is probably a better idea at least in back-branches. It also handles the case of twiddling vacuum_defer_cleanup_age, which tracking two xmins per transactions would not handle. Here's a patch. I also changed the warning per Robert's suggestion. Anyone see a hole in this? It would be helpful to have the dbname and schema in the message in addition to the relname. I added those to the original diagnostic patch as it was not clear that the messages were all related to the same page/table/dg. Also, in your comment you might mention that multiple databases are one way we could see oldestxmin move backwards. -dg -- David Gould da...@sonic.net 510 536 1443510 282 0869 If simplicity worked, the world would be overrun with insects. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On 08.03.2011 10:38, daveg wrote: I read this to mean that it is safe to ignore this warning and that these databases are not at risk for data corruption or wrong results so long as the warning is due to oldestxmin. Please correct me if I have misunderstood. Yes, that's correct. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Beginner question: Hacking environment?
Heyho! I'm poking around a bit in PostgreSQL's source for curiosity's sake. Is there a short howto on how to start / debug postgres from within the unpacked/compiled tar? Are there scripts that set up linker paths and Postgres' environment? thanks in advance -- vbi -- This statement is false. signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] pl/python tracebacks
On 07/03/11 22:55, Peter Eisentraut wrote: On mån, 2011-03-07 at 14:19 +0100, Jan Urbański wrote: On 07/03/11 14:01, Jan Urbański wrote: On 07/03/11 13:53, Peter Eisentraut wrote: On sön, 2011-03-06 at 13:14 +0100, Jan Urbański wrote: But fixing raise plpy.Fatal() to actually cause a FATAL is something that should be extracted from this patch and committed, even if the full patch does not make it. Um, what? I didn't find any details about this in this thread, nor a test case. So this in fact are three separate things, tracebacks, fix for plpy.Fatal and a one-line fix for reporting errors in Python iterators, that as I noticed has a side effect of changing the SQLCODE being raised :( I think I'll just respin the tracebacks patch as 3 separate ones, coming right up. Respun as three separate patches. Sorry for the confusion. BTW: looks like plpy.Fatal behaviour has been broken for quite some time now. Committed 1 and 2. Your traceback implementation in PLy_elog is now using two errdetail calls in one ereport call, which doesn't work (first one wins). Please reconsider that. Also, the comment still talks about the traceback going into detail. Gah, will look at this and fix. Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Basic Recovery Control functions for use in Hot Standby. Pause,
On Tue, Mar 8, 2011 at 12:04 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Feb 9, 2011 at 5:12 PM, Magnus Hagander mag...@hagander.net wrote: I was also worried about the non-hot-standby case, but I see that the patch makes sure you can't enable pause when not in hot standby mode. Which in itself might be surprising - perhaps we need a NOTICE for when that happens as well? I didn't include this fix in the patch because I prefer FATAL to NOTICE for that. NOTICE doesn't stop recovery. So we might be unable to notice such a NOTICE message and stop the recovery before it's too late, i.e., the recovery has completed at the undesirable point. So I think that emitting FATAL is safer. I included this fix in the patch, which emits FATAL if pause_at_recovery_target is enabled while hot standby is disabled and the recovery target is set. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center change_recovery_conf_sample_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v19
On Mon, Mar 7, 2011 at 4:54 AM, Robert Haas robertmh...@gmail.com wrote: On Mar 6, 2011, at 9:44 AM, Fujii Masao masao.fu...@gmail.com wrote: On Sun, Mar 6, 2011 at 5:02 PM, Yeb Havinga yebhavi...@gmail.com wrote: On Sun, Mar 6, 2011 at 8:58 AM, Fujii Masao masao.fu...@gmail.com wrote: If unfortunately all connection slots are used by backends waiting for replication, we cannot execute such a function. So it makes more sense to introduce something like pg_ctl standalone command? If it is only for shutdown, maybe pg_ctl stop -m standalone? It's for not only shutdown but also running the primary in standalone mode. So something like pg_ctl standalone is better. For now I think that pg_ctl command is better than built-in function because sometimes we might want to wake waiters up even during shutdown in order to cause shutdown to end. During shutdown, the server doesn't accept any new connection (even from the standby). So, without something like pg_ctl standalone, there is no way to cause shutdown to end. This sounds like an awful hack to work around a bad design. Surely once shutdown reaches a point where new replication connections can no longer be accepted, any standbys hung on commit need to close the connection without responding to the COMMIT, per previous discussion. It's completely unreasonable for sync rep to break the shutdown sequence. Yeah, let's think about how shutdown should work. I'd like to propose the following. Thought? * Smart shutdown Smart shutdown should wait for all the waiting backends to be acked, and should not cause them to forcibly exit. But this leads shutdown to get stuck infinitely if there is no walsender at that time. To enable them to be acked even in that situation, we need to change postmaster so that it accepts the replication connection even during smart shutdown (until we reach PM_SHUTDOWN_2 state). Postmaster has already accepted the superuser connection to cancel backup during smart shutdown. So I don't think that the idea to accept the replication connection during smart shutdown is so ugly. * Fast shutdown I agree with you about fast shutdown. Fast shutdown should cause all the backends including waiting ones to exit immediately. At that time, the non-acked backend should not return the success, according to the definition of sync rep. So we need to change a backend so that it gets rid of itself from the waiting queue and exits before returning the success, when it receives SIGTERM. This change leads the waiting backends to do the same even when pg_terminate_backend is called. But since they've not been acked yet, it seems to be reasonable to prevent them from returning the COMMIT. Comments? I'll create the patch barring objection. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
On Mon, Mar 7, 2011 at 10:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: I think what is happening here is that make launches concurrent sub-jobs to do make install in each of interfaces/libpq and interfaces/ecpg, and the latter launches a sub-sub-job to do make all in interfaces/libpq, and make has no idea that these are duplicate sub-jobs so it actually tries to run both concurrently. Whereupon you get all sorts of fun failures. I'm not sure if there is any cure that's not worse than the disease. BTW, how many people here have read Recursive Make Considered Harmful? http://aegis.sourceforge.net/auug97.pdf Because what we're presently doing looks mighty similar to what he's saying doesn't work and can't be made to work. I'm not sure whether it makes sense to go that far or not. But I think it'd make sense to at least try this for the backend. It does seem pretty silly to have a Makefile in every single directory. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
On 03/07/2011 10:28 PM, Tom Lane wrote: I wrote: I think what is happening here is that make launches concurrent sub-jobs to do make install in each of interfaces/libpq and interfaces/ecpg, and the latter launches a sub-sub-job to do make all in interfaces/libpq, and make has no idea that these are duplicate sub-jobs so it actually tries to run both concurrently. Whereupon you get all sorts of fun failures. I'm not sure if there is any cure that's not worse than the disease. BTW, how many people here have read Recursive Make Considered Harmful? http://aegis.sourceforge.net/auug97.pdf Because what we're presently doing looks mighty similar to what he's saying doesn't work and can't be made to work. Oh, yes, I read it a long time ago, before I started doing Postgres work. I recall vaguely thinking about it when I began with Postgres, but I thought people smarter than me had probably worked out the problems :-) (Working with people smarter than me is one of the things I like about Postgres work.) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
Excerpts from Robert Haas's message of mar mar 08 10:38:29 -0300 2011: On Mon, Mar 7, 2011 at 10:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: I think what is happening here is that make launches concurrent sub-jobs to do make install in each of interfaces/libpq and interfaces/ecpg, and the latter launches a sub-sub-job to do make all in interfaces/libpq, and make has no idea that these are duplicate sub-jobs so it actually tries to run both concurrently. Whereupon you get all sorts of fun failures. I'm not sure if there is any cure that's not worse than the disease. BTW, how many people here have read Recursive Make Considered Harmful? http://aegis.sourceforge.net/auug97.pdf Because what we're presently doing looks mighty similar to what he's saying doesn't work and can't be made to work. Yeah, I read it some years ago and considered it, but it was too disruptive or I was too new here, maybe both :-) The bit I looked at, at the time, was src/backend/mb/conversion_procs, because that was where the biggest hit on parallelization was taken (a single lib at a time -- the real time CPU usage chart clearly showed the problem. Not sure if that's still a problem). I'm not sure whether it makes sense to go that far or not. But I think it'd make sense to at least try this for the backend. It does seem pretty silly to have a Makefile in every single directory. We already do that for the backend. Not exactly a single Makefile, but the dependencies are all declared in indirectly in src/backend/Makefile with the common.mk tricks. Where it doesn't work is in the other subdirs, c.f. the current problem with interfaces/libpq and interfaces/ecpg. It would be a lot more difficult to fix there, I think, but maybe I'm wrong. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
On Tue, Mar 8, 2011 at 9:07 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: The bit I looked at, at the time, was src/backend/mb/conversion_procs, because that was where the biggest hit on parallelization was taken (a single lib at a time -- the real time CPU usage chart clearly showed the problem. Not sure if that's still a problem). I think it is, based on having noticed it spend what seemed like a disproportionate amount of time on that stuff when building, but I haven't actually tried to measure it. I'm not sure whether it makes sense to go that far or not. But I think it'd make sense to at least try this for the backend. It does seem pretty silly to have a Makefile in every single directory. We already do that for the backend. Not exactly a single Makefile, but the dependencies are all declared in indirectly in src/backend/Makefile with the common.mk tricks. I'm not sure that's really the same thing. It'd be interesting to redo it with just one Makefile and see whether it's faster. Where it doesn't work is in the other subdirs, c.f. the current problem with interfaces/libpq and interfaces/ecpg. It would be a lot more difficult to fix there, I think, but maybe I'm wrong. Yeah, that's a problem. I wondered if supplying -p to mkdir would ameliorate the problem to some degree... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Adrian von Bidder avbid...@fortytwo.ch wrote: Is there a short howto on how to start / debug postgres from within the unpacked/compiled tar? Are there scripts that set up linker paths and Postgres' environment? You should probably start by reading the developer FAQ page: http://wiki.postgresql.org/wiki/Developer_FAQ If you still have questions, be sure to mention your OS. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
On 07.03.2011 17:43, Tom Lane wrote: because two expressions that are equal() must necessarily have the same collation property. Peter, Tom, I am not able to see this. If 'abc' == 'abc' is not collation depending at all. It is only encoding depending. Collation is only needed for upper(), lower() and sorting. Means it tells if e.g. upper('i') will get Y or I. It tells if the German s-umlaut will be sorted together with 's' or after 'z'. Btw. the follows on implementing collations will be different - and I hope Peter is aware of it. My experience is that a huge follow will be that users will complain that the sorting isn't correct even when it is correct. Susanne -- Susanne Ebrecht - 2ndQuadrant PostgreSQL Development, 24x7 Support, Training and Services www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
Alvaro Herrera alvhe...@commandprompt.com writes: Where it doesn't work is in the other subdirs, c.f. the current problem with interfaces/libpq and interfaces/ecpg. It would be a lot more difficult to fix there, I think, but maybe I'm wrong. Right, it's specifically the interdependence between ecpg and libpq that's causing the main symptom Jeff is complaining of. Although when I was trying make -j12 install starting from a clean tree yesterday, I did see at least one failure in the backend. It's all pretty timing-dependent --- if you look at the make output, you can clearly see that the same sub-make tasks get launched repeatedly due to various makefiles trying to force prerequisites in other parts of the tree to be up to date. (Which is exactly one of the band-aid fixes that Miller talks about.) If two such tasks get launched close enough to the same time, they both try to do the same work, and then you get failures like ln complaining that the target is already there, or ar complaining that somebody corrupted its output file, etc etc. I think Miller's analysis is dead on and we ought to think seriously about adopting his approach. Obviously this is not a small task... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [DOCS] Sync rep doc corrections
On 8 March 2011 01:08, David Fetter da...@fetter.org wrote: On Mon, Mar 07, 2011 at 02:31:18PM -0500, Robert Haas wrote: On Mon, Mar 7, 2011 at 2:07 PM, David Fetter da...@fetter.org wrote: Re: docs, I'd actually like to see that list gone, as the separation of docs from code is one that's actively unhelpful. We don't have a separate docs team, and we rightly put the responsibility of documenting changes on the person or people patching them in. You're in the minority on this one. It happens, but I keep speaking up anyhow. Minority status isn't always permanent. :) Okay, to avoid any -hackers list noise, I'll exclusively send them to -docs in future, with the exception of replying to -committers or -hackers when a commit/patch contains fail. -- Thom Brown Twitter: @darkixion IRC (freenode): dark_ixion Registered Linux user: #516935 EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] How should the waiting backends behave in sync rep?
Hi, How should the backends waiting for replication behave when synchrnous_standby_names is set to '' and the configuration file is reloaded? Now they keep waiting for the ACK from the standby. But I think that it's more natural for them to get out of the wait state and complete the transaction in that case. If we'll change them in that way, we would no longer need something like pg_ctl standalone which I mentioned in another thread. Thought? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
Susanne Ebrecht susa...@2ndquadrant.com writes: On 07.03.2011 17:43, Tom Lane wrote: because two expressions that are equal() must necessarily have the same collation property. Peter, Tom, I am not able to see this. If 'abc' == 'abc' is not collation depending at all. It is only encoding depending. Sorry, I was using a term of art there. This isn't about data values, it's about parsed expression trees. See the equal() function in src/backend/nodes/equalfuncs.c. If two expression trees match according to equal(), then they must necessarily have the same collation property, because equal() compares all the fields and in particular the collation fields. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Hi, On Tuesday 08 March 2011 15.39:56 Kevin Grittner wrote: http://wiki.postgresql.org/wiki/Developer_FAQ thanks If you still have questions, be sure to mention your OS. Sorry, forgot. Linux. The Basic system testing item is where I'm kinda stuck. You advise to perform run time testing via psql. Which is a bit short, since obviously psql can't be directly started since there is no installation... Aagain, my question: are there wrappers/helpers to do this easily from the source/compile tree? Like: set up PATH, LD_LIBRARY_PATH, PG_WHATEVER, call initdb, ... Or do you advise to just make install and test from there? thanks -- vbi -- [...] if you have to do the occasional model for Japanese collectors, which are covered in reticulated ostrich testicle leather, then by all means continue to do so. -- Michael Reichmann http://www.luminous-landscape.com/essays/leica-open-letter.shtml signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] Theory of operation of collation patch
Martijn van Oosterhout klep...@svana.org writes: This phase of processing happens in the parse analysis, the end result being that every expression should have a collation set, and every operator where it matters has consistant collation information for its arguments. So at this point the collation should be attached to the expression. Right. I havn't looked at the patch, perhaps it confuses the information in the two phases but the basic idea seems to me: - parse analysis - collation in expressions - planning - collation in path key Not so right. A path key contains an expression tree, plus whatever *additional* information is needed to fully specify the sort ordering. If the collation is already fully determined by the expression tree, there is no need to duplicate that information in the PathKey node. And, as I said, doing so anyway has real negative consequences. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
On Tue, Mar 8, 2011 at 9:17 AM, Adrian von Bidder avbid...@fortytwo.ch wrote: Hi, On Tuesday 08 March 2011 15.39:56 Kevin Grittner wrote: http://wiki.postgresql.org/wiki/Developer_FAQ thanks If you still have questions, be sure to mention your OS. Sorry, forgot. Linux. The Basic system testing item is where I'm kinda stuck. You advise to perform run time testing via psql. Which is a bit short, since obviously psql can't be directly started since there is no installation... Aagain, my question: are there wrappers/helpers to do this easily from the source/compile tree? Like: set up PATH, LD_LIBRARY_PATH, PG_WHATEVER, call initdb, ... sure, linux distos provide packages that do all that stuff. if you are rolling your own from scratch, you have to do some of that yourself...create the user, set the path, etc. This is all documented here: http://www.postgresql.org/docs/9.0/interactive/install-post.html. If you are hacking postgres this will become second nature to you very quickly. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Adrian von Bidder avbid...@fortytwo.ch wrote: Or do you advise to just make install and test from there? Pretty much. For development you will want to specify a few options in the ./configure step. At a minimum I recommend: --prefix= --enable-debug --enable-cassert --enable-depend See this page for other options: http://www.postgresql.org/docs/9.0/interactive/install-procedure.html Of course, there are those who prefer to use eclipse or some other IDE. For setting up eclipse, see this page: http://wiki.postgresql.org/wiki/Working_with_Eclipse I see that page is a bit out of date (we've converted from CVS to git), but perhaps you can extrapolate. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
On 08/03/2011 07:44, Selena Deckelmann wrote: Hi! PostgreSQL is applying for GSoC again this year. We're looking for: * Mentors Ready to mentor again this year for phpPgAdmin * Project ideas Would you like to mentor? Please let me know! Our application closes on Friday, so please contact me *before* Friday. I've started a wiki page: http://wiki.postgresql.org/wiki/GSoC_2011 It's seeded with last year's todo lists and information. We need to add project ideas for students to it. I updated the page with last year PPA project The wiki pages for 2008 and 2010 are available, including links to the original student proposals: http://wiki.postgresql.org/wiki/GSoC_2010 http://wiki.postgresql.org/wiki/GSoC_2008 Thanks! -selena -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] #PgEast Schedule is up
Hey, The schedule for #PgEast is up. It can be found here: https://www.postgresqlconference.org/files/east_2011_schedule.html As usually we have a increasingly wide selection of content. Sincerely, JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v19
On Tue, Mar 8, 2011 at 7:05 AM, Fujii Masao masao.fu...@gmail.com wrote: Yeah, let's think about how shutdown should work. I'd like to propose the following. Thought? * Smart shutdown Smart shutdown should wait for all the waiting backends to be acked, and should not cause them to forcibly exit. But this leads shutdown to get stuck infinitely if there is no walsender at that time. To enable them to be acked even in that situation, we need to change postmaster so that it accepts the replication connection even during smart shutdown (until we reach PM_SHUTDOWN_2 state). Postmaster has already accepted the superuser connection to cancel backup during smart shutdown. So I don't think that the idea to accept the replication connection during smart shutdown is so ugly. * Fast shutdown I agree with you about fast shutdown. Fast shutdown should cause all the backends including waiting ones to exit immediately. At that time, the non-acked backend should not return the success, according to the definition of sync rep. So we need to change a backend so that it gets rid of itself from the waiting queue and exits before returning the success, when it receives SIGTERM. This change leads the waiting backends to do the same even when pg_terminate_backend is called. But since they've not been acked yet, it seems to be reasonable to prevent them from returning the COMMIT. The fast shutdown handling seems fine, but why not just handle smart shutdown the same way? I don't really like the idea of allowing replication connections for longer, and the idea that we don't want to keep waiting for a commit ACK once we're past the point where it's possible for one to occur seems to apply generically to any shutdown sequence. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] wrap alpha4 tomorrow ~9am Eastern (was: Alpha4 release blockers)
On Mon, Mar 7, 2011 at 5:03 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Mar 7, 2011 at 4:59 PM, Peter Eisentraut pete...@gmx.net wrote: On mån, 2011-03-07 at 15:19 -0500, Robert Haas wrote: Sorry, you're right. Still, as happy as I am that we've made so much progress with PL/python (and other things) this CommitFest, I think it's time to pick a date and time to ship alpha4 and call this one good. If the patch makes it, great; if not, oh well. We can't keep letting this drag out. Go for it. We might add the traceback patch afterwards, but it's not ready at the moment. OK, then I propose we tag and cut the tarball on Wednesday morning, say around 9am Eastern. Objections? Going once, going twice... I'll go ahead and do this, barring objections or some other volunteer. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
On tis, 2011-03-08 at 16:17 +0100, Adrian von Bidder wrote: Aagain, my question: are there wrappers/helpers to do this easily from the source/compile tree? Like: set up PATH, LD_LIBRARY_PATH, PG_WHATEVER, call initdb, ... Many developers have their own wrapper scripts, mainly to handle dealing with multiple versions or branches. But just to build and test one version you don't need to set any environment variables. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Hi, On Tuesday 08 March 2011 16.58:58 Kevin Grittner wrote: Adrian von Bidder avbid...@fortytwo.ch wrote: Or do you advise to just make install and test from there? Pretty much. Thanks for all your answers. I was just a bit confused because when I write stuff for myself I usually set up stuff to run directly in the build environment, so I was expecting something like that to be available. But make install into ~/pg will do fine as well. (I want to avoid having my trashed version version be available systemwide...) cheers -- vbi -- I've never been DPL (AFAICT) -- Pierre Habouzit on debian-vote, 2007-07-31 signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] Beginner question: Hacking environment?
Adrian von Bidder avbid...@fortytwo.ch writes: Thanks for all your answers. I was just a bit confused because when I write stuff for myself I usually set up stuff to run directly in the build environment, so I was expecting something like that to be available. But make install into ~/pg will do fine as well. (I want to avoid having my trashed version version be available systemwide...) No, nobody installs test versions into real locations. What you want is to set an install spot with configure --prefix. For instance, I usually do configure --prefix=/home/tgl/testversion --enable-debug --enable-cassert Be sure to add $PREFIX/bin to your PATH so that you can call the test-installation programs conveniently. It's also often a good idea to specify a non-default port number using --with-pgport so that you can run your test postmaster concurrently with a regular one. Lastly, setting PGDATA in your environment to a suitable test database location will save typing and avoid mistakes. As Peter mentioned, most of us have scripts to set up a preferred working environment of this sort. I think I've published mine at least once ... check the archives. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Tom Lane t...@sss.pgh.pa.us wrote: As Peter mentioned, most of us have scripts to set up a preferred working environment of this sort. I think I've published mine at least once That reminds me -- Greg Smith put something together which might make it easier to get started: https://github.com/gregs1104/peg/ -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
On Tue, Mar 8, 2011 at 3:21 PM, Tom Lane t...@sss.pgh.pa.us wrote: Not so right. A path key contains an expression tree, plus whatever *additional* information is needed to fully specify the sort ordering. If the collation is already fully determined by the expression tree, there is no need to duplicate that information in the PathKey node. And, as I said, doing so anyway has real negative consequences. Isn't the reason to copy that information outside the expression so that we can choose sometimes to ignore it? Namely, for == we can use an index with any defined collation even if it doesn't match the collation in the pathkey we're looking for? I think currently that's the only example but in theory we could have collations that are supersets of the desired collation. For example a UTF8 collation that sorts english in the desired way and sorts utf8 characters in some way that isn't relevant to the query. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
Greg Stark gsst...@mit.edu writes: On Tue, Mar 8, 2011 at 3:21 PM, Tom Lane t...@sss.pgh.pa.us wrote: Not so right. A path key contains an expression tree, plus whatever *additional* information is needed to fully specify the sort ordering. If the collation is already fully determined by the expression tree, there is no need to duplicate that information in the PathKey node. And, as I said, doing so anyway has real negative consequences. Isn't the reason to copy that information outside the expression so that we can choose sometimes to ignore it? Namely, for == we can use an index with any defined collation even if it doesn't match the collation in the pathkey we're looking for? No, that's nonsense. A PathKey is all about sort order. It is not relevant in any situation where sort order is ignorable. I think currently that's the only example but in theory we could have collations that are supersets of the desired collation. For example a UTF8 collation that sorts english in the desired way and sorts utf8 characters in some way that isn't relevant to the query. That sounds like nonsense as well. There is no such thing as a superset ordering: to have such a thing, you'd have to have some orderings that didn't fully determine the ordering of data values, which is pretty much unworkable. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
On 03/08/2011 01:04 PM, Kevin Grittner wrote: Tom Lanet...@sss.pgh.pa.us wrote: As Peter mentioned, most of us have scripts to set up a preferred working environment of this sort. I think I've published mine at least once That reminds me -- Greg Smith put something together which might make it easier to get started: https://github.com/gregs1104/peg/ FWIW, the buildfarm software https://github.com/PGBuildFarm/client-code has a couple of developer modes. They are particularly useful for testing patches, as the script runs through a complete build and test cycle with one command. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: PD_ALL_VISIBLE flag was incorrectly set happend during repeatable vacuum
On 08.03.2011 10:49, daveg wrote: On Tue, Mar 08, 2011 at 10:37:24AM +0200, Heikki Linnakangas wrote: On 08.03.2011 10:00, Heikki Linnakangas wrote: Another idea is to give up on the warning when it appears that oldestxmin has moved backwards, and assume that it's actually fine. We could still warn in other cases where the flag appears to be incorrectly set, like if there is a deleted tuple on the page. This is probably a better idea at least in back-branches. It also handles the case of twiddling vacuum_defer_cleanup_age, which tracking two xmins per transactions would not handle. Here's a patch. I also changed the warning per Robert's suggestion. Anyone see a hole in this? It would be helpful to have the dbname and schema in the message in addition to the relname. I added those to the original diagnostic patch as it was not clear that the messages were all related to the same page/table/dg. Hmm, we don't usually include database name and schema in messages like this. There's a couple of other warnings in vacuum, too, that only print the table name. I have to admit that the database name was crucial in tracking this down, but in hindsight you could've added database name to log_line_prefix for the same effect. If you have several databases with same schema, that's a good idea anyway. So in the end, I decided not to include it. Also, in your comment you might mention that multiple databases are one way we could see oldestxmin move backwards. Ok, I added a comment to GetOldestXmin() explaining that. Committed. Thanks David for your help in debugging this! And thanks to everyone else for helping out too. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: psql include file using relative path
Gurjeet! What about tab completion, like in \i command? On Fri, Feb 25, 2011 at 5:07 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 24, 2011 at 6:21 PM, Gurjeet Singh singh.gurj...@gmail.com wrote: psql has the ability to execute commands from a file, but if one wishes to develop and provide a modularized set of sql files, then psql is not very helpful because the \i command can open file paths either if they are absolute paths or if they are palced correctly relative to psql's current working directory. Attached patch adds a new meta-command to psql, '\ir' that allows the user to process files relative to currently processing file. Please add this patch to the currently open CommitFest at: https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Ibrar Ahmed -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] .gitignore patch for coverage builds
On ons, 2011-01-26 at 15:38 -0600, Kevin Grittner wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Patch attached. The coverage directory belongs under Local excludes in root directory. Version 2. I have committed a simplified version of this, except the coverage/ directory, which some people had issues with. I think it's OK to manually ignore that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Heyho again! Now this kind of stuff is what I was after :-) On Tuesday 08 March 2011 19.04:53 Kevin Grittner wrote: That reminds me -- Greg Smith put something together which might make it easier to get started: https://github.com/gregs1104/peg/ Since you don't allow anonymous editing of the wiki, nor have an obvious create account, I leave it to you to add this: +++ How can I easily test my modified postgres version? You'll want to install PostgreSQL into a local directory (in your home directory, for instance) to avoid conflicting with a system wide installation. Use the --prefix=... option to specify an installation location; --with-pgport to specify a non-standard default port may also be helpful. To run this instance, you will need to make sure that the correct binaries are used; depending on your operating system, environment variables like PATH and LD_LIBRARY_PATH (on most Linux/Unix-like systems) need to be set. Setting PGDATA will also be useful. To avoid having to set this environment up manually, you may want to use Greg Smith's [href=https://github.com/gregs1104/peg peg] scripts, or the [https://github.com/PGBuildFarm/client-code scripts] that are used on the buildfarm. +++ ... and you'll never have to answer to this stupid question again ... :-) thanks happy hacking -- vbi P.S.: those who remember me from many months back may guess that I'm trying to implement IMMUTABLE columns. But I'll come back when I actually have to show something, so far I've only managed to break 121 of 124 tests of make check. At least initdb miraculously works again. -- this email is protected by a digital signature: http://fortytwo.ch/gpg signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
Hello! I've following project proposal. Currently GiST index don't have any bulk load functionality. It have to create new index by entry insertion one by one. This makes new index creation relatively slow. There are various works in computer science about bulk operation on R-tree. Since gist in basic is generalization of R-tree, some of them seems to be applicable to GiST. Following work seems to be most applicable to GiST: Efficient Bulk Operations on Dynamic R-Trees, L. Arge, K. H. Hinrichs, J. Vahrenhold and J. S. Vitter http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.9894rep=rep1type=pdf I believe that implementation of methods of this work should allow to significally reduce GiST index creation time (probably, it'll be possible to achive acceleration of other operations too). Also, it's possible to achive some results in computer science, since GiST is more general data structure than R-tree. I'm going to create wiki page for this project proposal soon. I'm last year phd. student and I would like to participate in GSoC with PostgreSQL. I would be especially happy to work with project above or other project related to indexing. With best regards, Alexander Korotkov. On Tue, Mar 8, 2011 at 9:44 AM, Selena Deckelmann sel...@chesnok.comwrote: Hi! PostgreSQL is applying for GSoC again this year. We're looking for: * Mentors * Project ideas Would you like to mentor? Please let me know! Our application closes on Friday, so please contact me *before* Friday. I've started a wiki page: http://wiki.postgresql.org/wiki/GSoC_2011 It's seeded with last year's todo lists and information. We need to add project ideas for students to it. The wiki pages for 2008 and 2010 are available, including links to the original student proposals: http://wiki.postgresql.org/wiki/GSoC_2010 http://wiki.postgresql.org/wiki/GSoC_2008 Thanks! -selena -- http://chesnok.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: psql include file using relative path
Attached patch implements tab completion. It also introduces the long-form alternative \include_relative for \ir Regards, On Tue, Mar 8, 2011 at 2:18 PM, Ibrar Ahmed ibrar.ah...@gmail.com wrote: Gurjeet! What about tab completion, like in \i command? On Fri, Feb 25, 2011 at 5:07 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 24, 2011 at 6:21 PM, Gurjeet Singh singh.gurj...@gmail.com wrote: psql has the ability to execute commands from a file, but if one wishes to develop and provide a modularized set of sql files, then psql is not very helpful because the \i command can open file paths either if they are absolute paths or if they are palced correctly relative to psql's current working directory. Attached patch adds a new meta-command to psql, '\ir' that allows the user to process files relative to currently processing file. Please add this patch to the currently open CommitFest at: https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Ibrar Ahmed -- Gurjeet Singh EnterpriseDB http://www.enterprisedb.com/ Corporation The Enterprise PostgreSQL http://www.postgresql.org/ Company psql_ir.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Beginner question: Hacking environment?
Hi, On Tuesday 08 March 2011 21:12:53 Adrian von Bidder wrote: Since you don't allow anonymous editing of the wiki, nor have an obvious create account, I leave it to you to add this: Its linked on the mainpage: http://www.postgresql.org/community/signup Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
Yes, this would be very interesting project. It's not just about bulk load (CREATE INDEX) but also about bulk insert, which is important for update operations. As it adds two additional internal GiST methods, there should be no problem with compatibility. Oleg On Tue, 8 Mar 2011, Alexander Korotkov wrote: Hello! I've following project proposal. Currently GiST index don't have any bulk load functionality. It have to create new index by entry insertion one by one. This makes new index creation relatively slow. There are various works in computer science about bulk operation on R-tree. Since gist in basic is generalization of R-tree, some of them seems to be applicable to GiST. Following work seems to be most applicable to GiST: Efficient Bulk Operations on Dynamic R-Trees, L. Arge, K. H. Hinrichs, J. Vahrenhold and J. S. Vitter http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.9894rep=rep1type=pdf I believe that implementation of methods of this work should allow to significally reduce GiST index creation time (probably, it'll be possible to achive acceleration of other operations too). Also, it's possible to achive some results in computer science, since GiST is more general data structure than R-tree. I'm going to create wiki page for this project proposal soon. I'm last year phd. student and I would like to participate in GSoC with PostgreSQL. I would be especially happy to work with project above or other project related to indexing. With best regards, Alexander Korotkov. On Tue, Mar 8, 2011 at 9:44 AM, Selena Deckelmann sel...@chesnok.comwrote: Hi! PostgreSQL is applying for GSoC again this year. We're looking for: * Mentors * Project ideas Would you like to mentor? Please let me know! Our application closes on Friday, so please contact me *before* Friday. I've started a wiki page: http://wiki.postgresql.org/wiki/GSoC_2011 It's seeded with last year's todo lists and information. We need to add project ideas for students to it. The wiki pages for 2008 and 2010 are available, including links to the original student proposals: http://wiki.postgresql.org/wiki/GSoC_2010 http://wiki.postgresql.org/wiki/GSoC_2008 Thanks! -selena -- http://chesnok.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: o...@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
On mån, 2011-03-07 at 13:51 -0800, Jeff Davis wrote: On Sat, 2011-03-05 at 18:33 -0500, Bruce Momjian wrote: I am seeing the following compile problem with gmake -j2: For what it's worth, I'm still seeing this problem too: http://archives.postgresql.org/pgsql-hackers/2010-12/msg00123.php I can reproduce it every time. Fixed. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel make problem with git master
On mån, 2011-03-07 at 22:28 -0500, Tom Lane wrote: BTW, how many people here have read Recursive Make Considered Harmful? http://aegis.sourceforge.net/auug97.pdf Because what we're presently doing looks mighty similar to what he's saying doesn't work and can't be made to work. Yes, that's the better solution. It will probably just upset a lot of people's thinking. The main problem way back when I last considered this seriously was that it wasn't clear how many compilers don't support -o with -c. The paper doesn't offer a clear solution to that, but it might be that the problem is effectively gone now. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On mån, 2011-03-07 at 12:40 -0500, Tom Lane wrote: Fair enough, but throwing in fmgr_info_collation(DEFAULT_COLLATION) anytime we have a problem seems to me to introduce the exact same issue. Who's to say that that's really the appropriate value to use? It normally isn't the appropriate value to use. It's only appropriate if either that particular part of the code doesn't support collations yet or it's dealing with some hardcoded catalog lookups or some similar case. In most other cases you should be getting the collation passed in from somewhere, and if you aren't there is probably some work to do to get it there. That's at least sort of the experience from developing this. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
Peter Eisentraut pete...@gmx.net writes: On mån, 2011-03-07 at 12:40 -0500, Tom Lane wrote: Fair enough, but throwing in fmgr_info_collation(DEFAULT_COLLATION) anytime we have a problem seems to me to introduce the exact same issue. Who's to say that that's really the appropriate value to use? It normally isn't the appropriate value to use. It's only appropriate if either that particular part of the code doesn't support collations yet or it's dealing with some hardcoded catalog lookups or some similar case. In most other cases you should be getting the collation passed in from somewhere, and if you aren't there is probably some work to do to get it there. That's at least sort of the experience from developing this. Mph. Well, I guess in the case of the pg_statistic stats we can declare by fiat that we calculate the stats according to the default collation. They'll be a bit off when used for a query that is comparing according to some other collation, but it's probably no worse than any other approximation we make in the selectivity code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
On mån, 2011-03-07 at 12:52 -0500, Tom Lane wrote: It looks like indcollation is acting as a substitute for including a CollateClause in the index key expression, which doesn't seem like a particularly good tradeoff considering all the overhead you must introduce into the default case. Conceptually, this characterization is correct. But assuming you do away with the indcollation column and instead create expression indexes on (col COLLATE x), you'd then have an interesting time matching those expressions to expressions in the query when choosing an index. Example: CREATE TABLE test (a text); CREATE INDEX test_idx ((a COLLATE foo)); -- expression index -- easy to choose the index here SELECT * FROM test WHERE a COLLATE foo 'x'; -- not so easy here SELECT * FROM test WHERE a 'x' COLLATE foo; -- more difficult CREATE TABLE test2 (b text COLLATE foo); SELECT * FROM test WHERE x (SELECT b FROM test2 WHERE something); Note that this situation already exists: If you create a plain column index using CREATE INDEX test_idx (a COLLATE foo), all of the above cases should pick up the index; if you create an expression index using CREATE INDEX test_idx ((a COLLATE foo)), they won't, except the first one. (That is modulo your earlier demonstration that it apparently doesn't work at all for you, but I'm just saying in case you want to try out various ways of changing this.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
On mån, 2011-03-07 at 19:15 -0500, Tom Lane wrote: Or, to put it another way: the properties that define a sort order are the sort comparison operator, the collation, the ASC/DESC bit, and the NULLS FIRST/LAST bit. Given the way that the SQL committee has constructed the language, the operator and the two flag bits are attached to the ORDER BY clause, but the collation is a property of the expression-to-be-sorted. If we fail to preserve that distinction in the internal representation, we're just creating problems for ourselves. You are probably right; it could be simplified. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alpha4 release blockers (was Re: wrapping up this CommitFest)
On tis, 2011-03-08 at 17:43 -0500, Tom Lane wrote: Mph. Well, I guess in the case of the pg_statistic stats we can declare by fiat that we calculate the stats according to the default collation. They'll be a bit off when used for a query that is comparing according to some other collation, but it's probably no worse than any other approximation we make in the selectivity code. That was my assumption for the time being. Unfortunately, it's hard to create test coverage for these types of issues. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
Peter Eisentraut pete...@gmx.net writes: On mån, 2011-03-07 at 12:52 -0500, Tom Lane wrote: It looks like indcollation is acting as a substitute for including a CollateClause in the index key expression, which doesn't seem like a particularly good tradeoff considering all the overhead you must introduce into the default case. Conceptually, this characterization is correct. But assuming you do away with the indcollation column and instead create expression indexes on (col COLLATE x), you'd then have an interesting time matching those expressions to expressions in the query when choosing an index. Hmm ... interesting point, but I think it already doesn't work in all cases for expression indexes. There is an analogous issue for binary-compatible datatypes, which we hack around by stripping RelabelType nodes before doing this type of comparison. Maybe we'll have to do something similar with CollateClauses. I never much cared for that hack, though ... wonder if there is a cleaner way. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
Hi Alexander, On Tue, Mar 8, 2011 at 12:13 PM, Alexander Korotkov aekorot...@gmail.com wrote: I've following project proposal. Thanks for getting a head start! One warning - we haven't been accepted into GSoC yet. :) I'll be sure to contact you if we are accepted! Currently GiST index don't have any bulk load functionality. It have to create new index by entry insertion one by one. This makes new index creation relatively slow. There are various works in computer science about bulk operation on R-tree. Since gist in basic is generalization of R-tree, some of them seems to be applicable to GiST. Following work seems to be most applicable to GiST: Efficient Bulk Operations on Dynamic R-Trees, L. Arge, K. H. Hinrichs, J. Vahrenhold and J. S. Vitter http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.9894rep=rep1type=pdf I believe that implementation of methods of this work should allow to significally reduce GiST index creation time (probably, it'll be possible to achive acceleration of other operations too). Also, it's possible to achive some results in computer science, since GiST is more general data structure than R-tree. I'm going to create wiki page for this project proposal soon. That sounds great. Please go ahead and create the page now. -selena -- http://chesnok.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
Hi! PostgreSQL is applying for GSoC again this year. We're looking for: * Mentors Can I be a metor for pgpool-II? -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp * Project ideas Would you like to mentor? Please let me know! Our application closes on Friday, so please contact me *before* Friday. I've started a wiki page: http://wiki.postgresql.org/wiki/GSoC_2011 It's seeded with last year's todo lists and information. We need to add project ideas for students to it. The wiki pages for 2008 and 2010 are available, including links to the original student proposals: http://wiki.postgresql.org/wiki/GSoC_2010 http://wiki.postgresql.org/wiki/GSoC_2008 Thanks! -selena -- http://chesnok.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
On Tue, Mar 8, 2011 at 3:52 PM, Tatsuo Ishii is...@postgresql.org wrote: Hi! PostgreSQL is applying for GSoC again this year. We're looking for: * Mentors Can I be a metor for pgpool-II? Yes! I can't guarantee that we will get student proposals for it. But, we will have a greater chance of student proposals for things if we specifically list out projects that you'd like a student to tackle. Could you add a few items to the wiki page for pgpool-II? http://wiki.postgresql.org/wiki/GSoC_2011#Project_Ideas Thanks! -selena -- http://chesnok.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2011 - Mentors? Projects?
Can I be a metor for pgpool-II? Yes! I can't guarantee that we will get student proposals for it. But, we will have a greater chance of student proposals for things if we specifically list out projects that you'd like a student to tackle. Could you add a few items to the wiki page for pgpool-II? http://wiki.postgresql.org/wiki/GSoC_2011#Project_Ideas Ok, I will do it. Thanks! Thanks! -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Theory of operation of collation patch
Another interesting item ... I see that you added a collation field to TypeName, apparently on the grounds that the SQL spec includes collation in data type. However, it seems to me that that is nonsense up with which we should not put. data type is basically only used in CAST and column definitions (plus some features related to methods, which we haven't got). In CAST, they have to have a syntax rule stating that you must not use COLLATE in the target type name. In column definitions, they call out collate clause as a separate syntactic element, and then they have to have a syntax rule stating that you can't use both that and a collate clause embedded in the type name. This is just stupid. And it conflates the notion of collation with that of type --- based on our discussions so far, it seems like it'd be a lot better to keep those separate. I think we should drop collate clause from TypeName and just have it in columnDef and the expression syntax. This might also ease the ambiguity problem that evidently led you to restrict the expression production's argument to c_expr. It would also allow us to meet the letter of the spec for column definition, in that collate clause is not required to immediately follow data type. I see that right now you don't allow collate clause in ColQualList, so it's not accepting syntax that the spec says is valid; and you really can't fix that as long as collate clause can be part of TypeName, because then there would be two ways to parse the case where COLLATE does immediately follow the type name. Thoughts? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Fix for pg_upgrade discriptor leaks
I have applied the attached patch to fix pg_upgrade file descriptor leaks in error paths. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + diff --git a/contrib/pg_upgrade/check.c b/contrib/pg_upgrade/check.c index 9ef63ec..1c4847a 100644 *** a/contrib/pg_upgrade/check.c --- b/contrib/pg_upgrade/check.c *** check_for_isn_and_int8_passing_mismatch( *** 515,523 PQfinish(conn); } if (found) { - fclose(script); pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains \/contrib/isn\ functions\n --- 515,525 PQfinish(conn); } + if (script) + fclose(script); + if (found) { pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains \/contrib/isn\ functions\n *** check_for_reg_data_type_usage(ClusterInf *** 616,624 PQfinish(conn); } if (found) { - fclose(script); pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains one of the reg* data types in\n --- 618,628 PQfinish(conn); } + if (script) + fclose(script); + if (found) { pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains one of the reg* data types in\n diff --git a/contrib/pg_upgrade/file.c b/contrib/pg_upgrade/file.c index deaca46..803e0a0 100644 *** a/contrib/pg_upgrade/file.c --- b/contrib/pg_upgrade/file.c *** pg_scandir_internal(const char *dirname, *** 302,308 --- 302,311 (size_t) ((name_num + 1) * sizeof(struct dirent *))); if (*namelist == NULL) + { + closedir(dirdesc); return -1; + } entrysize = sizeof(struct dirent) - sizeof(direntry-d_name) + strlen(direntry-d_name) + 1; *** pg_scandir_internal(const char *dirname, *** 310,316 --- 313,322 (*namelist)[name_num] = (struct dirent *) malloc(entrysize); if ((*namelist)[name_num] == NULL) + { + closedir(dirdesc); return -1; + } memcpy((*namelist)[name_num], direntry, entrysize); diff --git a/contrib/pg_upgrade/server.c b/contrib/pg_upgrade/server.c index c7684f8..84bd03e 100644 *** a/contrib/pg_upgrade/server.c --- b/contrib/pg_upgrade/server.c *** get_major_server_version(ClusterInfo *cl *** 144,154 if (fscanf(version_fd, %63s, cluster-major_version_str) == 0 || sscanf(cluster-major_version_str, %d.%d, integer_version, fractional_version) != 2) - { pg_log(PG_FATAL, could not get version from %s\n, datadir); ! fclose(version_fd); ! return 0; ! } return (100 * integer_version + fractional_version) * 100; } --- 144,152 if (fscanf(version_fd, %63s, cluster-major_version_str) == 0 || sscanf(cluster-major_version_str, %d.%d, integer_version, fractional_version) != 2) pg_log(PG_FATAL, could not get version from %s\n, datadir); ! ! fclose(version_fd); return (100 * integer_version + fractional_version) * 100; } diff --git a/contrib/pg_upgrade/version.c b/contrib/pg_upgrade/version.c index e321538..8ba7e98 100644 *** a/contrib/pg_upgrade/version.c --- b/contrib/pg_upgrade/version.c *** new_9_0_populate_pg_largeobject_metadata *** 62,71 PQfinish(conn); } if (found) { - if (!check_mode) - fclose(script); report_status(PG_WARNING, warning); if (check_mode) pg_log(PG_WARNING, \n --- 62,72 PQfinish(conn); } + if (script) + fclose(script); + if (found) { report_status(PG_WARNING, warning); if (check_mode) pg_log(PG_WARNING, \n diff --git a/contrib/pg_upgrade/version_old_8_3.c b/contrib/pg_upgrade/version_old_8_3.c index 5b226b2..3ec4b59 100644 *** a/contrib/pg_upgrade/version_old_8_3.c --- b/contrib/pg_upgrade/version_old_8_3.c *** old_8_3_check_for_name_data_type_usage(C *** 87,95 PQfinish(conn); } if (found) { - fclose(script); pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains the \name\ data type in\n --- 87,97 PQfinish(conn); } + if (script) + fclose(script); + if (found) { pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains the \name\ data type in\n *** old_8_3_check_for_tsquery_usage(ClusterI *** 175,183 PQfinish(conn); } if (found) { - fclose(script); pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains the \tsquery\ data type.\n --- 177,187 PQfinish(conn); } + if (script) + fclose(script); + if (found) { pg_log(PG_REPORT, fatal\n); pg_log(PG_FATAL, | Your installation contains the \tsquery\ data type.\n *** old_8_3_rebuild_tsvector_tables(ClusterI *** 314,323
Re: [HACKERS] Fix for pg_upgrade discriptor leaks
Bruce Momjian br...@momjian.us writes: I have applied the attached patch to fix pg_upgrade file descriptor leaks in error paths. It seems rather pointless to spend code closing descriptors immediately before a fatal exit. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fix for pg_upgrade discriptor leaks
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: I have applied the attached patch to fix pg_upgrade file descriptor leaks in error paths. It seems rather pointless to spend code closing descriptors immediately before a fatal exit. Well, it is not before a fatal but rather before it returns -1, which might fatal or might not. I also had code in to close file descriptors that was a little too tricky about using a single variable to indicate two things so I cleaned it up. I got a private email report about these so obviously they were confusing. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep v19
On Tue, Mar 8, 2011 at 11:58 AM, Robert Haas robertmh...@gmail.com wrote: The fast shutdown handling seems fine, but why not just handle smart shutdown the same way? currently, smart shutdown means no new connections, wait until existing ones close normally. for consistency, it should behave the same for sync rep. +1 for Fujii's proposal -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Sync Rep and shutdown Re: [HACKERS] Sync Rep v19
On Wed, Mar 9, 2011 at 2:14 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Tue, Mar 8, 2011 at 11:58 AM, Robert Haas robertmh...@gmail.com wrote: The fast shutdown handling seems fine, but why not just handle smart shutdown the same way? currently, smart shutdown means no new connections, wait until existing ones close normally. for consistency, it should behave the same for sync rep. Agreed. I think that user who wants to request smart shutdown expects all the existing connections to basically be closed normally by the client. So it doesn't seem to be good idea to forcibly close the connection and prevent the COMMIT from being returned in smart shutdown case. But I'm all ears for better suggestions. Anyway, we got the consensus about how fast shutdown should work with sync rep. So I created the patch. Please feel free to comment and commit the patch first ;) Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center fast_shutdown_syncrep_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers