Re: [HACKERS] Hot Standby status
Devrim GÜNDÜZ wrote: Is there a reason why recovery.conf.sample does not include (sample) entries for recovery_connections and max_standby_delay? No, they probably should be included. I'll add them, thanks. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Hot standby, freezing
While reading through the patch for what must be the 100th time by now, it occurred to me that this comment in heap_xlog_freeze: + /* +* Freezing tuples does not require conflict processing +*/ is plain wrong. In the master, we can freeze the xmin of a tuple that's not yet visible to all read-only transactions in the standby. We do need conflict processing there. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] PostgreSQL 8.3.8 on AIX5.3 : compilation failed
Tom Lane wrote: The problem is that both _LARGE_FILES and _LARGE_FILE_API are #defined in this case, which makes #include unistd.h fail. Does anyone have an idea how to best fix this problem in the source tree? I'm willing to implement and test. I've committed changes for this in CVS, please verify it fixes your problem. Problem fixed. Thanks for the quick work! Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parsing config files in a directory
On Tue, 2009-11-10 at 20:14 -0800, Josh Berkus wrote: On 11/10/09 5:59 AM, Bruce Momjian wrote: Simon Riggs wrote: All of this *also* applies to shared_preload_libraries. We also need to be able to specify new load libraries without editing the same darn parameter. And to search_path, though that's even more complex because the order of the entries is significant. Let's NOT start that discussion again. Bruce's comments were a useful addition to the technical discussion. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby status
On Tue, 2009-11-10 at 22:00 +0200, Devrim GÜNDÜZ wrote: On Tue, 2009-11-10 at 20:36 +0200, Heikki Linnakangas wrote: Attached is the latest and greatest patch against CVS head, taken from the hs-riggs branch in my git repository. Is there a reason why recovery.conf.sample does not include (sample) entries for recovery_connections and max_standby_delay? No reason. I'll add entries for those. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby status
On Tue, 2009-11-10 at 15:11 -0500, Michael Glaesemann wrote: I skimmed through the documentation to get a better handle on what this will mean. Thanks for this and any further corrections/additions. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] total execution time as reported by auto_explain
Jaime Casanova jcasa...@systemguards.com.ec writes: I was using auto_explain to log plans (with analyze option on) from queries that take more than 500ms, something i found i little odd is that the duration says 779ms but actual time says 269ms (almost 3 times lower), maybe some kind of instrumentation cleanup can explain this? Client slow about absorbing the rows, perhaps? The actual data output process isn't counted as part of the plan tree's runtime ... but it would be included in the total elapsed time. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deadlock on the same object?
Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp writes: I encountered the following log in 8.4.1 and HEAD. The deadlock occured on the same object (relation 17498 of database 17497). Is it reasonable? I think this is an artifact of the fact that SQL functions parse the whole querystring before executing any of it. Parsing of DELETE FROM a will result in acquiring ROW EXCLUSIVE lock on a, and then when the LOCK commands are executed, you have a lock-upgrade scenario and the deadlock is unsurprising. There was some discussion of changing that awhile ago, but I forget what the conclusion was. In any case nothing's been done about it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, freezing
On Wed, 2009-11-11 at 12:35 +0200, Heikki Linnakangas wrote: While reading through the patch for what must be the 100th time by now, :-) it occurred to me that this comment in heap_xlog_freeze: + /* +* Freezing tuples does not require conflict processing +*/ is plain wrong. In the master, we can freeze the xmin of a tuple that's not yet visible to all read-only transactions in the standby. We do need conflict processing there. I agree. Hmph, I wonder why I thought otherwise? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
bruce wrote: This brings up a concern I have --- that the number of patch committers/managers is decreasing while our patch volume is increasing. Consider that Heikki is working mostly on Hot Standby and Streaming Replication, Alvaro isn't as involved in applying patches, Neil Conway isn't involved with Postgres anymore, I am in a 42-day travel period, and Robert Haas is feeling burnt-out --- that is not a pretty sight. Much of the patch burden is falling on Tom. Now, Tom isn't complaining, but I am concerned about placing too much of a burden on him. I know we are growing new patch reviewers who will eventually be able to review _and_ apply patches on their own, but getting them to that point is going to take time. I have no real answers, but I am concerned. I have talked to many of you privately about this to get your input. There is a more worrisome/sinister possibility that I didn't want to mention in my first email --- that companies are hiring our most experienced developers and having them work almost exclusively on company-related or closed-source projects. Unfortunately I can think of at least half-a-dozen cases of this happening. Now, this was expected, but the hope was that this kind of skill siphoning would be offset by additional people being paid to be involved in Postgres development, and that clearly is happening. What I am worried about is that this is not happening as much among our most experienced people --- that in fact they are the most likely to be hired and perhaps placed into roles where they are less involved in the community than they were before. (Of course, there are counter-examples where experience developers are hired to work on community Postgres full-time.) There is not much we as a community can do to prevent skill siphoning, except perhaps publicly complaining about companies that do this. If this is indeed a pattern, it has serious long-term consequences because it means we will always have an unnaturally small pool of very skilled people. Up to this point we have been able to maintain a happy group of developers. If things become unbalanced and people are regularly required to do things they don't like doing, it will lead to Postgres no longer being fun for them, which leads to burn-out and them leaving the project. That happens frequently in other open source projects, but it has been a very rare occurance for us, and I hope it stays that way. Our ability to retain people for many years has benefitted us in countless ways. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Partitioning option for COPY
Hi all,
I have extracted the partitioning option for COPY (removed the error
logging part) from the previous patch. The documentation and test suite
sample are provided as well.
More details are on the wiki page at
http://wiki.postgresql.org/wiki/Auto-partitioning_in_COPY. Ignore the
error logging related comments that do not apply here.
Looking forward to your feedback
Emmanuel
--
Emmanuel Cecchet
Aster Data
Web: http://www.asterdata.com
Index: src/test/regress/parallel_schedule
===
RCS file: /home/manu/cvsrepo/pgsql/src/test/regress/parallel_schedule,v
retrieving revision 1.57
diff -c -r1.57 parallel_schedule
*** src/test/regress/parallel_schedule 24 Aug 2009 03:10:16 - 1.57
--- src/test/regress/parallel_schedule 11 Nov 2009 03:17:48 -
***
*** 47,53
# execute two copy tests parallel, to check that copy itself
# is concurrent safe.
# --
! test: copy copyselect
# --
# Another group of parallel tests
--- 47,53
# execute two copy tests parallel, to check that copy itself
# is concurrent safe.
# --
! test: copy copyselect copy_partitioning
# --
# Another group of parallel tests
Index: src/backend/utils/adt/ruleutils.c
===
RCS file: /home/manu/cvsrepo/pgsql/src/backend/utils/adt/ruleutils.c,v
retrieving revision 1.314
diff -c -r1.314 ruleutils.c
*** src/backend/utils/adt/ruleutils.c 5 Nov 2009 23:24:25 - 1.314
--- src/backend/utils/adt/ruleutils.c 11 Nov 2009 03:17:48 -
***
*** 218,224
static Node *processIndirection(Node *node, deparse_context *context,
bool printit);
static void printSubscripts(ArrayRef *aref, deparse_context *context);
! static char *generate_relation_name(Oid relid, List *namespaces);
static char *generate_function_name(Oid funcid, int nargs, List *argnames,
Oid
*argtypes, bool *is_variadic);
static char *generate_operator_name(Oid operid, Oid arg1, Oid arg2);
--- 218,224
static Node *processIndirection(Node *node, deparse_context *context,
bool printit);
static void printSubscripts(ArrayRef *aref, deparse_context *context);
! char *generate_relation_name(Oid relid, List *namespaces);
static char *generate_function_name(Oid funcid, int nargs, List *argnames,
Oid
*argtypes, bool *is_variadic);
static char *generate_operator_name(Oid operid, Oid arg1, Oid arg2);
***
*** 6347,6353
* We will forcibly qualify the relation name if it equals any CTE name
* visible in the namespace list.
*/
! static char *
generate_relation_name(Oid relid, List *namespaces)
{
HeapTuple tp;
--- 6347,6353
* We will forcibly qualify the relation name if it equals any CTE name
* visible in the namespace list.
*/
! char *
generate_relation_name(Oid relid, List *namespaces)
{
HeapTuple tp;
Index: doc/src/sgml/ref/copy.sgml
===
RCS file: /home/manu/cvsrepo/pgsql/doc/src/sgml/ref/copy.sgml,v
retrieving revision 1.92
diff -c -r1.92 copy.sgml
*** doc/src/sgml/ref/copy.sgml 21 Sep 2009 20:10:21 - 1.92
--- doc/src/sgml/ref/copy.sgml 11 Nov 2009 03:17:48 -
***
*** 41,46
--- 41,47
ESCAPE 'replaceable class=parameterescape_character/replaceable'
FORCE_QUOTE { ( replaceable class=parametercolumn/replaceable [,
...] ) | * }
FORCE_NOT_NULL ( replaceable class=parametercolumn/replaceable [,
...] )
+ PARTITIONING [ replaceable class=parameterboolean/replaceable ]
/synopsis
/refsynopsisdiv
***
*** 282,287
--- 283,301
/listitem
/varlistentry
+varlistentry
+ termliteralPARTITIONING//term
+ listitem
+ para
+ In literalPARTITIONING/ mode, commandCOPY TO/ a parent
+ table will automatically move each row to the child table that
+ has the matching constraints. This feature can be used with
+ literalERROR_LOGGING/ to capture rows that do not match any
+ constraint in the table hierarchy. See the notes below for the
+ limitations.
+ /para
+ /listitem
+/varlistentry
/variablelist
/refsect1
***
*** 384,389
--- 398,421
commandVACUUM/command to recover the wasted space.
/para
+para
+ literalPARTITIONING/ mode scans for each child table constraint in the
+ hierarchy to find a match. As an optimization, a cache of the last child
+ tables where tuples have been routed is kept and tried first. The size
+ of the cache is set by the literalcopy_partitioning_cache_size/literal
+
Re: [HACKERS] next CommitFest
Hi! On Tue, Nov 10, 2009 at 10:40 PM, Greg Smith gsm...@gregsmith.com wrote: On Sun, 8 Nov 2009, Robert Haas wrote: I would personally prefer not to be involved in the management of the next CommitFest. Having done all of the July CommitFest and a good chunk of the September CommitFest, I am feeling a bit burned out. I was just poking around on the Wiki, and it looks like the role of the CommitFest manager isn't very well documented yet. Since you've done all of them since introducing the new CF software, I'm not sure if anyone else even knows exactly what you've been doing. The transition over to that was so successful there isn't even a copy of the schedule for 8.5 on the Wiki itself. Could you find some time this week to rattle off an outline of the work involved? It's hard to decide whether to volunteer to help without having a better idea of what's required. It's pretty straightforward. Robert has actually done a great job of communicating about this to the patch reviewers. I'd be happy to get together at some pre-appointed hour this weekend (Saturday / Sunday) to talk it over by phone / IRC. PDXPUG was already planning to get together to review some patches this Sunday from 3-5pm PST, so that is a convenient time for me. I can also help with commitfest admin tasks, but not in a dedicated way until after Thanksgiving as I'm going to be be traveling or on vacation. -selena -- http://chesnok.com/daily - me http://endpoint.com - work -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On ons, 2009-11-11 at 10:25 -0500, Bruce Momjian wrote: There is a more worrisome/sinister possibility that I didn't want to mention in my first email --- that companies are hiring our most experienced developers and having them work almost exclusively on company-related or closed-source projects. I can't claim to know everyone's employment terms, but I think it's a bit of an illusion to think that the above hasn't always happened. Except for a handful of people who have very special job arrangements, *everyone* is working almost exclusively on company-related or closed-source projects. The patches that get sent in are almost always either fallout from a customer/company project, or stuff that one of the closed-sourced forks has developed that they don't want to maintain, or stuff people do at night to make their lives easier in the distant future. All of those things are already special arrangements that people need to make with their employers and their lives, but they have discernible benefits. But you can't expect a lot of people or employers to reserve time on top of that for handholding other people's patches and for other community stuff that has no easy to measure benefits. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Unpredictable shark slowdown after migrating to 8.4
Hello community, Second time after migration 8.3.7 -- 8.4.1 I was caught by this problem. Migration was 8 days ago. (note, I never seen such situation on 8.3) Environment: PostgreSQL 8.4.1 + patch http://archives.postgresql.org/pgsql-committers/2009-10/msg00056.php CentOS release 5.4 (Final) SunFire X4600 M2; 3U; 8xOpteron 8380 2.5GHz; 96GB; 12x146GB 15K RPM DB size ~90G TPS~1000 Symptoms: In short period of time total query execution time increases incredibly. Time Sum duration (ms) 17:17 12811 17:18 8870 17:19 33744 17:20 9991 17:21 13392 17:22 163928 17:23 1151709 17:24 12112797 -- here it cuts due to connection threshold 17:25 12305390 17:26 12970853 17:27 12957648 LA changes rather insignificantly from ~6 to ~8. CPU user time increases from ~350 to ~600 ), system from ~50 to ~250. Neither additional significant disc activity nor iowait. Another thing I noticed is autoanalyze finish on tables that few minutes later were used in the first and mostly canceled by timeout queries. First time I assigned the blame to multiply locks on one of the mentioned above tables. There was heavy delete. I started delete rows manually by small batches and found a record that hung my delete for a long time (many times greater then stmt timeout) even when I tried to delete it by PK. Situation was saved by disabling some functional that uses this table until next day when I got rid of the aggressive deletes. Second time I didn't find any reason that explains the situation. Was this situation mentioned before and is there a solution or workaround? (I didn't find any) If not please give me a glue where to dig or what information should I provide? Thank you. -- Regards, Sergey Konoplev -- PostgreSQL articles in english russian http://gray-hemp.blogspot.com/search/label/postgresql/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Peter Eisentraut wrote: On ons, 2009-11-11 at 10:25 -0500, Bruce Momjian wrote: There is a more worrisome/sinister possibility that I didn't want to mention in my first email --- that companies are hiring our most experienced developers and having them work almost exclusively on company-related or closed-source projects. I can't claim to know everyone's employment terms, but I think it's a bit of an illusion to think that the above hasn't always happened. Except for a handful of people who have very special job arrangements, Yes, it has always happened. I think the big difference is that those special employments are currently a fixed size, and the volume of patches has increased as our user-base has increased. I also think the bad economy is making it harder for people/companies to devote time to community stuff when paid work is available. I always had an assumption that those special employments would grow as the community grew, but I am not sure I am seeing that. Combine that with skill siphoning, and I get concerned. *everyone* is working almost exclusively on company-related or closed-source projects. The patches that get sent in are almost always either fallout from a customer/company project, or stuff that one of the closed-sourced forks has developed that they don't want to maintain, or stuff people do at night to make their lives easier in the distant future. All of those things are already special arrangements that people need to make with their employers and their lives, but they have discernible benefits. But you can't expect a lot of people or employers to reserve time on top of that for handholding other people's patches and for other community stuff that has no easy to measure benefits. Totally agree. It is that zero-return work that is hard to justify for people and companies. It is clearly something that requires self-sacrifice, and personally I think a culture of self-sacrifice is what has given us such great success and such a nuturing community environment. Historically, there was a golden era when Tom, Heikki, Neil Conway, Alvaro, and others all handled patches and things ran with much less of an individual burden than we have now, but that might have been an aberration. Personally, I don't think we have a major problem now, but I do think there is the chance of this getting worse in the future, and making people unhappy. I am brining it up now in case there is something we can do to avoid such unhappiness. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parsing config files in a directory
Let's NOT start that discussion again. Bruce's comments were a useful addition to the technical discussion. Yes, I'm just trying to avoid sidelining this into a discussion of search_path management commands, which we already failed to come to a consensus spec for earlier this year. Not that we don't need to have better search-path management, just that any discussion on that should be a separate thread. Overall, I'm seeing this patch proposal suffer from an extreme excess of bike-shedding. The original proposal was to have a directory where one could put config files; the revised spec was to allow directory includes in postgresql.conf. From there, this veered off into a discussion of how people *ought* to manage their configuration files. While interesting, it's irrelevant to the patch (and really ought to take place on pgsql-admin, anyway), which does nothing other than give DBAs and tool-writers more flexibility on how to manage PostgreSQL configurations. And in this project, we've *always* been about more flexibility, so it's hard for me to understand any objections to allowing directory includes ... especially when we already allow file includes. My proposal is this: (1) that we support the idea of a patch which allows people to add directory includes to postgresql.conf, in the same manner that we now support file includes, with files in the included directory to be processed alphanumerically. (2) that we put out a TODO for making the configuration variables which take lists able to take an accumulator as well as an assignment, syntax TBA. These two above seem like nice, small incremental changes to 8.5 which require no sweeping redesigns of how people handle conf files, but do allow people who want to develop new management strategies to do so. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Bruce, Well, I think the answer is to promote some new committers. When was the last time we added a committer? We have added a bunch of new reviewers and major contributors over the last 2 years. It's time to review their work and see who can be promoted to more responsibility for other people's patches. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Josh Berkus wrote: Bruce, Well, I think the answer is to promote some new committers. When was the last time we added a committer? We have added a bunch of new reviewers and major contributors over the last 2 years. It's time to review their work and see who can be promoted to more responsibility for other people's patches. Yes, I did consider that, and it might be a solution. The larger problem is that with many of the patches, it isn't commit-rights that is preventing people from being involved, but rather the expert knowledge necessary to make sure the patch works properly. For example, probably only Simon and Heikki are knowledge enough to apply the HS and SR patchs --- but those patch is handled. The harder part is the other patches where there are only a small pool of people knowledgeable enough to apply the patch, but many of those knowledgeable people are tied up with non-community things. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Boszormenyi Zoltan escribió: I have split up (and cleaned up a little) the dynamic cursorname patch into smaller logical, easier-to-review pieces. Descriptions below. 1) 1a-unified-optfromin-fetch-ctxdiff.patch ecpg supports optional FROM/IN in FETCH and MOVE statements (mainly because it's required by Informix-compatibility). Unify core and ecpg grammar as per Tom Lane's suggestion. I have applied this patch after some tinkering. I mainly added support for fetch_args: FORWARD opt_from_in name and BACKWARD opt_from_in name in ecpg.addons which apparently you forgot. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Alvaro Herrera alvhe...@commandprompt.com writes: I have applied this patch after some tinkering. Shouldn't there be a docs change in that commit? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
For example, probably only Simon and Heikki are knowledge enough to apply the HS and SR patchs --- but those patch is handled. The harder part is the other patches where there are only a small pool of people knowledgeable enough to apply the patch, but many of those knowledgeable people are tied up with non-community things. People learn to do this by doing it. That's how you learned, and Tom and Heikki. Ultimately, it's a source code control system; if someone screws up, we can always back it out. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Josh Berkus wrote: For example, probably only Simon and Heikki are knowledge enough to apply the HS and SR patchs --- but those patch is handled. The harder part is the other patches where there are only a small pool of people knowledgeable enough to apply the patch, but many of those knowledgeable people are tied up with non-community things. People learn to do this by doing it. That's how you learned, and Tom and Heikki. Ultimately, it's a source code control system; if someone screws up, we can always back it out. True, but even I avoid patches I don't understand, and practicing by applying them could lead to a very undesirable outcome, e.g. instability. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
True, but even I avoid patches I don't understand, and practicing by applying them could lead to a very undesirable outcome, e.g. instability. Right, but being responsible for applying the patch is the only real incentive anyone has to learn enough to understand its effects. If a contributor is not responsible, they can always think oh, Tom will get it, I'll learn that later. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Tom Lane escribió: Alvaro Herrera alvhe...@commandprompt.com writes: I have applied this patch after some tinkering. Shouldn't there be a docs change in that commit? True -- fixed. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Couldn't you have a policy that every patch is reviewed first by someone who wants to be an expert in that area, and then by someone who is currently an expert. Then you have the best of both worlds. The patch is reviewed by someone will make sure it won't cause problems, and the want to be expert gets training and will eventually be able to be the expert. On Nov 11, 2009, at 12:58 PM, Josh Berkus wrote: True, but even I avoid patches I don't understand, and practicing by applying them could lead to a very undesirable outcome, e.g. instability. Right, but being responsible for applying the patch is the only real incentive anyone has to learn enough to understand its effects. If a contributor is not responsible, they can always think oh, Tom will get it, I'll learn that later. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Rick Gigger wrote: Couldn't you have a policy that every patch is reviewed first by someone who wants to be an expert in that area, and then by someone who is currently an expert. Then you have the best of both worlds. The patch is reviewed by someone will make sure it won't cause problems, and the want to be expert gets training and will eventually be able to be the expert. The wannabe-experts can do all the reviewing they want today (and they do). But we cannot make this a hard requirement, or we risk stalling the project for patches that are not reviewed timely by non-experts. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On Wed, Nov 11, 2009 at 12:45 PM, Peter Eisentraut pete...@gmx.net wrote: The patches that get sent in are almost always either fallout from a customer/company project, or stuff that one of the closed-sourced forks has developed that they don't want to maintain, or stuff people do at night to make their lives easier in the distant future. All of those things are already special arrangements that people need to make with their employers and their lives, but they have discernible benefits. But you can't expect a lot of people or employers to reserve time on top of that for handholding other people's patches and for other community stuff that has no easy to measure benefits. Well, I'm perfectly willing to handhold other people's patches and do community stuff that has no easy to measure benefits. I'm willing to do it precisely because I believe it will make it easier to get my own patches accepted. AIUI, the whole point of CommitFests is that I agree to review other people's patches and they in turn agree to review mine, and we all get to scratch our respective itches. In fact, because I enjoy community work, I'm even willing to spend MORE time helping other people get their patches in than I do writing my own patches - but I'm only willing to spend 2x or 3x the time on other people's stuff that I spend on my own, not 10x. In the last two CommitFests it has become evident that we have a problem with freeloading. During the September CommitFest, the set of reviewers and the set of patch submitters were almost disjoint. Now, fortunately, we still had enough reviewers; the real shortage was of committers. But I think that some of the reviews were not as good as they could have been had they been reviewed by more experienced contributors, or even just by multiple people. A fair amount of easy stuff slipped through the review process. Tom cleaned it up, but he shouldn't have to be responsible for things that a read-through of the patch by an experienced C programmer should have caught. I tried to help, but I was fairly tied up with overall CommitFest management and did not have time for a full read-through of every patch. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Boszormenyi Zoltan escribió: 2) 1b-cursor_name-ctxdiff.patch name - cursor_name transition in core grammar and ecpg grammar. Currently it does nothing, it's a preparation phase. Depends on patch 1. Applied this part too. I cannot apply the other ones -- they belong to the ECPG maintainer. I hope I cleared his road a bit. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On Wed, Nov 11, 2009 at 2:51 PM, Bruce Momjian br...@momjian.us wrote: Josh Berkus wrote: For example, probably only Simon and Heikki are knowledge enough to apply the HS and SR patchs --- but those patch is handled. The harder part is the other patches where there are only a small pool of people knowledgeable enough to apply the patch, but many of those knowledgeable people are tied up with non-community things. People learn to do this by doing it. That's how you learned, and Tom and Heikki. Ultimately, it's a source code control system; if someone screws up, we can always back it out. True, but even I avoid patches I don't understand, and practicing by applying them could lead to a very undesirable outcome, e.g. instability. Well, if we're going to add committers (and I'm generally in favor of that) we need to pick people who won't apply patches that they don't understand. In a way, that's really the only HARD requirement for a committer, I think. If someone doesn't understand any aspect of the system well enough to commit patches, but they know that, then having them as a committer will be useless, but not actively harmful (beyond being a waste of time). On the other hand, someone who actually does know some parts of the system well enough to commit patches but doesn't have the discretion to avoid things that are more than they can tackle will create chaos. In other words, we should be looking for people who (1) know enough to be able to commit at least some patches without breaking the world and (2) are responsible enough to know which things they can commit and which they should leave alone. Of course, anyone you pick will probably make some mistakes, but that's the general idea... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Bruce Momjian wrote: True, but even I avoid patches I don't understand, and practicing by applying them could lead to a very undesirable outcome, e.g. instability. The usual type of practice here should come from applying trivial patches, or ones that don't impact code quality. Docs patches come to mind as a good way someone could get used to the commit process without introducing much potential mayhem along the way. As far as keeping new people away from complicated patches, ultimately you just have to trust that anyone who can commit has a reasonable idea of their own capabilities. I seriously doubt you're going to find a new committer jumping right in by committing hot standby out of the gate just because they could do so. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Robert Haas wrote: I tried to help, but I was fairly tied up with overall CommitFest management and did not have time for a full read-through of every patch. I think it's completely unreasonable to expect the CF manager to do any patch review themselves. It's a hard enough job to keep going without actually getting your hands into the details. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Selena Deckelmann wrote: Hi! On Tue, Nov 10, 2009 at 10:40 PM, Greg Smith gsm...@gregsmith.com wrote: On Sun, 8 Nov 2009, Robert Haas wrote: I would personally prefer not to be involved in the management of the next CommitFest. Having done all of the July CommitFest and a good chunk of the September CommitFest, I am feeling a bit burned out. I was just poking around on the Wiki, and it looks like the role of the CommitFest manager isn't very well documented yet. Since you've done all of them since introducing the new CF software, I'm not sure if anyone else even knows exactly what you've been doing. The transition over to that was so successful there isn't even a copy of the schedule for 8.5 on the Wiki itself. Could you find some time this week to rattle off an outline of the work involved? It's hard to decide whether to volunteer to help without having a better idea of what's required. It's pretty straightforward. Robert has actually done a great job of communicating about this to the patch reviewers. agreed I'd be happy to get together at some pre-appointed hour this weekend (Saturday / Sunday) to talk it over by phone / IRC. PDXPUG was already planning to get together to review some patches this Sunday from 3-5pm PST, so that is a convenient time for me. I can also help with commitfest admin tasks, but not in a dedicated way until after Thanksgiving as I'm going to be be traveling or on vacation. I would be interested in helping out as well but I won't be able to dedicate a lot of time before that timeframe as well :( Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] grant with hierarchy option
On fre, 2009-10-30 at 00:49 -0400, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: There is a gap in the permission scheme for inheritance setups. Say you have this: CREATE TABLE persons (...); CREATE TABLE employees (...) INHERITS (persons); GRANT SELECT ON persons TO foo; Then user foo can extract who the employees are using SELECT * FROM persons EXCEPT SELECT * FROM ONLY persons; And this is a problem why exactly? It's entirely likely that employee-ness can be determined just from what is visible in the persons view, anyway. Not to mention tableoid. Yeah, tableoid is a deal-breaker. But perhaps using ONLY should at least require SELECT privilege, because it effectively allows you to select a subset of the table's rows. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Sun, Nov 8, 2009 at 8:52 PM, Robert Haas robertmh...@gmail.com wrote: The next CommitFest is scheduled to start in a week. So far, it doesn't look too bad, though a lot could change between now and then. I would personally prefer not to be involved in the management of the next CommitFest. Having done all of the July CommitFest and a good chunk of the September CommitFest, I am feeling a bit burned out. why we need a full time manager at all? why not simply use -rrreviewers to track the status of a patch? of course, we hope the author or reviewer to change the status as appropiate but we have seen many people including missing discussions and changing the status of hanging patches... i guess we can make the commitfest app send an email stating that reviewer_foo is taking the patch bar, and maybe sending emails after some days if nothing has happened with that patch... and an email every week or every few days saying how many patches are, how many are being reviewed, how many hasn't been reviewed, and so on... then the remaining work should be not that much, no? -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On Wed, Nov 11, 2009 at 3:57 PM, Greg Smith g...@2ndquadrant.com wrote: Robert Haas wrote: I tried to help, but I was fairly tied up with overall CommitFest management and did not have time for a full read-through of every patch. I think it's completely unreasonable to expect the CF manager to do any patch review themselves. It's a hard enough job to keep going without actually getting your hands into the details. Well, that's another reason I'd kind of like to take a break from CF management - it would free up more of my time for actual patch review. I have been doing some review anyhow, but given as how I don't get paid for any of this, there's only a few hours a day that are potentially available for this, and it has to compete with the dishwasher breaking and my wife wanting to play Pandemic. By the way, if anyone wants to buy me a new dishwasher, I will put reviewing your patch at the top of my to do list. :-) ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] not logging caught exceptions
Is it at all reasonable to try to create some mechanism so that exceptions (elog) that are caught and not rethrown do not end up in the log? For example, when you write code that does something like try insert catch unique_constraint_violation update end try this will end up cluttering the logs with all the constraint violation messages. Now, we probably don't want to just suppress all logging in a TRY block until the END_TRY. So maybe this could be an argument to TRY or a separate statement that would typically be run right after TRY that names exception types to handle specially. Higher level languages such as PL/pgSQL could then peek ahead to the catch block to set this up. Comments? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Wed, Nov 11, 2009 at 4:21 PM, Jaime Casanova jcasa...@systemguards.com.ec wrote: why we need a full time manager at all? why not simply use -rrreviewers to track the status of a patch? of course, we hope the author or reviewer to change the status as appropiate but we have seen many people including missing discussions and changing the status of hanging patches... Well, actually, that's precisely were I've been putting in a ton of work - making sure patches aren't left hanging. If reviewers would do more of that work, this process would run a lot smoother and be much less onerous for the CM. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Bruce Momjian wrote: I also think the bad economy is making it harder for people/companies to devote time to community stuff when paid work is available. I think this explains away more of the recent situation than you're giving it credit for. When everybody's fat and happy and it's easy to generate/raise money, it's also easy to throw money toward the community. When times are tight, giving away work that you might charge for (or have already charged for) is harder for a company to justify. It's easy to plan to have someone do community work when you hire them, only to realize down the line that business has dried up enough that you're stuck with the choice between them doing that and a job that will make or break and upcoming payroll. And that's where a lot more businesses are at right now than at any time in a long while. After looking for an example of the boom/bust cycle impacting this community's work that's old enough to be clearer in hindsight, I would suggest noting that Great Bridge was officially announced in May of 2000 and was gone by the end of 2001. Overlay those dates on top of http://www.google.com/finance?q=INDEXNASDAQ:.IXIC after switching Zoom to show 10 years. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] grant with hierarchy option
Peter Eisentraut pete...@gmx.net writes: On fre, 2009-10-30 at 00:49 -0400, Tom Lane wrote: And this is a problem why exactly? It's entirely likely that employee-ness can be determined just from what is visible in the persons view, anyway. Not to mention tableoid. Yeah, tableoid is a deal-breaker. But perhaps using ONLY should at least require SELECT privilege, because it effectively allows you to select a subset of the table's rows. By that argument, WHERE clauses are a security hazard. It's still not apparent to me why it would be essential, or even a good idea, to prevent people from figuring out which rows belong to which subtable. Or do you mean that ONLY should be treated as requiring column select privilege on TABLEOID? Perhaps that's sensible. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Selena, I'd be happy to get together at some pre-appointed hour this weekend (Saturday / Sunday) to talk it over by phone / IRC. PDXPUG was already planning to get together to review some patches this Sunday from 3-5pm PST, so that is a convenient time for me. Aren't you running OpenSQL this weekend? --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Wed, Nov 11, 2009 at 4:40 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 11, 2009 at 4:21 PM, Jaime Casanova jcasa...@systemguards.com.ec wrote: why we need a full time manager at all? why not simply use -rrreviewers to track the status of a patch? of course, we hope the author or reviewer to change the status as appropiate but we have seen many people including missing discussions and changing the status of hanging patches... Well, actually, that's precisely were I've been putting in a ton of work - making sure patches aren't left hanging. that's why i guess sending automatic mails would be a good way to remember that a reviewer had a patch in their control or to tell reviewers/committers there are still patches for review/commit -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Selena Deckelmann wrote: On Tue, Nov 10, 2009 at 10:40 PM, Greg Smith gsm...@gregsmith.com wrote: I was just poking around on the Wiki, and it looks like the role of the CommitFest manager isn't very well documented yet. It's pretty straightforward. Robert has actually done a great job of communicating about this to the patch reviewers. That's good to hear. What I was hinting at was that some of the community knowledge here should start getting written down now that the process has matured, rather than trying to directly transfer just to one other person. I'm not sure if Robert has shared 100% of what he does with the reviewers or not, but in general the easiest way to divest yourself of a position is to document how someone else can do it. I don't know that having to poke through list archives or chat with someone is necessarily the best way to transfer that knowledge. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com
Re: [HACKERS] next CommitFest
On Wed, Nov 11, 2009 at 5:06 PM, Jaime Casanova jcasa...@systemguards.com.ec wrote: On Wed, Nov 11, 2009 at 4:40 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 11, 2009 at 4:21 PM, Jaime Casanova jcasa...@systemguards.com.ec wrote: why we need a full time manager at all? why not simply use -rrreviewers to track the status of a patch? of course, we hope the author or reviewer to change the status as appropiate but we have seen many people including missing discussions and changing the status of hanging patches... Well, actually, that's precisely were I've been putting in a ton of work - making sure patches aren't left hanging. that's why i guess sending automatic mails would be a good way to remember that a reviewer had a patch in their control or to tell reviewers/committers there are still patches for review/commit I think an automatic system would probably not be too valuable, but you're welcome to submit a patch against commitfest.postgresql.org (source code is published at git.postgresql.org). I'd recommend proposing a design on -hackers first. It's easy to generate systems that spew out a lot of email, but the system doesn't really have any understanding of what is really going on. When I send out emails to nag people, I actually put quite a bit of thought into what I say. Sometimes I try to summarize the current status of the patch, sometimes I add my own thoughts, sometimes I just fire off a one-liner. I think that adds value, but perhaps I overestimate myself. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Wed, Nov 11, 2009 at 5:14 PM, Greg Smith g...@2ndquadrant.com wrote: Selena Deckelmann wrote: On Tue, Nov 10, 2009 at 10:40 PM, Greg Smith gsm...@gregsmith.com wrote: I was just poking around on the Wiki, and it looks like the role of the CommitFest manager isn't very well documented yet. It's pretty straightforward. Robert has actually done a great job of communicating about this to the patch reviewers. That's good to hear. What I was hinting at was that some of the community knowledge here should start getting written down now that the process has matured, rather than trying to directly transfer just to one other person. I'm not sure if Robert has shared 100% of what he does with the reviewers or not, but in general the easiest way to divest yourself of a position is to document how someone else can do it. I don't know that having to poke through list archives or chat with someone is necessarily the best way to transfer that knowledge. I'll try to write something up. Stand by. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Wed, Nov 11, 2009 at 10:25:05PM +0100, Joachim Wieland wrote: Hi, Attached is a patch for a new listen/notify implementation. In a few words, the patch reimplements listen/notify as an slru-based queue which works similar to the sinval structure. Essentially it is a ring buffer on disk with pages mapped into shared memory for read/write access. While I can't really comment on the implementation, from your description it looks like a big improvement. Nice work! Have a nice day, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Please line up in a tree and maintain the heap invariant while boarding. Thank you for flying nlogn airlines. signature.asc Description: Digital signature
Re: [HACKERS] Listen / Notify rewrite
On Nov 11, 2009, at 4:25 PM, Joachim Wieland wrote: Hi, Attached is a patch for a new listen/notify implementation. In a few words, the patch reimplements listen/notify as an slru- based queue which works similar to the sinval structure. Essentially it is a ring buffer on disk with pages mapped into shared memory for read/write access. Additionally the patch does the following (see below for details): 1. It removes the pg_listener relation and 2. adds the possibility to specify a payload parameter, i.e. executing in SQL NOTIFY foo 'payload'; and 'payload' will be delivered to any listening backend. 3. Every distinct notification is delivered. 4. Order is preserved, i.e. if txn 1 first does NOTIFY foo, then NOTIFY bar, a backend (listening to both foo and bar) will always first receive the notification foo and then the notification bar. 5. It's now listen to a channel, not listen to a relation anymore... Hi Joachim, Thank you for implementing this- LISTEN/NOTIFY without a payload has been a major problem to work around for me. I understand that coalescing multiple notifications of the same name happens now, but I never understood why. I had hoped that someone revisiting this feature would remove it. My use case is autonomous transactions. From a developer's standpoint, NOTIFY specifies a form of remote trigger for further action- outside the transaction- to occur. From that point of view, I don't see why NOTIFY foo; NOTIFY foo; is equivalent to NOTIFY foo;. I understand the use case where a per-row trigger could generate lots of spurious notifications, but it seems that if anything is generating spurious notifications, the notification is in the wrong place. The documentation makes the strong implication that notification names are usually table names, but with the new payload, this becomes even less the case. I changed this table, take a look at it to see what's new. But no such association is enforced by the NOTIFY and LISTEN commands. http://www.postgresql.org/docs/8.4/static/sql-notify.html With the coalescing, I feel like I am being punished for using NOTIFY with something other than a table name use case. At least with this new payload, I can set the payload to the transaction ID and be certain that all the notifications I sent are processed (and in order even!) but could you explain why the coalescing is still necessary? While coalescing could be achieved on the receiving-end NOTIFY callback ( if(payload ID was already processed) continue; ), non- coalescing behavior cannot be achieved when the backend does the coalescing. For backwards compatibility, payload-less NOTIFY could coalesce while NOTIFYs with a payload would not coalesce, but, on the other hand, that might be confusing. In any case, thank you for improving this long-neglected subsystem! Best regards, M -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
2. adds the possibility to specify a payload parameter, i.e. executing in SQL NOTIFY foo 'payload'; and 'payload' will be delivered to any listening backend. Thank you for implementing this- LISTEN/NOTIFY without a payload has been a major problem to work around for me. +1 -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Robert Haas escreveu: I think an automatic system would probably not be too valuable I have the same impression. It's easy to generate systems that spew out a lot of email, but the system doesn't really have any understanding of what is really going on. When I send out emails to nag people, I actually put quite a bit of thought into what I say. Sometimes I try to summarize the current status of the patch, sometimes I add my own thoughts, sometimes I just fire off a one-liner. I think that adds value, but perhaps I overestimate myself. But I think we should try to have multiple CMs so we don't burn out someone else. That group (a small one) could do the same job Robert has been done (AFAICS a very good job) but in a distributed way. I certainly could be a CM if I don't have to dedicate too much time at the CF month. I don't know if it would be as effective as it has been done but it will take that weight off our CM's shoulders. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On 11/11/09 12:55 PM, Greg Smith wrote: I seriously doubt you're going to find a new committer jumping right in by committing hot standby out of the gate just because they could do so. We'd be lucky to get them to *read* the Hot Standby patch and comment on it. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
/* + * This function is executed for every notification found in the queue in order + * to check if the current backend is listening on that channel. Not sure if we + * should further optimize this, for example convert to a sorted array and + * allow binary search on it... + */ + static bool + IsListeningOn(const char *channel) I think a bsearch would be needed. Very busy servers that make heavy use of notifies would be quite a penalty. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deadlock on the same object?
Tom Lane t...@sss.pgh.pa.us wrote: I think this is an artifact of the fact that SQL functions parse the whole querystring before executing any of it. Parsing of DELETE FROM a will result in acquiring ROW EXCLUSIVE lock on a, and then when the LOCK commands are executed, you have a lock-upgrade scenario and the deadlock is unsurprising. Thanks. It's a surprise for me :-). There was some discussion of changing that awhile ago, but I forget what the conclusion was. In any case nothing's been done about it. We cannot make a package of locking controls in a function under the current behavior. It would be good to improve this area. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Partitioning option for COPY
Emmanuel Cecchet m...@asterdata.com wrote: I have extracted the partitioning option for COPY (removed the error logging part) from the previous patch. We can use an INSERT trigger to route tuples into partitions even now. Why do you need an additional router for COPY? Also, it would be nicer that the router can works not only in COPY but also in INSERT. BTW, I'm working on meta data of partitioning now. Your partitioning option in COPY could be replaced with the catalog. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Thu, Nov 12, 2009 at 1:04 AM, Andrew Chernow a...@esilo.com wrote: I think a bsearch would be needed. Very busy servers that make heavy use of notifies would be quite a penalty. In such an environment, how many relations/channels is a backend typically listening to? Do you have average / maximal numbers? Regards, Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] not logging caught exceptions
Peter Eisentraut pete...@gmx.net writes: Is it at all reasonable to try to create some mechanism so that exceptions (elog) that are caught and not rethrown do not end up in the log? For example, when you write code that does something like try insert catch unique_constraint_violation update end try this will end up cluttering the logs with all the constraint violation messages. Really? It's not supposed to. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Partitioning option for COPY
Hi, I have extracted the partitioning option for COPY (removed the error logging part) from the previous patch. We can use an INSERT trigger to route tuples into partitions even now. Why do you need an additional router for COPY? Tom has already explained on the list why using a trigger was a bad idea (and I know we can use a trigger since I am the one who wrote it). If you look at the code you will see that you can do optimizations in the COPY code that you cannot do in the trigger. Also, it would be nicer that the router can works not only in COPY but also in INSERT. As 8.5 will at best provide a syntactic hack on top of the existing constraint implementation, I think that it will not hurt to have routing in COPY since we will not have it anywhere otherwise. BTW, I'm working on meta data of partitioning now. Your partitioning option in COPY could be replaced with the catalog. This implementation is only for the current 8.5 and it will not be needed anymore once we get a fully functional partitioning in Postgres which seems to be for a future version. Best regards, Emmanuel -- Emmanuel Cecchet Aster Data Web: http://www.asterdata.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Joachim Wieland wrote: On Thu, Nov 12, 2009 at 1:04 AM, Andrew Chernow a...@esilo.com wrote: I think a bsearch would be needed. Very busy servers that make heavy use of notifies would be quite a penalty. In such an environment, how many relations/channels is a backend typically listening to? Do you have average / maximal numbers? We have a system where many libpq clients, ~2000 - 4000 per server (depends on hardware), maintain a persistent backend connection. Each connection listens for notifies, LISTEN 'client_xxx'. There are maybe 10 different reasons why a NOTIFY 'client_xxx' is fired. Sometimes, notifies are broadcasted to all client connections, or just portions of them. The goal is real-time messaging to a large groups of computers/devices. Past 4000, the problem is distributed to a second, third, etc... server. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Andrew Chernow a...@esilo.com writes: + * This function is executed for every notification found in the queue in order + * to check if the current backend is listening on that channel. Not sure if we + * should further optimize this, for example convert to a sorted array and + * allow binary search on it... I think a bsearch would be needed. Very busy servers that make heavy use of notifies would be quite a penalty. Premature optimization is the root of all evil ;-). Unless you've done some profiling and can show that this is a hot spot, making it more complicated isn't the thing to be doing now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Partitioning option for COPY
Emmanuel Cecchet m...@asterdata.com wrote: If you look at the code you will see that you can do optimizations in the COPY code that you cannot do in the trigger. Since the optimizations is nice, I hope it will work not only in COPY but also in INSERT. An idea is moving the partitioning cache into Relation cache, and also moving the routing routines into heap_insert(). My concern is just in the modified position; I think you don't have to change your logic itself. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Martijn == Martijn van Oosterhout klep...@svana.org writes: Hi, Attached is a patch for a new listen/notify implementation. In a few words, the patch reimplements listen/notify as an slru-based queue which works similar to the sinval structure. Essentially it is a ring buffer on disk with pages mapped into shared memory for read/write access. Martijn While I can't really comment on the implementation, from your Martijn description it looks like a big improvement. Does it cope with the case where a trigger is doing NOTIFY, and you do a whole-table update, therefore dumping potentially millions of notifications in at once? (for example a rare maintenance operation on a table which has a listen/notify arrangement triggered by single inserts or updates) The existing implementation copes with that just fine. -- Andrew (irc:RhodiumToad) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Partitioning option for COPY
We can use an INSERT trigger to route tuples into partitions even now. Why do you need an additional router for COPY? Also, it would be nicer that the router can works not only in COPY but also in INSERT. Yeah, but performance on an insert trigger is impractical for large volumes of data. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] NULL-handling in aggregate(DISTINCT ...)
Quoth the comments in nodeAgg.c:
* We don't currently implement DISTINCT aggs for aggs having more
* than one argument. This isn't required for anything in the SQL
* spec, but really it ought to be implemented for
* feature-completeness. FIXME someday.
and:
* DISTINCT always suppresses nulls, per SQL spec, regardless of the
* transition function's strictness.
(What the SQL spec actually says is that aggregate calls which are
general set operation ignore all nulls regardless of whether they
are ALL or DISTINCT. Other kinds of aggregates are not permitted by
the spec to use ALL or DISTINCT.)
Currently we have this behaviour:
postgres=# select array_agg(all a) from (values (1),(null)) v(a);
array_agg
---
{1,NULL}
(1 row)
postgres=# select array_agg(distinct a) from (values (1),(null)) v(a);
array_agg
---
{1}
(1 row)
which personally I feel is somewhat wrong, since 1 and NULL are in
fact distinct, but which is due to the logic expressed in the second
comment above. (The spec does not allow array_agg(distinct a) so it
is no help here.)
Now the question: If the limit of one argument for DISTINCT aggs were
removed (which I'm considering doing as part of an update to the
aggregate ORDER BY patch I posted a while back), what should be the
behaviour of agg(distinct x,y) where one or both of x or y is null?
And should it depend on the strictness of the transition function?
--
Andrew (irc:RhodiumToad)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] NULL-handling in aggregate(DISTINCT ...)
Andrew Gierth and...@tao11.riddles.org.uk writes: Now the question: If the limit of one argument for DISTINCT aggs were removed (which I'm considering doing as part of an update to the aggregate ORDER BY patch I posted a while back), what should be the behaviour of agg(distinct x,y) where one or both of x or y is null? And should it depend on the strictness of the transition function? I think you could probably just change it: make DISTINCT work as per regular DISTINCT (treat null like a value, keep one copy). All the spec-conforming aggregates are strict and would ignore the null in the next step anyway. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Unpredictable shark slowdown after migrating to 8.4
On Wed, Nov 11, 2009 at 12:50 PM, Sergey Konoplev gray...@gmail.com wrote: Was this situation mentioned before and is there a solution or workaround? (I didn't find any) If not please give me a glue where to dig or what information should I provide? I think you should use log_min_duration_statement or auto_explain to figure out which queries are giving you grief. I don't believe that 8.4 is in general slower than 8.3, so there must be something about how you are using it that is making it slower for you. But without more information it's hard to speculate what it might be. It's also not entirely clear that your 8.4 setup is really the same as your 8.3 setup. You might have different configuration, differences in your tables or table data, differences in workload, etc. Without controlling for all those factors it's hard to draw any conclusions. Also, I don't believe this is an appropriate topic for pgsql-hackers. If you have EXPLAIN ANALYZE results for the affected queries, try pgsql-performance. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] NULL-handling in aggregate(DISTINCT ...)
Tom == Tom Lane t...@sss.pgh.pa.us writes: Andrew Gierth and...@tao11.riddles.org.uk writes: Now the question: If the limit of one argument for DISTINCT aggs were removed (which I'm considering doing as part of an update to the aggregate ORDER BY patch I posted a while back), what should be the behaviour of agg(distinct x,y) where one or both of x or y is null? And should it depend on the strictness of the transition function? Tom I think you could probably just change it: make DISTINCT work as Tom per regular DISTINCT (treat null like a value, keep one copy). Tom All the spec-conforming aggregates are strict and would ignore Tom the null in the next step anyway. Change it for single-arg DISTINCT too? And the resulting change to the established behaviour of array_agg is acceptable? Just want to be clear here. -- Andrew. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] NULL-handling in aggregate(DISTINCT ...)
Andrew Gierth and...@tao11.riddles.org.uk writes: Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom I think you could probably just change it: make DISTINCT work as Tom per regular DISTINCT (treat null like a value, keep one copy). Tom All the spec-conforming aggregates are strict and would ignore Tom the null in the next step anyway. Change it for single-arg DISTINCT too? And the resulting change to the established behaviour of array_agg is acceptable? Just want to be clear here. I doubt that very many people are depending on the behavior of array_agg(DISTINCT); and anyway it could be argued that the present behavior is a bug, since it doesn't work like standard DISTINCT. I don't see a problem with changing it, though it should be release-noted. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Premature optimization is the root of all evil ;-). Unless you've done some profiling and can show that this is a hot spot, making it more complicated isn't the thing to be doing now. I'm thinking of how our system uses/abuses notifies, and began wondering if several thousand backends listening with a large queue would perform decently behind a linear search. At this point, I have no data either way; only an assumption based off being burnt by sequential scans in the past ;) -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Wed, Nov 11, 2009 at 5:48 PM, A.M. age...@themactionfaction.com wrote: At least with this new payload, I can set the payload to the transaction ID and be certain that all the notifications I sent are processed (and in order even!) but could you explain why the coalescing is still necessary? Christmas comes early this year! :-). three reasons: *) it works that way now...a lot of people use this feature for all kinds of subtle things and the behavior chould change as little as possible *) legacy issues aside, I think it's generally better behavior (how many times do you need to be tapped on the shoulder?) *) since you can trivially differentiate it (using xid, sequence, etc), what's the fuss? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] write ahead logging in standby (streaming replication)
Hi, Should the standby also have to follow the WAL rule during recovery? The current patch doesn't care about the write order of the data page and WAL in the standby. So, after both servers fail, restarting the ex-standby by itself might corrupt the data. If the standby follows the WAL rule, walreceiver might delay in writing WAL records until the startup process' or bgwriter's fsync have been finished. I'm a bit concerned that such delay might increase the performance overhead on the primary. Thought? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
On Wed, Nov 11, 2009 at 5:27 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 11, 2009 at 5:14 PM, Greg Smith g...@2ndquadrant.com wrote: Selena Deckelmann wrote: On Tue, Nov 10, 2009 at 10:40 PM, Greg Smith gsm...@gregsmith.com wrote: I was just poking around on the Wiki, and it looks like the role of the CommitFest manager isn't very well documented yet. It's pretty straightforward. Robert has actually done a great job of communicating about this to the patch reviewers. That's good to hear. What I was hinting at was that some of the community knowledge here should start getting written down now that the process has matured, rather than trying to directly transfer just to one other person. I'm not sure if Robert has shared 100% of what he does with the reviewers or not, but in general the easiest way to divest yourself of a position is to document how someone else can do it. I don't know that having to poke through list archives or chat with someone is necessarily the best way to transfer that knowledge. I'll try to write something up. Stand by. Here's an attempt. http://wiki.postgresql.org/wiki/Running_a_CommitFest ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
Fujii Masao masao.fu...@gmail.com writes: Should the standby also have to follow the WAL rule during recovery? The current patch doesn't care about the write order of the data page and WAL in the standby. So, after both servers fail, restarting the ex-standby by itself might corrupt the data. Surely the receiver should fsync the WAL itself to disk before acknowledging it. Assuming you've done that, I don't see any corruption risk. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Nov 11, 2009, at 9:28 PM, Merlin Moncure wrote: On Wed, Nov 11, 2009 at 5:48 PM, A.M. age...@themactionfaction.com wrote: At least with this new payload, I can set the payload to the transaction ID and be certain that all the notifications I sent are processed (and in order even!) but could you explain why the coalescing is still necessary? Christmas comes early this year! :-). three reasons: *) it works that way now...a lot of people use this feature for all kinds of subtle things and the behavior chould change as little as possible *) legacy issues aside, I think it's generally better behavior (how many times do you need to be tapped on the shoulder?) *) since you can trivially differentiate it (using xid, sequence, etc), what's the fuss? Except for the fact that the number of times a notification occurred may be valuable information. I thought of a compromise: add the number of times a notification was generated (coalesced count+1) to the callback data. That would satisfy any backwards compatibility concerns and my use case too! Cheers, M -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Joachim Wieland j...@mcknight.de writes: However, if for some reason we cannot write to the slru files in the pg_notify/ directory we might want to roll back the current transaction but with the proposed patch we cannot because we have already committed... I think this is a deal-breaker, and arguing about how the pg_notify directory ought to be writable is not even slightly acceptable --- out of disk space is an obvious risk. The existing implementation guarantees that notifications obey ACID: if you commit, they are sent, and if you don't, they aren't. You can't just put in something that works most of the time instead. One possible solution would be to write to the queue before committing and adding the TransactionID. Then other backends can check if our TransactionID has successfully committed or not. Not sure if this is worth the overhead however... That sounds reasonable, and it's certainly no more overhead than the tuple visibility checks that happen in the current implementation. 2. The payload parameter is optional. A notifying client can either call NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is currently limited to 128 characters... Not sure if we should allow longer payload strings... Might be a good idea to make the max the same as the max length for prepared transaction GUIDs? Not sure anyone would be shipping those around, but it's a pre-existing limit of about the same size. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
I thought of a compromise: add the number of times a notification was generated (coalesced count+1) to the callback data. That would satisfy any backwards compatibility concerns and my use case too! If you are suggesting that the server poke data into the notifier's opaque payload, I vote no. Maybe the NOTIFY command can include a switch to enable this behavior. No syntax suggestions at this point. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
The following email expresses my personal opinion and does not reflect the opinion of my employers. Bruce Momjian wrote: I also think the bad economy is making it harder for people/companies to devote time to community stuff when paid work is available. Actually the bad economy should be a booster for open source projects. There should be more developers with time to acquire new skills on projects that will get them a better job when the economy comes back. I think the problems are more rooted in the developer community itself. The pg-hackers mailing list is probably the less socially skilled developer community I have ever seen in all the open source projects I have been involved with. A very high standard is set for contributions, which is good for the quality of the code, but the lack of development process and clear decision chain turns every new contributor into endless frustration. For a patch to be committed, a vague consensus has to arise among the strong technical voice(s) (usually convincing Tom is enough). If a more complex feature needs to be implemented, the lack of decision process ends up in a first long round of emails until everybody gets tired of it. Then sometimes later someone tries to re-activate the debate for another round and so on (partitioning is a good example). You lost potential committers at each of these rounds. The way I see it, most companies try to push their agenda, contribute their patches back to the community if it works and just go with their own fork and closed implementation if this is too much work or burden. Whatever the economy, very few people can commit to an indefinite amount of time to get a feature integrated in Postgres. Now you should probably ask yourself what you should do as a community to get more committers? Like it was said at PGCon, to get a patch in, it is going to be hard and painful. How do you make it less hard and less painful? On the other end, how do we, simple developers, become better to reach the status of committers? How can we endure the constant bashing especially in the early stages of our learning phase (especially if you are not paid to do it)? How do we justify to our employers that they should persist through this long process without visibility, so that eventually their contribution will make it back to the community? How do we make it easier for companies to contribute code? The lightness of the development process (no project manager, no steering committee, no voting, no product management, ...) is both a strength and a weakness that makes Postgres what it is today. The commitfest started to address some of the most obvious issues but there is probably much more that can be done by looking at what is happening in the other open source communities. Even if the economy is hard, I found time to invest my own 2 cents in this email ;-) Emmanuel -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
Andrew Chernow a...@esilo.com writes: I thought of a compromise: add the number of times a notification was generated (coalesced count+1) to the callback data. That would satisfy any backwards compatibility concerns and my use case too! If you are suggesting that the server poke data into the notifier's opaque payload, I vote no. Maybe the NOTIFY command can include a switch to enable this behavior. No syntax suggestions at this point. I agree, we should not have the system modifying the payload string for this. And don't bother suggesting a third column in the result --- we'd have to change the FE/BE protocol for that, and it's not going to happen. The existing precedent is that the system collapses identical notifications without payloads. So we could possibly get away with saying that identical payload-less notifies are collapsed but those with a payload are not. That doesn't really seem to satisfy the POLA though. I think Joachim's definition is fine, and anyone who needs delivery of distinct notifications can easily make his payload strings unique to ensure it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
On Thu, Nov 12, 2009 at 12:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: Should the standby also have to follow the WAL rule during recovery? The current patch doesn't care about the write order of the data page and WAL in the standby. So, after both servers fail, restarting the ex-standby by itself might corrupt the data. Surely the receiver should fsync the WAL itself to disk before acknowledging it. Assuming you've done that, I don't see any corruption risk. acknowledging it means letting the startup process know the arrival of WAL records? If so, I agree that there is no risk of data corruption. The problem is that fsync needs to be issued too frequently, which would be harmless in asynchronous replication, but not in synchronous one. A transaction would have to wait for the primary's and standby's fsync before returning a success to a client. So I'm inclined to change the startup process and bgwriter, instead of walreceiver, so as to fsync the WAL for the WAL rule. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
On Nov 11, 2009, at 10:43 PM, Tom Lane wrote: Andrew Chernow a...@esilo.com writes: I thought of a compromise: add the number of times a notification was generated (coalesced count+1) to the callback data. That would satisfy any backwards compatibility concerns and my use case too! If you are suggesting that the server poke data into the notifier's opaque payload, I vote no. Maybe the NOTIFY command can include a switch to enable this behavior. No syntax suggestions at this point. I agree, we should not have the system modifying the payload string for this. And don't bother suggesting a third column in the result --- we'd have to change the FE/BE protocol for that, and it's not going to happen. The existing precedent is that the system collapses identical notifications without payloads. So we could possibly get away with saying that identical payload-less notifies are collapsed but those with a payload are not. That doesn't really seem to satisfy the POLA though. I think Joachim's definition is fine, and anyone who needs delivery of distinct notifications can easily make his payload strings unique to ensure it. The notification count could be a secondary payload which does not affect the first, but I guess I'm the only one complaining about the coalescing... -M -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Listen / Notify rewrite
2. The payload parameter is optional. A notifying client can either call NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is currently limited to 128 characters... Not sure if we should allow longer payload strings... Might be a good idea to make the max the same as the max length for prepared transaction GUIDs? Not sure anyone would be shipping those around, but it's a pre-existing limit of about the same size. I don't see any reason to impose such a small limit on payload size. Surely some limit must exist, but 128 characters seems awfully small. I already have at few places in mind that would require more bytes. Why not 8K, 64K, 256K, 1M or even more? Is there some other factor in play forcing this limitation? -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] NULL-handling in aggregate(DISTINCT ...)
Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom I think you could probably just change it: make DISTINCT work as Tom per regular DISTINCT (treat null like a value, keep one copy). Tom All the spec-conforming aggregates are strict and would ignore Tom the null in the next step anyway. Change it for single-arg DISTINCT too? And the resulting change to the established behaviour of array_agg is acceptable? Just want to be clear here. Tom I doubt that very many people are depending on the behavior of Tom array_agg(DISTINCT); and anyway it could be argued that the Tom present behavior is a bug, since it doesn't work like standard Tom DISTINCT. I don't see a problem with changing it, though it Tom should be release-noted. A followup question: currently the code uses the datum interface for tuplesort. Obviously with multiple columns the slot interface is used instead; but is there any performance advantage for staying with the datum interface for the single-column case? -- Andrew. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
Hi, True, but even I avoid patches I don't understand, and practicing by applying them could lead to a very undesirable outcome, e.g. instability. How about having a staging server to help around novice committers? Basically the selected new band of people can take a patch, review it and if they deem it fit for checkin, they could check it into the staging server. Then they should try out the regression runs (we do have very few test cases) and other sanity checks to ensure that things are sane. Probably the buildfarm infrastructure can also be used to run against this staging server on all platforms. That ways, we can avoid the initial instability upto some extent and then noob committer can then muster courage to atleast tell the core-committers what they think about the staged patch commit. Regards, Nikhils -- http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] next CommitFest
Robert Haas wrote: Here's an attempt. http://wiki.postgresql.org/wiki/Running_a_CommitFest Perfect, that's the sort of thing I was looking for the other day but couldn't find anywhere. I just made a pass through better wiki-fying that and linking it to the related pages in this area. Two things look to be true at the moment: 1) The call for reviewers is already running late and needs to start ASAP. 2) Some of the experienced helpers from the previous CFs, like Selena, should eventually be able to help, just everybody is busy during when the first round of action has to happen here. Given all that, I'm thinking that unless we get an enthusiastic volunteer by tomorrow, I'll kick off the call for reviewers myself and follow that through to initial patch assignments. I don't expect to have as much time as Robert put into the last couple of CommitFests after that, but this one looks smaller and with more familiar patches than those. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] write ahead logging in standby (streaming replication)
Fujii Masao wrote: The problem is that fsync needs to be issued too frequently, which would be harmless in asynchronous replication, but not in synchronous one. A transaction would have to wait for the primary's and standby's fsync before returning a success to a client. So I'm inclined to change the startup process and bgwriter, instead of walreceiver, so as to fsync the WAL for the WAL rule. Let's keep it simple for now. Just make the walreceiver do the fsync. We can optimize later. For now, we're only going to have async mode anyway. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch committers
On ons, 2009-11-11 at 22:30 -0500, Emmanuel Cecchet wrote: On the other end, how do we, simple developers, become better to reach the status of committers? How can we endure the constant bashing especially in the early stages of our learning phase (especially if you are not paid to do it)? How do we justify to our employers that they should persist through this long process without visibility, so that eventually their contribution will make it back to the community? That's basically just it: Assume bashing is part of the process. Don't think of it as bashing. Take the constructive criticism from it, ignore the rest. Assume only one out of three feature ideas will make it. Apply the prerequisite amount of gamesmanship to the system and tune your bikeshedding detectors. Don't take anything personally. Live and learn. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
