Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 23, 2013 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: I think there's been something of a professionalization of PostgreSQL development over the last few years. More and more people are able to get paid to work on PostgreSQL as part or in a few cases all of their job. This trend includes me, but also a lot of other people. There are certainly good things about this, but I think it increases the pressure to get patches committed. Also many of the new developers who were previously working from proprietary companies (includes me) may not have seen so long cycles for feature development, from design to acceptance/rejection. The problem aggravates because the same developer was previously doing development at much much faster pace and will find our style very conservative. Of course, the quality of work might be a notch lower when you are coding features at that pace and you may introduce regression and bugs on the way, but that probably gets compensated by more structured QA and testing that proprietary companies do. And many of these developers might be working on equally important and mission critical products even before. I know we are very conscious of our code quality and product stability, and for the right reasons, but I wonder the overall productivity will go up if we do the same i.e. have quick feature acceptance cycles compensated by more QA. The problem is being a community driven project we attract more developers but very less QA people. Would it help to significantly enhance our regression coverage so that bugs are caught early (assuming that's a problem) ? If so, may be we should have a month-long QAfest once where every developer is only writing test cases. Would it help to have a formal product manager (or a team) that gets to decide whether we want a particular feature or not ? This might be similar to our current model of discussion on hackers but more time bound and with the authority to the team to accept/reject ideas at the end of the time period and not keeping it vague for later decision after people have put in a lot of efforts already. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Thursday, January 24, 2013 2:58 AM Stephen Frost wrote: Heikki, * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: FWIW, here's how I feel about some the patches. It's not an exhaustive list. Thanks for going through them and commenting on them. built-in/SQL Command to edit the server configuration file (postgresql.conf) I think this should be a pgfoundry project, not in core. At least for now. I don't think it would ever get deployed or used then.. I think for this feature there has been lot of test already done and reviewed by quite a few people. About the feature's value and high level design, there is quite a lot of discuss already happened. At this point, there is no design level concern for this patch, there are few issues which can be dealt with minor changes. Do you see any lack of importance for this feature? With Regards, Amit Kapila. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Tue, Jan 22, 2013 at 1:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah, and a lot more fairly-new developers who don't understand all the connections in the existing system. Let me just push back a bit here: based on the amount of time I've had to spend fixing bugs over the past five months, 9.2 was our worst release ever. I don't like that trend, and I don't want to see it continued because we get laxer about accepting patches. IMO we are probably too lax already. Really? Hmm, that's not good. I seem to recall 8.4.x being pretty bad, and some of the recent bugs we fixed were actually 9.1.x problems that slipped through the cracks. For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. It seems though that less and less of that is actually happening, so I wonder whether the commitfest process is encouraging inefficiency by making people think they should start a discussion with a mostly-complete patch. Or maybe CF review is just crowding out any serious discussion of rough ideas. There was some discussion at the last dev meeting of creating a design review process within commitfests, but nothing got done about it ... I think there's been something of a professionalization of PostgreSQL development over the last few years. More and more people are able to get paid to work on PostgreSQL as part or in a few cases all of their job. This trend includes me, but also a lot of other people. There are certainly good things about this, but I think it increases the pressure to get patches committed. If you are developing a patch on your own time and it doesn't go in, it may be annoying but that's about it. If you're getting paid to get that patch committed and it doesn't go in, either your boss cares or your customer cares, or both, so now you have a bigger problem. Of course, this isn't always true: I don't know everyone's employment arrangements, but there are some people who are paid to work on PostgreSQL with no real specific agenda, just a thought of generally contributing to the community. However, I believe this to be less common than an arrangement involving specific deliverables. Whatever the arrangement, jobs where you get to work on PostgreSQL as part of your employment mean that you can get more stuff done. Whatever you can get done during work time plus any nonwork time you care to contribute will be more than what you could get done in the latter time alone. And I think we're seeing that, too. That then puts more pressure on the people who need to do reviews and commits, because there's just more stuff to look at, and you know, a lot of it is really good stuff. A lot of it has big problems, too, but we could be doing a lot worse. Nonwithstanding, it's a lot of work, and the number of people who know the system well enough to review the really difficult patches is growing, but not as fast as the number of people who have time to write them. For all of that, I'm not sure that people failing to seek consensus before coding is really so much of a problem as you seem to think. A lot of the major efforts underway for this release have been discussed extensively on multiple threads. The fact that some of those ideas may be less than half-baked at this point may in some cases be the submitter's fault, but there also cases where it's just that they haven't got enough looking-at from the people who know enough to evaluate them in detail, and perhaps some cases that are really nobody's fault: nothing in life is going to be perfect all the time no matter how hard everyone tries. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 2013-01-23 11:44:29 -0500, Robert Haas wrote: On Tue, Jan 22, 2013 at 1:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah, and a lot more fairly-new developers who don't understand all the connections in the existing system. Let me just push back a bit here: based on the amount of time I've had to spend fixing bugs over the past five months, 9.2 was our worst release ever. I don't like that trend, and I don't want to see it continued because we get laxer about accepting patches. IMO we are probably too lax already. Really? Hmm, that's not good. I seem to recall 8.4.x being pretty bad, and some of the recent bugs we fixed were actually 9.1.x problems that slipped through the cracks. FWIW I concur with Tom's assessment. On Tue, Jan 22, 2013 at 1:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. It seems though that less and less of that is actually happening, so I wonder whether the commitfest process is encouraging inefficiency by making people think they should start a discussion with a mostly-complete patch. Or maybe CF review is just crowding out any serious discussion of rough ideas. There was some discussion at the last dev meeting of creating a design review process within commitfests, but nothing got done about it ... Are you thinking of specific patches? From what I remember all all the bigger patches arround the 9.3 cycle were quite heavily discussed during multiple stages of their development. I agree that its not necessarily the case for smaller patches but at least for me in those cases its often hard to judge how complex something is before doing an initial prototype. One aspect of this might be that its sometimes easier to convince -hackers of some idea if you can prove its doable. Another that the amount of bikeshedding seems to be far lower if a patch already shapes a feature in some way. I think there's been something of a professionalization of PostgreSQL development over the last few years. More and more people are able to get paid to work on PostgreSQL as part or in a few cases all of their job. This trend includes me, but also a lot of other people. Yes. There are certainly good things about this, but I think it increases the pressure to get patches committed. If you are developing a patch on your own time and it doesn't go in, it may be annoying but that's about it. If you're getting paid to get that patch committed and it doesn't go in, either your boss cares or your customer cares, or both, so now you have a bigger problem. And it also might shape the likelihood of getting paid for future work, be that a specific patch or time for hacking/maintenance. For all of that, I'm not sure that people failing to seek consensus before coding is really so much of a problem as you seem to think. A lot of the major efforts underway for this release have been discussed extensively on multiple threads. The fact that some of those ideas may be less than half-baked at this point may in some cases be the submitter's fault, but there also cases where it's just that they haven't got enough looking-at from the people who know enough to evaluate them in detail, and perhaps some cases that are really nobody's fault: nothing in life is going to be perfect all the time no matter how hard everyone tries. I agree. Take logical replication/decoding for example. While we developed a prototype first, we/I tried to take in as much feedback from the community as we could. Just about no code, and only few concepts, from the initial prototype remain, and thats absolutely good. I don't think a more talk about it first approach would have helped here. Do others disagree? But as soon as the rough, rough design was laid out the amount of specific feedback shrank. Part of that is that some people involved in the discussions had changes in their work situation that influenced the amount of time they could spend on it, but I think one other problem is that at some point it gets hard to give feedback on a complex, evolving patch without it eating up big amount of time. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 02:04:14PM -0500, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: IMHO that's the single most important task of a review. Really? I'd say the most important task for a review is does the patch do what it says it does?. That is, if the patch is supposed to implement feature X, does it actually? If it's a performance patch, does performance actually improve? If the patch doesn't implement what it's supposed to, who cares what the code looks like? But even before that, you have to ask whether what it's supposed to do is something we want. Yep. Our TODO list has a pretty short summary of this at the top: Desirability - Design - Implement - Test - Review - Commit -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/23/2013 09:08 AM, Andres Freund wrote: On 2013-01-23 11:44:29 -0500, Robert Haas wrote: On Tue, Jan 22, 2013 at 1:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah, and a lot more fairly-new developers who don't understand all the connections in the existing system. Let me just push back a bit here: based on the amount of time I've had to spend fixing bugs over the past five months, 9.2 was our worst release ever. I don't like that trend, and I don't want to see it continued because we get laxer about accepting patches. IMO we are probably too lax already. Really? Hmm, that's not good. I seem to recall 8.4.x being pretty bad, and some of the recent bugs we fixed were actually 9.1.x problems that slipped through the cracks. FWIW I concur with Tom's assessment. The only way to fix increasing bug counts is through more-comprehensive regular testing. Currently we have regression/unit tests which cover maybe 30% of our code. Performance testing is largely ad-hoc. We don't require comprehensive acceptance testing for new patches. And we have 1m lines of code. Of course our bug count is increasing. I'm gonna see if I can do something about improving our test coverage. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/23/2013 09:51 AM, Josh Berkus wrote: The only way to fix increasing bug counts is through more-comprehensive regular testing. Currently we have regression/unit tests which cover maybe 30% of our code. Performance testing is largely ad-hoc. We don't require comprehensive acceptance testing for new patches. And we have 1m lines of code. Of course our bug count is increasing. And... slow down the release cycle or slow down the number of features that are accepted. Don't get me wrong I love everything we have and are adding every cycle but there does seem to be a definite weight difference between # of features added and time spent allowing those features to settle. Sincerely, JD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
* Robert Haas (robertmh...@gmail.com) wrote: For all of that, I'm not sure that people failing to seek consensus before coding is really so much of a problem as you seem to think. For my part, I don't think the lack of consensus-finding before submitting patches is, in itself, a problem. The problem, imv, is that everyone is expecting that once they've written a patch and put it on a commitfest that it's going to get committed- and it seems like committers are feeling under pressure that, because something's on the CF app, it needs to be committed in some form. There's a lot of good stuff out there, sure, and even more good *ideas*, but it's important to make sure we can provide a stable system with regular releases. As discussed, we really need to be ready to truely triage the remaining patch set, figure out who is going to work on what, and punt the rest til post-9.3. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 23, 2013 at 1:44 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: For all of that, I'm not sure that people failing to seek consensus before coding is really so much of a problem as you seem to think. For my part, I don't think the lack of consensus-finding before submitting patches is, in itself, a problem. The problem, imv, is that everyone is expecting that once they've written a patch and put it on a commitfest that it's going to get committed- and it seems like committers are feeling under pressure that, because something's on the CF app, it needs to be committed in some form. FWIW, I have NO delusions that something I propose or submit or put in a CF is necessarily going to get committed. For me it's not committed until I can see it in 'git log' and even then, I've seen stuff get reverted. I would hope that if a committer isn't comfortable with a patch they would explain why, and decline to commit. Then it's up to the submitter as to whether or not they want to make changes, try to explain why they are right and the committer is wrong, or withdraw the patch. There's a lot of good stuff out there, sure, and even more good *ideas*, but it's important to make sure we can provide a stable system with regular releases. As discussed, we really need to be ready to truely triage the remaining patch set, figure out who is going to work on what, and punt the rest til post-9.3. Thanks, Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 23, 2013 at 3:23 PM, Phil Sorber p...@omniti.com wrote: On Wed, Jan 23, 2013 at 1:44 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: For all of that, I'm not sure that people failing to seek consensus before coding is really so much of a problem as you seem to think. For my part, I don't think the lack of consensus-finding before submitting patches is, in itself, a problem. The problem, imv, is that everyone is expecting that once they've written a patch and put it on a commitfest that it's going to get committed- and it seems like committers are feeling under pressure that, because something's on the CF app, it needs to be committed in some form. FWIW, I have NO delusions that something I propose or submit or put in a CF is necessarily going to get committed. For me it's not committed until I can see it in 'git log' and even then, I've seen stuff get reverted. I would hope that if a committer isn't comfortable with a patch they would explain why, and decline to commit. Then it's up to the submitter as to whether or not they want to make changes, try to explain why they are right and the committer is wrong, or withdraw the patch. I think that's the right attitude, but it doesn't always work out that way. Reviewers and committers sometimes spend a lot of time writing a review and then get flamed for their honest opinion about the readiness of a patch. Of course, reviewers and committers can be jerks, too. As far as I know, we're all human, here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 23.01.2013 20:44, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: For all of that, I'm not sure that people failing to seek consensus before coding is really so much of a problem as you seem to think. For my part, I don't think the lack of consensus-finding before submitting patches is, in itself, a problem. I feel the same. Posting a patch before design and consensus may be a huge waste of time for the submitter himself, but it's not a problem for others. The problem, imv, is that everyone is expecting that once they've written a patch and put it on a commitfest that it's going to get committed- and it seems like committers are feeling under pressure that, because something's on the CF app, it needs to be committed in some form. I'm sure none of the committers have a problem rejecting a patch, when there's clear grounds for it. Rejecting is the easiest way to deal with a patch. However, at least for me, 50% of the patches in any given commitfest don't interest me at all. I don't object to them, per se, but I don't want to spend any time on them either. I can imagine the same to be true for all other committers too. If a patch doesn't catch the interest of any committer, it's going to just sit there in the commitfest app for a long time, even if it's been reviewed. As discussed, we really need to be ready to truely triage the remaining patch set, figure out who is going to work on what, and punt the rest til post-9.3. FWIW, here's how I feel about some the patches. It's not an exhaustive list. Event Triggers: Passing Information to User Functions (from 2012-11) I don't care about this whole feature, and haven't looked at the patch.. Probably not worth the complexity. But won't object if someone else wants to deal with it. Extension templates Same as above. Checksums (initdb-time) I don't want this feature, and I've said that on the thread. Identity projection (partitioning) Nothing's happened for over a month. Seems dead for that reason. Remove unused targets from plan Same here. Store additional information in GIN index Same here. Index based regexp search for pg_trgm I'd like to see this patch go in, but it still needs a fair amount of work. Probably needs to be pushed to next commitfest for that reason. allowing privileges on untrusted languages Seems like a bad idea to me, for the reasons Tom mentioned on that thread. Skip checkpoint on promoting from streaming replication Given that we still don't have an updated patch for this, it's probably getting too late for this. Would be nice to see the patch or an explanation of the new design Simon's been working. Patch to compute Max LSN of Data Pages (from 2012-11) Meh. Seems like a really special-purpose tool. Probably better to put this on pgfoundry. logical changeset generation v4 This is a boatload of infrastructure for supporting logical replication, yet we have no code actually implementing logical replication that would go with this. The premise of logical replication over trigger-based was that it'd be faster, yet we cannot asses that without a working implementation. I don't think this can be committed in this state. built-in/SQL Command to edit the server configuration file (postgresql.conf) I think this should be a pgfoundry project, not in core. At least for now. json generator function enhacements Json API and extraction functions To be honest, I don't understand why the json datatype had to be built-in to begin with. But it's too late to object to that now, and if the datatype is there, these functions probably should be, too. Or could these be put into a separate json-extras extension? I dunno. psql watch Meh. In practice, for the kind of ad-hoc monitoring this would be useful for, you might as well just use watch psql -c 'select ...' . Yes, that reconnects for each query, but so what. plpgsql_check_function I don't like this in its current form. A lot of code that mirrors pl_exec.c. I think we'll have to find a way to somehow re-use the existing code for this. Like, compile the function as usual, but don't stop on error. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Heikki, * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: FWIW, here's how I feel about some the patches. It's not an exhaustive list. Thanks for going through them and commenting on them. Event Triggers: Passing Information to User Functions (from 2012-11) I don't care about this whole feature, and haven't looked at the patch.. Probably not worth the complexity. But won't object if someone else wants to deal with it. I'd like to see it happen. Extension templates Same as above. This one isn't actually all that complex, though it does add a few additional catalogs for keeping track of things. For my part, I'd like to see this go in as I see it being another step closer to Packages that a certain other RDBMS has. Checksums (initdb-time) I don't want this feature, and I've said that on the thread. I see a lot of value in this. Identity projection (partitioning) Nothing's happened for over a month. Seems dead for that reason. I don't think this is dead.. Remove unused targets from plan Same here. Store additional information in GIN index Same here. Havn't been following these so not sure. Some of these are in a state of lack of progress for having not been reviewed. Index based regexp search for pg_trgm I'd like to see this patch go in, but it still needs a fair amount of work. Probably needs to be pushed to next commitfest for that reason. Agreed. allowing privileges on untrusted languages Seems like a bad idea to me, for the reasons Tom mentioned on that thread. On the fence about this one. I like the idea of reducing the need to be a superuser, but there are risks associated with this change also. Skip checkpoint on promoting from streaming replication Given that we still don't have an updated patch for this, it's probably getting too late for this. Would be nice to see the patch or an explanation of the new design Simon's been working. Patch to compute Max LSN of Data Pages (from 2012-11) Meh. Seems like a really special-purpose tool. Probably better to put this on pgfoundry. Agreed on these two. logical changeset generation v4 This is a boatload of infrastructure for supporting logical replication, yet we have no code actually implementing logical replication that would go with this. The premise of logical replication over trigger-based was that it'd be faster, yet we cannot asses that without a working implementation. I don't think this can be committed in this state. Andres had a working implementation of logical replication, with code to back it up and performance numbers showing how much faster it is, at PGCon last year, as I recall... Admittedly, it probably needs changing and updating due to the changes which the patches have been going through, but I don't think the claim that we don't know it's faster is fair. built-in/SQL Command to edit the server configuration file (postgresql.conf) I think this should be a pgfoundry project, not in core. At least for now. I don't think it would ever get deployed or used then.. json generator function enhacements Json API and extraction functions To be honest, I don't understand why the json datatype had to be built-in to begin with. But it's too late to object to that now, and if the datatype is there, these functions probably should be, too. Or could these be put into a separate json-extras extension? I dunno. There were good reasons to add json as a data type, I thought, though I don't have them offhand. psql watch Meh. In practice, for the kind of ad-hoc monitoring this would be useful for, you might as well just use watch psql -c 'select ...' . Yes, that reconnects for each query, but so what. I do that pretty often. A better approach, imv, would be making psql a bit more of a 'real' shell, with loops, conditionals, better variable handling, etc. plpgsql_check_function I don't like this in its current form. A lot of code that mirrors pl_exec.c. I think we'll have to find a way to somehow re-use the existing code for this. Like, compile the function as usual, but don't stop on error. Havn't looked at this yet, but your concerns make sense to me. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Hello I do that pretty often. A better approach, imv, would be making psql a bit more of a 'real' shell, with loops, conditionals, better variable handling, etc. after a few years prototyping on this area I am not sure so this is good idea. Maybe better to start some new console from scratch. plpgsql_check_function I don't like this in its current form. A lot of code that mirrors pl_exec.c. I think we'll have to find a way to somehow re-use the existing code for this. Like, compile the function as usual, but don't stop on error. Havn't looked at this yet, but your concerns make sense to me. I invite any moving in this subject - again I wrote lot of variants and current is a most maintainable (my opinion) - there is redundant structure (not code) - and simply code. After merging the code lost readability :(. But I would to continue in work - I am sure so it has a sense and people and some large companies use it with success, so it should be in core - in any form. Regards Pavel Thanks! Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Tom Lane t...@sss.pgh.pa.us writes: In this connection I refer you to Sturgeon's Law(*): 90% of everything is crud. Applied to our problem, it says that 90% of all patch ideas are bad. Therefore, we should be expecting to reject a large fraction of submitted patches. It distresses me that people seem to think the default result for a submitted patch is that it gets committed. That's backwards. +1 I still think it takes loads of stupid ideas and discussion before reaching a point where a patch can be brewed and proposed. Once you reach a certain point though, typically when we begin talking about careful implementation details, then my feeling is that a patch is necessary for the discussion to remain a productive use of everyone's time. So the goal in your proposed terms would be to only spend time cooking a patch for about 10% of your ideas, and be prepared to rewrite it from about scratch a least a couple of times (for a simple enough patch). That matches my experience. For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. It seems though that less and less of that is actually happening, so I wonder whether the commitfest process is encouraging inefficiency by making people think they should start a discussion with a mostly-complete patch. Or maybe CF review is just crowding out any serious discussion of rough ideas. There was some discussion at the last dev meeting of creating a design review process within commitfests, but nothing got done about it ... I share that feeling that while commit fest is a giant step forward as far as allowing patches to make progress and hit the next release without delaying said release, it might be encouraging people to cook patches too early: there's no entry for crazy ideas or design. I guess in getting more formal it's harder for newcomers to just throw (not so) random ideas on list, as it used to be the only way to begin a contribution to PostgreSQL. My understanding is that we already have too many lists, but maybe we could have another one to just speak about those 10% ideas that turn into patches or commit fest entries (pgsql-workers) and keep hackers for crazy idea, design, community processes, etc. I'm not sold on it myself, but it could maybe help in encouraging design ideas again? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Tue, Jan 22, 2013 at 11:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: In this connection I refer you to Sturgeon's Law(*): 90% of everything is crud. Applied to our problem, it says that 90% of all patch ideas are bad. That reminds of my conversation with our masters thesis guide who is a accomplished professor and an IEEE fellow with loads of research work to his credit. I was disappointed that I'm not doing a *significant* research work, to which he replied that 90% work is just like that. But every once in a while someone gets inspired by that work and either take that further or creates a landmark work out of it. I was satisfied with his answer. So the 90% crap you mentioned is probably not all that crap and we need that so that newer ideas are conceived while discussing some of them. Therefore, we should be expecting to reject a large fraction of submitted patches. It distresses me that people seem to think the default result for a submitted patch is that it gets committed. That's backwards. Yeah, probably that expectation needs to be set. What will help more is to somehow guide the patch submitter to a related area and encourage to handle that issue or the same issue in a little different but more acceptable way. I think we already do a lot of that but more can be done. Its OK to push back patch submitter and they will be fine as long as their second or third attempt succeeds. If Tom or equivalent feels that the idea is no starter, just reject it at the first go and before the submitter has done multiple iterations to save everyone's time and reduce frustration. For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. It seems though that less and less of that is actually happening, Agree. Many a times its important though that senior folks give their opinion one way or the other, early in the review cycle. That does not happen always. I remember during HOT development I used to wait for days and weeks for Tom to take note of my emails and respond, even if he is ripping apart my ideas. But the fact that HE has looked into the work was a great satisfaction. Once someone is taking interest then there is a chance of convincing that person. Thankfully we have a lot more Tom-alike (not Tom yet :-)) now than 5 years back and thats why we are seeing a lot more intrusive work. There was some discussion at the last dev meeting of creating a design review process within commitfests, but nothing got done about it ... Yeah, agree. May be we need to put that in the process itself. So no patch be submitted unless the idea has been discussed and agreed upon to some extent. Of course, few things you will only know once you start writing the code. But at least the major points must have been accepted by at least one major developer or a committer. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Robert Haas robertmh...@gmail.com writes: I'm not going to pretend that all review comments are constructive, but I also think that to some degree the difference between these two things depends on your perspective. I recall, in particular, the email that prompted the famous in short: -1 from me regards tom lane T-shirt, which I believe to be this one: http://www.postgresql.org/message-id/28927.1236820...@sss.pgh.pa.us That's not a positive review, but when it comes down to it, it's a pretty factual email. IMHO, anyway, and YMMV. This email is not saying anything about the patch's content, not offering any judgement (you did it the wrong way), it is all about offering some pieces of advice on the complexity of SET and RESET. My own experience is different from yours, I guess. I actually like it when I post a patch, or suggest a concept, and Tom fires back with a laundry list of reasons it won't work. It often induces me to step back and approach the same problem from a different and better angle, and the result is often better for it. What I don't like is when I What I was talking about is judgements on a patch the commenter didn't read. Being offered advices by people in the know is awesome, and if that happens early in the patch life (design), it's even better. (or anyone) posts a patch and somebody says something that boils down to no one wants that. *That* ticks me off. Because you know what? At a minimum, *I* want that. If I didn't, I wouldn't have written a patch. And usually, the customers I support want that, too. Now, somebody else may not want it, and that is fine. But IMHO, posting a patch should be considered prima facie evidence of non-zero demand for the associated feature. That part reminds me too much of the Inline Extension patch series for me to comment any further. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Yeah, agree. May be we need to put that in the process itself. So no patch be submitted unless the idea has been discussed and agreed upon to some extent. Of course, few things you will only know once you start writing the code. But at least the major points must have been accepted by at least one major developer or a committer. This is little bit problem, because some discussion starts too late - in commitfest. In my experience +/- 30% of commited patches was zero response after proposal. Usually people watching some interesting subset of topics and doesn't comment uninteresting proposals. There are no timeout for rejection or accepting proposals. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/22/2013 01:15 AM, Tom Lane wrote: Yeah, and a lot more fairly-new developers who don't understand all the connections in the existing system. I think it's just in the nature of the beast we're dealing with to be much more conservative about what we accept than it might be for some other projects. This helps to contribute to our deserved reputation for stability and reliability. For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. Getting this idea across can be surprisingly difficult, especially in cultures not used to dealing with open source. Large commercial ventures with a history in proprietary software tend to want to come to the table with a shiny new feature patch, all gift wrapped and sparkly. And they are not alone - even new individual contributors want to work this way. I wouldn't blame the commitfest procedure - it's just something we need to keep hammering on. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 22/01/13 22:35, Dimitri Fontaine wrote: Tom Lane t...@sss.pgh.pa.us writes: In this connection I refer you to Sturgeon's Law(*): 90% of everything is crud. Applied to our problem, it says that 90% of all patch ideas are bad. Therefore, we should be expecting to reject a large fraction of submitted patches. It distresses me that people seem to think the default result for a submitted patch is that it gets committed. That's backwards. +1 I still think it takes loads of stupid ideas and discussion before reaching a point where a patch can be brewed and proposed. Once you reach a certain point though, typically when we begin talking about careful implementation details, then my feeling is that a patch is necessary for the discussion to remain a productive use of everyone's time. So the goal in your proposed terms would be to only spend time cooking a patch for about 10% of your ideas, and be prepared to rewrite it from about scratch a least a couple of times (for a simple enough patch). That matches my experience. For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. It seems though that less and less of that is actually happening, so I wonder whether the commitfest process is encouraging inefficiency by making people think they should start a discussion with a mostly-complete patch. Or maybe CF review is just crowding out any serious discussion of rough ideas. There was some discussion at the last dev meeting of creating a design review process within commitfests, but nothing got done about it ... I share that feeling that while commit fest is a giant step forward as far as allowing patches to make progress and hit the next release without delaying said release, it might be encouraging people to cook patches too early: there's no entry for crazy ideas or design. I guess in getting more formal it's harder for newcomers to just throw (not so) random ideas on list, as it used to be the only way to begin a contribution to PostgreSQL. My understanding is that we already have too many lists, but maybe we could have another one to just speak about those 10% ideas that turn into patches or commit fest entries (pgsql-workers) and keep hackers for crazy idea, design, community processes, etc. I'm not sold on it myself, but it could maybe help in encouraging design ideas again? Regards, Maybe we should have a list named 'crazy talk' for such ideas. Possibly encourage Tom and other pg heavy weights to write brief notes about their intentions. So people can see that even the best pg developers can have half baked ideas, to encourage newcomers and less experienced developers to let people know well in advance what their intentions are. Most people don't realize tat to be really stupid, one needs to be both highly intelligent and creative! More to the point, perhaps, is that the most effective developers often have many silly ideas that are: well intentioned but not practicable, initially implemented poorly,or develop something that seem good until grim reality hits. However, when they strike gold, they improve pg: in considerable ways, make trivial changes that are disproportionately useful,or put a lot of effort into making what superficially looks like a simple idea to actually work without bad side effects. I am sure with even my 40+ years of development experience in other areas, that if I attempted a 'trivial' change to pg, that I would most likely cause unwanted side effects without realizing it - assuming I got it working at all! Tom and others would then quite rightly reject my patch, or give me constructive criticism (that I might take as negative feedback if I was depressed!). Howevever, if I first proposed my idea on a 'crazy talk' mailing list, then possibly I would either get suggestions as to how to proceed to avoid such side effects, or be told that what I proposed was not worth the effort. In the latter case, it would be up to me to research reasons for proceeding, if I felt strongly that I was right. Cheers, Gavin
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
IMHO that's the single most important task of a review. Really? I'd say the most important task for a review is does the patch do what it says it does?. That is, if the patch is supposed to implement feature X, does it actually? If it's a performance patch, does performance actually improve? If the patch doesn't implement what it's supposed to, who cares what the code looks like? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Josh Berkus j...@agliodbs.com writes: IMHO that's the single most important task of a review. Really? I'd say the most important task for a review is does the patch do what it says it does?. That is, if the patch is supposed to implement feature X, does it actually? If it's a performance patch, does performance actually improve? If the patch doesn't implement what it's supposed to, who cares what the code looks like? But even before that, you have to ask whether what it's supposed to do is something we want. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
But even before that, you have to ask whether what it's supposed to do is something we want. The reviewer can't usually answer that though. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 2:09 PM, Josh Berkus j...@agliodbs.com wrote: But even before that, you have to ask whether what it's supposed to do is something we want. The reviewer can't usually answer that though. They can answer whether THEY want it, though. And Tom, Andrew, and I all just got through arguing that that is one of the most, if not the most, important parts of a review. Seriously. Opinions are good. Lack of opinions leads to ivory tower syndrome, which of course we've got, but I think most of us are sufficiently self-aware to at least know that it isn't a good thing. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Robert Haas robertmh...@gmail.com writes: On Sun, Jan 20, 2013 at 7:07 PM, Jeff Janes jeff.ja...@gmail.com wrote: As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. Even if you as a reviewer don't know the answer to those questions, clearly delineating the key issues may enable others to comment without having to read and understand the whole patch, which can expedite things considerably. We wouldn't have to convince anyone about a judgement being hasty if only those making judgement took some time and read the patch before making those judgements on list (even if doing so is not changing the judgement, it changes the hasty part already). Offering advice and making judgments are two very different things. And I'm lucky enough to have received plenty of very good advice from senior developers here that seldom did read my patches before offering them. I'm yet to see good use of anyone's time once a hasty judgement has been made about a patch. And I think there's no value judgement in calling a judgement hasty here: it's a decision you took and published *about a patch* before reading the patch. That's about it. If you don't have the time to read the patch, offer advice. Anything more, and you can set a watch to count the cumulative time we are all loosing on trying to convince each other about something else. Ok, now you will tell me there's no difference and it's just playing with words. It's not. Judgement is about how the patch do things, Advice is about how to do things in general. Ok, here's an hypothetical example, using some words we tend to never use here because we are all very polite and concerned about each other happiness when talking about carefully crafted code: Advice:You don't do things that way, this way is the only one we will ever accept, because we've been sweating blood over the years to get in a position where it now works. Hint: it's not talking about the patch content, but what is supposedly a problem with the patch. It's easy to answer I'm so happy I didn't actually do it that way. Judgement: You need to think about the problem you want to solve before sending a patch, because there's a hole in it too big for me to be able to count how many bugs are going to to dance in there. It's not a patch, it's a quagmire. BTW, I didn't read it, it stinks too much. Hint: imagine it was your patch and now you have to try and convince any other commiter to have a look at it. Now, I've been reviewing patches in all commit fests but one, and began reviewing well before the idea of a commit fest even existed. My idea of a good review is being able to come up with one of only 3 possible summaries: - it looks good, please consider a commit, any remaining work is going to have to be done by a commiter anyway - it's not ready, please fix this and that, think about this use case, review docs, whatever - it's ahead of me, this patch now needs a commiter (or just someone else) to read it then weigth in. at least it compiles, follow the usual code and comment style, the documentation is complete and free or errors, and I tried it and it does what it says (perfs, features, etc etc, bonus points for creative usage and trying to make it crash). Of course, reviewers don't get much feedback in general, so I can only hope that guideline is good enough for most commiters. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 8:18 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). FWIW, I reviewed at least one at the point you did this survey, and I did update the CF page, but I didn't put my name into that box marked reviewers because it is an extra step (Edit Patch) that isn't immediatly obvious. Isn't there a way to automatically populate that based on people linking in their reviews? I guess it might be difficult when a CF manager comes along to add them and they aren't their reviews. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 6:23 PM, Phil Sorber p...@omniti.com wrote: On Wed, Jan 16, 2013 at 8:18 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). FWIW, I reviewed at least one at the point you did this survey, and I did update the CF page, but I didn't put my name into that box marked reviewers because it is an extra step (Edit Patch) that isn't immediatly obvious. Isn't there a way to automatically populate that based on people linking in their reviews? Sadly, the CF application doesn't have access to the user name - real name mappings. And while it's only mildly annoying that the updates are displayed under user name rather than realname, showing username for the author and reviewers fields would be really annoying, especially because not all patch authors necessarily even have accounts. I guess it might be difficult when a CF manager comes along to add them and they aren't their reviews. That's also an issue. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 5:48 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Advice:You don't do things that way, this way is the only one we will ever accept, because we've been sweating blood over the years to get in a position where it now works. Hint: it's not talking about the patch content, but what is supposedly a problem with the patch. It's easy to answer I'm so happy I didn't actually do it that way. Judgement: You need to think about the problem you want to solve before sending a patch, because there's a hole in it too big for me to be able to count how many bugs are going to to dance in there. It's not a patch, it's a quagmire. BTW, I didn't read it, it stinks too much. Hint: imagine it was your patch and now you have to try and convince any other commiter to have a look at it. I'm not going to pretend that all review comments are constructive, but I also think that to some degree the difference between these two things depends on your perspective. I recall, in particular, the email that prompted the famous in short: -1 from me regards tom lane T-shirt, which I believe to be this one: http://www.postgresql.org/message-id/28927.1236820...@sss.pgh.pa.us That's not a positive review, but when it comes down to it, it's a pretty factual email. IMHO, anyway, and YMMV. My own experience is different from yours, I guess. I actually like it when I post a patch, or suggest a concept, and Tom fires back with a laundry list of reasons it won't work. It often induces me to step back and approach the same problem from a different and better angle, and the result is often better for it. What I don't like is when I (or anyone) posts a patch and somebody says something that boils down to no one wants that. *That* ticks me off. Because you know what? At a minimum, *I* want that. If I didn't, I wouldn't have written a patch. And usually, the customers I support want that, too. Now, somebody else may not want it, and that is fine. But IMHO, posting a patch should be considered prima facie evidence of non-zero demand for the associated feature. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Robert, http://www.postgresql.org/message-id/28927.1236820...@sss.pgh.pa.us That's not a positive review, but when it comes down to it, it's a pretty factual email. IMHO, anyway, and YMMV. Really? I've always thought that was a pretty constructive review. It certainly gave me the laundry list of things I'd have to fix to ever get that change in, and how what looked like a simple patch is actually fairly complicated. My own experience is different from yours, I guess. I actually like it when I post a patch, or suggest a concept, and Tom fires back with a laundry list of reasons it won't work. This can be a problem with new submitters, though. If you're not used to the current community dialog, that email can be taken as your idea is stupid because rather than what it actually means, which is fix these issues and resubmit, please. That's often not clearly communicated, and is important with new submitters. It often induces me to step back and approach the same problem from a different and better angle, and the result is often better for it. What I don't like is when I (or anyone) posts a patch and somebody says something that boils down to no one wants that. *That* ticks me off. Because you know what? At a minimum, *I* want that. If I didn't, I wouldn't have written a patch. And usually, the customers I support want that, too. Now, somebody else may not want it, and that is fine. But IMHO, posting a patch should be considered prima facie evidence of non-zero demand for the associated feature. On the other hand, saying this patch has potential side effects / peformance penalty / makes admin more complex / has a lot of code to maintain. How common is the use-case you're talking about? is legit. Features which impose a cost on people who don't use them need to be justified as useful enough to be worth the cost. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 8:23 PM, Robert Haas robertmh...@gmail.com wrote: What I don't like is when I (or anyone) posts a patch and somebody says something that boils down to no one wants that. *That* ticks me off. Because you know what? At a minimum, *I* want that. If I didn't, I wouldn't have written a patch. And usually, the customers I support want that, too. Now, somebody else may not want it, and that is fine. But IMHO, posting a patch should be considered prima facie evidence of non-zero demand for the associated feature. +1 I'd much rather hear what you are trying to accomplish is much better done this other way. rather than, why would you want to do this? or as you said no one wants that. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 8:47 PM, Josh Berkus j...@agliodbs.com wrote: My own experience is different from yours, I guess. I actually like it when I post a patch, or suggest a concept, and Tom fires back with a laundry list of reasons it won't work. This can be a problem with new submitters, though. If you're not used to the current community dialog, that email can be taken as your idea is stupid because rather than what it actually means, which is fix these issues and resubmit, please. That's often not clearly communicated, and is important with new submitters. +1 to this as well. I definitely felt that way when submitting my first patch. Robert might even recall a convo at a conference that he and I had about it. If not for that I might have given up long ago. Now I can say I am used to it though, and I appreciate the honest and constructive criticism I receive because over time I have seen that Tom and others are even critical of themselves and it's really for the betterment of the final product. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Mon, Jan 21, 2013 at 8:47 PM, Josh Berkus j...@agliodbs.com wrote: http://www.postgresql.org/message-id/28927.1236820...@sss.pgh.pa.us That's not a positive review, but when it comes down to it, it's a pretty factual email. IMHO, anyway, and YMMV. Really? I've always thought that was a pretty constructive review. It certainly gave me the laundry list of things I'd have to fix to ever get that change in, and how what looked like a simple patch is actually fairly complicated. I agree that it was constructive, but it wasn't exactly endorsing the concept you put forward, so it was not a positive, aka favorable, review. Also, you did put it on a T-shirt. :-) My own experience is different from yours, I guess. I actually like it when I post a patch, or suggest a concept, and Tom fires back with a laundry list of reasons it won't work. This can be a problem with new submitters, though. If you're not used to the current community dialog, that email can be taken as your idea is stupid because rather than what it actually means, which is fix these issues and resubmit, please. That's often not clearly communicated, and is important with new submitters. Yep. Even long-time participants in the process sometimes get demoralized. It's always fun to be the smartest guy in the room, and one rarely gets that luxury with any regularity on pgsql-hackers. It often induces me to step back and approach the same problem from a different and better angle, and the result is often better for it. What I don't like is when I (or anyone) posts a patch and somebody says something that boils down to no one wants that. *That* ticks me off. Because you know what? At a minimum, *I* want that. If I didn't, I wouldn't have written a patch. And usually, the customers I support want that, too. Now, somebody else may not want it, and that is fine. But IMHO, posting a patch should be considered prima facie evidence of non-zero demand for the associated feature. On the other hand, saying this patch has potential side effects / peformance penalty / makes admin more complex / has a lot of code to maintain. How common is the use-case you're talking about? is legit. Features which impose a cost on people who don't use them need to be justified as useful enough to be worth the cost. Totally agreed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Tue, Jan 22, 2013 at 7:57 AM, Phil Sorber p...@omniti.com wrote: On Mon, Jan 21, 2013 at 8:47 PM, Josh Berkus j...@agliodbs.com wrote: My own experience is different from yours, I guess. I actually like it when I post a patch, or suggest a concept, and Tom fires back with a laundry list of reasons it won't work. This can be a problem with new submitters, though. If you're not used to the current community dialog, that email can be taken as your idea is stupid because rather than what it actually means, which is fix these issues and resubmit, please. That's often not clearly communicated, and is important with new submitters. +1 to this as well. I definitely felt that way when submitting my first patch. Robert might even recall a convo at a conference that he and I had about it. If not for that I might have given up long ago. Now I can say I am used to it though, and I appreciate the honest and constructive criticism I receive because over time I have seen that Tom and others are even critical of themselves and it's really for the betterment of the final product. For me our reluctance for any kind of change is a major demoralizing factor. For example, the recent discussion on Jeff Davis's patch on PD_ALL_VISIBLE flag. Our committers are quite candid in accepting that they may not have put in so much thought or did any real testing while adding the original code, but when it comes to making a change we show reluctance, ask for several test results and the patch author really needs to show extra worth for the change. That looks a bit unfair because the original code may not have received the same attention. Though I completely understand that the committers usually have a better sense and gut feel. If Heikki had not put that page level flag in the first place and if someone would have suggested adding that flag today, my sense is that we would have pushed him/her back and asked for many explanations like why waste a bit in the page or the performance increase is just 10% and that too in the worst case, so not worth it. Another example is the SKIP_PAGES_THRESHOLD in vacuumlazy.c. I'm quite convinced that the current default of 32 is quite arbitrary and the entire logic to not skip heap pages unless there are 32 in sequence is hard to justify, but if that needs to be changed, we would need hours of testing to show its worth. (BTW, I know both these changes are probably committed by Heikki. But thats just a coincidence :-) I've great respect for Heikki's talent and he is often right in his judgement) Having said that, I quite understand the reasoning behind the reluctance for change - we are a small community and can't afford to spend cycles spending on unnecessary regressions. But is that changing in the recent years ? I mean, aren't there a lot more developers and a lot more companies using/testing the new features/releases and reporting issues ? I see a marked increase in the number of bugs reported (haven't really counted but just observation and it could be wrong). Not sure if its a result of increased testing/adaption or a slight drop in the quality or just because of the sheer increase in the number of features/changes we are doing. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Pavan Deolasee pavan.deola...@gmail.com writes: For me our reluctance for any kind of change is a major demoralizing factor. I hardly think we're reluctant for any kind of change --- the rate of commits belies that. What we want is a convincing case that a proposed change is an improvement over what we have. (You're entirely right that there are plenty of dubious places in the existing code, but most of them replaced nothing at all, and were thus improvements. To improve further requires, well, clear improvement.) In the case of PD_ALL_VISIBLE, I believe there is very serious reason to think removing it would be a performance loss at least some of the time. We need proof it's an overall improvement, not lack of proof it isn't. Having said that, I quite understand the reasoning behind the reluctance for change - we are a small community and can't afford to spend cycles spending on unnecessary regressions. But is that changing in the recent years ? I mean, aren't there a lot more developers and a lot more companies using/testing the new features/releases and reporting issues ? Yeah, and a lot more fairly-new developers who don't understand all the connections in the existing system. Let me just push back a bit here: based on the amount of time I've had to spend fixing bugs over the past five months, 9.2 was our worst release ever. I don't like that trend, and I don't want to see it continued because we get laxer about accepting patches. IMO we are probably too lax already. In this connection I refer you to Sturgeon's Law(*): 90% of everything is crud. Applied to our problem, it says that 90% of all patch ideas are bad. Therefore, we should be expecting to reject a large fraction of submitted patches. It distresses me that people seem to think the default result for a submitted patch is that it gets committed. That's backwards. For a very long time we've tried to encourage people to submit rough ideas to pgsql-hackers for discussion *before* they start coding. The point of that is to weed out, or fix if possible, (some of) the bad ideas before a lot of effort has gotten expended on them. It seems though that less and less of that is actually happening, so I wonder whether the commitfest process is encouraging inefficiency by making people think they should start a discussion with a mostly-complete patch. Or maybe CF review is just crowding out any serious discussion of rough ideas. There was some discussion at the last dev meeting of creating a design review process within commitfests, but nothing got done about it ... regards, tom lane (*) The wikipedia entry is good enough. Whether Ted had the percentage spot-on is not the key issue here. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Jan 22, 2013 1:31 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 21, 2013 at 6:23 PM, Phil Sorber p...@omniti.com wrote: On Wed, Jan 16, 2013 at 8:18 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). FWIW, I reviewed at least one at the point you did this survey, and I did update the CF page, but I didn't put my name into that box marked reviewers because it is an extra step (Edit Patch) that isn't immediatly obvious. Isn't there a way to automatically populate that based on people linking in their reviews? Sadly, the CF application doesn't have access to the user name - real name mappings. And while it's only Sure it does. It only has it for users that have logged in at least once. But that wouldn't be a problem for this scenario as they would have already logged in to post said link. /Magnus
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Sat, Jan 19, 2013 at 5:21 PM, Jeff Janes jeff.ja...@gmail.com wrote: Sometime this type of high-level summary review does happen, at the senior person's whim, but is not a formal part of the commit fest process. What I don't know is how much work it takes for one of those senior people to make one of those summary judgments, compared to how much it takes for them to just do an entire review from scratch. IME, making such summary judgements can often be done in a few minutes, but convincing that the patch submitter that you haven't created the objection purely as an obstacle to progress is the work of a lifetime. We could perhaps do better at avoiding perverse incentives, there. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 20 January 2013 18:42, Robert Haas robertmh...@gmail.com wrote: On Sat, Jan 19, 2013 at 5:21 PM, Jeff Janes jeff.ja...@gmail.com wrote: Sometime this type of high-level summary review does happen, at the senior person's whim, but is not a formal part of the commit fest process. What I don't know is how much work it takes for one of those senior people to make one of those summary judgments, compared to how much it takes for them to just do an entire review from scratch. IME, making such summary judgements can often be done in a few minutes, but convincing that the patch submitter that you haven't created the objection purely as an obstacle to progress is the work of a lifetime. We could perhaps do better at avoiding perverse incentives, there. (Without meaning to paraphrase you in any negative way...) Judgements made in a few minutes are very frequently wrong, and it takes a lot of time to convince the person making snap decisions that they should revise their thinking in light of new or correct information. I feel we are very prone to making judgements on little information. This is especially true with regard to voting, with people zooming in from the side without having even read a patch to vote one way or the other, voting for the person not the idea. It can be a big problem telling the difference between a patch submitter that really is in possession of information that should be heeded and someone so blinded by their patch/idea that they mistakenly push forwards. At times, I have been both and I've also witnessed both as committer. There is a clear and understandable conservatism in being a reviewer/committer that people that haven't done it don't understand. I definitely didn't until I was a committer and I remember clearly me pushing for HS to go into 8.4 when it was a long way from being ready. I think it would be useful to expand the pool of committers and perhaps that can be done via some intermediate stage, though I do worry that the sense of responsibility might not be as strong in the intermediate rank ($NAME). I don't think we should be encouraging people to make fast decisions. Senior or not. (Though we must make decisions and have some coming soon). This is high in my mind right now since I've been looking at skip checkpoint patch for months, seeing how I feel about it. Nervous, basically. From that I think that some areas of the code are more critical than others and harder to fix in production if they go wrong. We need to be taking more care in critical areas and it would be useful to be able to mark a patch for its level of risk, rather than just is it ready. That way we can gauge the risk/benefit. Examples of high risk would be checksums, examples of low risk would be logical replication patches. Anyway, some thoughts to discuss in May. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
2013/1/20 Simon Riggs si...@2ndquadrant.com: On 20 January 2013 18:42, Robert Haas robertmh...@gmail.com wrote: On Sat, Jan 19, 2013 at 5:21 PM, Jeff Janes jeff.ja...@gmail.com wrote: Sometime this type of high-level summary review does happen, at the senior person's whim, but is not a formal part of the commit fest process. What I don't know is how much work it takes for one of those senior people to make one of those summary judgments, compared to how much it takes for them to just do an entire review from scratch. IME, making such summary judgements can often be done in a few minutes, but convincing that the patch submitter that you haven't created the objection purely as an obstacle to progress is the work of a lifetime. We could perhaps do better at avoiding perverse incentives, there. (Without meaning to paraphrase you in any negative way...) Judgements made in a few minutes are very frequently wrong, and it takes a lot of time to convince the person making snap decisions that they should revise their thinking in light of new or correct information. I feel we are very prone to making judgements on little information. This is especially true with regard to voting, with people zooming in from the side without having even read a patch to vote one way or the other, voting for the person not the idea. It can be a big problem telling the difference between a patch submitter that really is in possession of information that should be heeded and someone so blinded by their patch/idea that they mistakenly push forwards. At times, I have been both and I've also witnessed both as committer. There is a clear and understandable conservatism in being a reviewer/committer that people that haven't done it don't understand. I definitely didn't until I was a committer and I remember clearly me pushing for HS to go into 8.4 when it was a long way from being ready. I think it would be useful to expand the pool of committers and perhaps that can be done via some intermediate stage, though I do worry that the sense of responsibility might not be as strong in the intermediate rank ($NAME). I don't think we should be encouraging people to make fast decisions. Senior or not. (Though we must make decisions and have some coming soon). This is high in my mind right now since I've been looking at skip checkpoint patch for months, seeing how I feel about it. Nervous, basically. From that I think that some areas of the code are more critical than others and harder to fix in production if they go wrong. We need to be taking more care in critical areas and it would be useful to be able to mark a patch for its level of risk, rather than just is it ready. That way we can gauge the risk/benefit. Examples of high risk would be checksums, examples of low risk would be logical replication patches. Anyway, some thoughts to discuss in May. +1 Pavel -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Sun, Jan 20, 2013 at 2:39 PM, Simon Riggs si...@2ndquadrant.com wrote: (Without meaning to paraphrase you in any negative way...) Judgements made in a few minutes are very frequently wrong, and it takes a lot of time to convince the person making snap decisions that they should revise their thinking in light of new or correct information. I feel we are very prone to making judgements on little information. This is especially true with regard to voting, with people zooming in from the side without having even read a patch to vote one way or the other, voting for the person not the idea. It can be a big problem telling the difference between a patch submitter that really is in possession of information that should be heeded and someone so blinded by their patch/idea that they mistakenly push forwards. At times, I have been both and I've also witnessed both as committer. There is a clear and understandable conservatism in being a reviewer/committer that people that haven't done it don't understand. I definitely didn't until I was a committer and I remember clearly me pushing for HS to go into 8.4 when it was a long way from being ready. I think it would be useful to expand the pool of committers and perhaps that can be done via some intermediate stage, though I do worry that the sense of responsibility might not be as strong in the intermediate rank ($NAME). I don't think we should be encouraging people to make fast decisions. Senior or not. (Though we must make decisions and have some coming soon). This is high in my mind right now since I've been looking at skip checkpoint patch for months, seeing how I feel about it. Nervous, basically. From that I think that some areas of the code are more critical than others and harder to fix in production if they go wrong. We need to be taking more care in critical areas and it would be useful to be able to mark a patch for its level of risk, rather than just is it ready. That way we can gauge the risk/benefit. Examples of high risk would be checksums, examples of low risk would be logical replication patches. Anyway, some thoughts to discuss in May. I agree with some but not all of your observations here, but they're along a different line than the point I was trying to make. I would distinguish between value judgements (i.e. this patch is bad because we don't want that) and architectural judgments (i.e. this patch is bad because the logic you've added needs to go in the planner, not the parser). I often disagree with the value judgements that others make, regardless of how much or little time they've spent on them, because, well, at the end of the day, it's an opinion. Our experiences inform our opinions, and since we all have different experiences, we won't always have the same opinions about everything. Architectural judgements are something else again. If Tom says that a particular piece of logic needs to live in the planner rather than the parser, the chances that he is right are upwards of 90%, in my experience. There are more complicated or controversial cases, of course, but I find it's often not difficult to answer the question assuming we were going to do this, how ought we to do it?. People don't always like hearing those answers, and they're not *always* easy, but there are many cases where I think they are. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Sunday, January 20, 2013, Simon Riggs wrote: On 20 January 2013 18:42, Robert Haas robertmh...@gmail.comjavascript:; wrote: On Sat, Jan 19, 2013 at 5:21 PM, Jeff Janes jeff.ja...@gmail.comjavascript:; wrote: Sometime this type of high-level summary review does happen, at the senior person's whim, but is not a formal part of the commit fest process. What I don't know is how much work it takes for one of those senior people to make one of those summary judgments, compared to how much it takes for them to just do an entire review from scratch. IME, making such summary judgements can often be done in a few minutes, but convincing that the patch submitter that you haven't created the objection purely as an obstacle to progress is the work of a lifetime. We could perhaps do better at avoiding perverse incentives, there. As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. That would provide me with some direction, rather than just having me pondering whether a certain variable name ought to have one more or one fewer underscores in it. On the other hand if a summary judgement is that the patch is fundamentally sound but needs some line-by-line code-lawyering, or some performance testing, or documentation/usability testing, or needs to ponder the implications to part XYZ of the code base (which neither I nor the other have even heard of before), that would also be good to know. My desire is not for you to tell me what the outcome of the review should be, but rather what the focus of it should be. As it is now, when I see a patch that needs review but has no comments, I don't know if that is because no senior developer has read it, or because they have read it but didn't think it needed comment. The wiki page does list points to consider when reviewing a submission, but not all points are equally relevant to all patches--and it is not always obvious to me which are more relevant. (Without meaning to paraphrase you in any negative way...) Judgements made in a few minutes are very frequently wrong, and it takes a lot of time to convince the person making snap decisions that they should revise their thinking in light of new or correct information. This is true, but I'd like to know up front that convincing them to revise their thinking is the main thing I need to do during the review process. Restating and clarifying the submitter's arguments, perhaps with the addition of some experimental evidence, is one part where I think I can contribute, provided I know that that is what I need to be doing. Having them reserve their opinion until after it is marked ready for commiter doesn't make it easier to change their mind, I don't think. As a reviewer I can't help address their specific concerns, if I don't know what those were. Cheers, Jeff
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Sun, Jan 20, 2013 at 7:07 PM, Jeff Janes jeff.ja...@gmail.com wrote: As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. That's a hard question. I'm not sure there's a categorical answer. I take positions both in support of and in opposition to the positions of other reviewers, and I suppose I don't see why you shouldn't do the same. It is of course worth bearing in mind that unless at least one committer is willing to support a given approach, it's not going anywhere ... but on the other hand, committers are just people, and do sometimes change their minds, too. So, really, I think you should try to persuade the person that you think is wrong, whoever that is. What I like least is when someone writes a review and says some people think this is a good idea and some people think it's a bad idea. When I go back to look at the discussion and make a decision about whether to commit something, I want to have a clear idea whether most people liked it or most people didn't like it, and why. That helps to inform my thinking. When it's just me and the submitter, it's hard to tell whether anyone cares at all, one way or the other. As it is now, when I see a patch that needs review but has no comments, I don't know if that is because no senior developer has read it, or because they have read it but didn't think it needed comment. The wiki page does list points to consider when reviewing a submission, but not all points are equally relevant to all patches--and it is not always obvious to me which are more relevant. The main things I personally like to see reviewers do, in descending order of importance, are: 1. express about whether we want it or not, with supporting reasoning 2. review the architecture and draw attention to any possibly worrisome points 3. checklist items (does it have docs? does it have regression tests? coding style OK? etc.) This is true, but I'd like to know up front that convincing them to revise their thinking is the main thing I need to do during the review process. Restating and clarifying the submitter's arguments, perhaps with the addition of some experimental evidence, is one part where I think I can contribute, provided I know that that is what I need to be doing. Agreed. Having them reserve their opinion until after it is marked ready for commiter doesn't make it easier to change their mind, I don't think. As a reviewer I can't help address their specific concerns, if I don't know what those were. Also agreed. I think one important duty of a reviewer (alluded to above) is to try to draw focus to whatever the central issues around the patch are. For some patches, the big question is is it OK to break backward compatibility like this?, whereas for others it may be is this actually useful? and for still others it may be does the SQL standard have anything to say about this?. Even if you as a reviewer don't know the answer to those questions, clearly delineating the key issues may enable others to comment without having to read and understand the whole patch, which can expedite things considerably. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/20/2013 09:37 PM, Robert Haas wrote: On Sun, Jan 20, 2013 at 7:07 PM, Jeff Janes jeff.ja...@gmail.com wrote: As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. That's a hard question. I don't think so. IMNSHO your job is neither - it is to give us your independent judgment. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Sun, Jan 20, 2013 at 10:04 PM, Andrew Dunstan and...@dunslane.net wrote: On 01/20/2013 09:37 PM, Robert Haas wrote: On Sun, Jan 20, 2013 at 7:07 PM, Jeff Janes jeff.ja...@gmail.com wrote: As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. That's a hard question. I don't think so. IMNSHO your job is neither - it is to give us your independent judgment. That's pretty much what I went on to say. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Robert Haas robertmh...@gmail.com writes: On Sun, Jan 20, 2013 at 7:07 PM, Jeff Janes jeff.ja...@gmail.com wrote: As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. ... I think you should try to persuade the person that you think is wrong, whoever that is. Absolutely. Form your own opinion and marshal an argument for it. At the end of the day, most of us are reasonable people and can be convinced by appropriate evidence. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 21.01.2013 02:07, Jeff Janes wrote: As a junior reviewer, I'd like to know if my main task should be to decide between 1) writing a review convincing you or Tom that your judgement is hasty, or 2) to convince the author that your judgement is correct. That would provide me with some direction, rather than just having me pondering whether a certain variable name ought to have one more or one fewer underscores in it. On the other hand if a summary judgement is that the patch is fundamentally sound but needs some line-by-line code-lawyering, or some performance testing, or documentation/usability testing, or needs to ponder the implications to part XYZ of the code base (which neither I nor the other have even heard of before), that would also be good to know. My desire is not for you to tell me what the outcome of the review should be, but rather what the focus of it should be. Often a patch contains a contentious change buried deep in the patch. The patch submitter might not realize there's anything out of the ordinary in changing parser behavior based on a GUC, or doing things in the executor that should be done in the planner, so he doesn't mention it in the submission email or highlight it with any comments. The longer the patch, the easier it is for things like that to hide in the crowd. If a reviewer can bring those potentially contentious things that don't smell right to light, that's extremely helpful. It's not so important what judgment you make on it; just bringing it up, in a short, separate reply to the email thread, will allow the submitter to reconsider, and a committer to jump in early to say yeah, you can't do that, because X.. IMHO that's the single most important task of a review. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Thursday, January 17, 2013, Magnus Hagander wrote: Would it help to step up a few developers and create a second line of committers ? The commits by the second line committers will still be reviewed by the first line committers before they make into the product, but may be at later stage or when we are in beta. Perhaps this is because I've never needed or played with prepared transactions; but to me a commit is a commit is a commit. If it isn't in the product, then whatever it was was not a commit. .. While we can certainly do that, it would probably help just to havae a second line of reviewers. Basically a set of more senior reviewers - so a patch would go submission - reviewer - senior reviewer - committer. With the second line of reviewers focusing more on the whole how to do things, etc. That order seems partially backwards to me. If someone needs to glance at a patch and say That whole idea is never going to work or more optimistically you need to be making that change in smgr, not lmgr, it is probably not going to be a novice reviewer who does that. Sometime this type of high-level summary review does happen, at the senior person's whim, but is not a formal part of the commit fest process. What I don't know is how much work it takes for one of those senior people to make one of those summary judgments, compared to how much it takes for them to just do an entire review from scratch. Cheers, Jeff
Re: [HACKERS] CF3+4
On Jan 17, 2013 8:15 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-17 16:05:05 +0900, michael.paqu...@gmail.com wrote: Is it really necessary to create a new commit fest just to move the items? Marking the patches that are considered as being too late for 9.3 should be just returned with feedback. Opening 2013-03 is not so much to move existing patches, but to give people a place to submit *new* post-9.3 patches without interrupting the 2013-01 CF. Yeah, and +1 for doing that. The sooner the better. By whichever of the time frames for the cf that had been discussed, it should certainly not be open for accepting new patches for 9.3 anymore. /Magnus
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Thu, Jan 17, 2013 at 8:19 AM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Wed, Jan 16, 2013 at 1:51 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? I know various people (myself included) have been trying to keep CF3 moving, e.g. sending followup mail, adjusting patch status, etc. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? ISTM that even committers are often overwhelmed with the work, their own as well as that of reviewing other's patches and committing them. Especially when a patch is large or touches core areas, I can feel the significant work that the committer has to do even after some help and initial review from CF reviewers. On the other hand, if the patches are not committed in time, we loose context, interest dies down and when the patch is actually picked up by a committer who is often not involved in the original discussion, many points need to be revisited and reworked. Would it help to step up a few developers and create a second line of committers ? The commits by the second line committers will still be reviewed by the first line committers before they make into the product, but may be at later stage or when we are in beta. I thought of even suggesting that the master branch will contain only commits by the first line committers. We then maintain a secondary branch which also have commits from the second line committers in addition to all commits from the master branch. The first line committers can then cherry pick from the secondary branch at some later stage. But not sure if this will add more overhead and defeat the problem at hand. While we can certainly do that, it would probably help just to havae a second line of reviewers. Basically a set of more senior reviewers - so a patch would go submission - reviewer - senior reviewer - committer. With the second line of reviewers focusing more on the whole how to do things, etc. As you, I'm not sure if it creates more overhead than it solves, but it might be worth a try. In a way it already exists, because I'm sure committers pay slightly more attention to reviews by people who ahve been doing it a lot and are known to process those things, than to new entries. All that woudl bee needed was for those people to realize it would be helpful for them to do a second-stage review even if somebody else has done the first one. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On 01/17/2013 04:43 PM, Magnus Hagander wrote: On Jan 17, 2013 8:15 AM, Abhijit Menon-Sen a...@2ndquadrant.com mailto:a...@2ndquadrant.com wrote: At 2013-01-17 16:05:05 +0900, michael.paqu...@gmail.com mailto:michael.paqu...@gmail.com wrote: Is it really necessary to create a new commit fest just to move the items? Marking the patches that are considered as being too late for 9.3 should be just returned with feedback. Opening 2013-03 is not so much to move existing patches, but to give people a place to submit *new* post-9.3 patches without interrupting the 2013-01 CF. Yeah, and +1 for doing that. The sooner the better. By whichever of the time frames for the cf that had been discussed, it should certainly not be open for accepting new patches for 9.3 anymore. I've moved all pending patches from 2012-11 to 2013-01. I'll go through and poke them for aliveness and start chasing things up; in the mean time, any chance of closing 2012-11? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services
Re: [HACKERS] CF3+4
On Thu, Jan 17, 2013 at 8:17 AM, Craig Ringer cr...@2ndquadrant.com wrote: I've moved all pending patches from 2012-11 to 2013-01. I'll go through and poke them for aliveness and start chasing things up; in the mean time, any chance of closing 2012-11? Done. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 9:21 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Not sure. One start might be to actually start having commitfest managers. I haven't actually checked numbers, but my feeling is that last round most commitfests (maybe not all) had a commitfest manager that worked (usually hard) at keeping things moving, whereas this round most (or all?) commitfests have been unmanaged. I'm not sure if that's the reason for it, but it's a feeling I have.. I know various people (myself included) have been trying to keep CF3 moving, e.g. sending followup mail, adjusting patch status, etc. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? I'm not one of the committers that pick up the most patches, but from reading messages on the lists I think fairly often patches that are marked as ready, aren't. Sometimes they require a small change, which is fine, but more often than not once it hits a committer it ends up with a lot of feedback requiring rather extensive changes. As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. Would it help at all to move all pending items (i.e. anything less than ready) from CF3 to CF4, just so that the committers have only one list to look at, while reviewers can work on the other? Only psychological, but maybe that's better than the current situation? No, it would. They should've been bounced to the next commitfest when the last one closed. The problem was we never closed it... Tom also wrote: In case you hadn't noticed, we've totally lost control of the CF process. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement in Ottawa that nothing big would be accepted after CF3. At this point I'd bet against releasing 9.3 during 2013. Well, if we said that, why don't we just single out those patches, and bounce them right now. Before people put more work into it. We also talked about the one-patch-one-review. Did someone ever check if that worked out - did we get that spread, or did we end up with the same ratio as last time? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 16 January 2013 08:21, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Totally lost control is an overstatement. The current situation is that there are clearly more patches than people working on them; a situation that we've been in, with various degrees for all of the last 9 years of my involvement with PostgreSQL. AFAIK there was no CF manager assigned to Nov2012 CF, so not surprising it is off track. Not sure we ever agreed a process for assigning CF mgrs, so again, not surprising it is off track. I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 1:12 PM, Simon Riggs si...@2ndquadrant.com wrote: On 16 January 2013 08:21, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Totally lost control is an overstatement. The current situation is that there are clearly more patches than people working on them; a situation that we've been in, with various degrees for all of the last 9 years of my involvement with PostgreSQL. AFAIK there was no CF manager assigned to Nov2012 CF, so not surprising it is off track. Not sure we ever agreed a process for assigning CF mgrs, so again, not surprising it is off track. I don't think we have a process for it, and that's perhaps something we need to discuss for the future. I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. If he's willing to do that, then +1 from me. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
At 2013-01-16 13:08:27 +0100, mag...@hagander.net wrote: One start might be to actually start having commitfest managers. (I'm skipping over this point, since Craig's nomination as CF manager is being discussed elsewhere in this thread.) As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. It may be best to return the patch with feedback in that case (i.e. fail fast) rather than fix the problem. With some advice about the right way to do things, the original author or someone else can do the remaining work before a committer looks at it again. If that works, then everyone involved will have a better idea of where each patch stands, rather than having the opaque backlog we do now. In the best case, more people can get some hands-on experience in what it takes to prepare properly committable patches, and thus improve any reviews they do in future. I can only speak for myself, but I would be happy to help polish up patches like this. I don't know if this will help (and I haven't checked to see if it's been discussed before), but perhaps it's worth trying? Right now there seems to be very little else we non-committers can do to help directly. Aside: I thought about importing all CF patches into separate branches in one git repository, so that one doesn't have to play hunt-the-wumpus to find the latest version of a patch, and it's easier to submit small fixes, easier to reject or fix patches that don't apply, easier to fix whitespace errors once and for all (or even run pg_indent), etc. If any committers think it would be helpful, please let me know. We also talked about the one-patch-one-review. Did someone ever check if that worked out - did we get that spread, or did we end up with the same ratio as last time? Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). First, authors: 1 Abhijit Menon-Sen 1 Alexander Korotkov, Oleg Bartunov 1 Alexander Lakhin 1 Amit Kapila, Hari Babu 1 Asif Rehman 1 Atri Sharma 1 cjk 1 Dean Rasheed 1 Etsuro Fujita 1 Greg Smith 1 Heikki Linnakangas 1 Jameison Martin 1 Jan Wieck 1 Jeff Janes 1 Jeremy Evans 1 Joachim Wieland 1 Josh Kupershmidt 1 Kevin Grittner 1 Kyotaro Horiguichi 1 Lars Kanis 1 Marko Tiikkaja, Joel Jacobson 1 Oskari Saarenmaa 1 Phil Sorber 1 Robert Haas 1 Satoshi Nagayasu 1 Vik Reykja 1 Will Leinweber 2 Álvaro Herrera 2 Andres Freund 2 Fujii Masao 2 Kyotaro Horiguchi 2 Peter Eisentraut 3 Amit Kapila 3 Dimitri Fontaine 3 Pavel Stehule 3 Tomas Vondra 3 Zoltán Böszörményi 7 Alexander Korotkov 13 Karl O. Pinc Second, reviewers: 1 Alastair Turner 1 Ali Dar 1 Amit Khandekar 1 Andrew Dunstan 1 Asif Rehman 1 Dimitri Fontaine 1 Etsuro Fujita 1 Hari Babu 1 Heikki Linnakangas 1 Ibrar Ahmed 1 Jeevan Chalke 1 Josh Berkus 1 Josh Kupershmidt 1 KaiGai Kohei 1 Kevin Grittner 1 Kyotaro Horiguchi 1 Magnus Hagander 1 Michael Paquier 1 Muhammad Usama 1 Piyush Newe 1 Simon Riggs 1 Vik Reykja 1 Zoltán Böszörményi 2 Abhijit Menon-Sen 2 Andres Freund 2 Fujii Masao 2 Karl O. Pinc 2 Marko Tiikkaja 2 Noah Misch 2 Shigeru Hanada 2 Tom Lane 3 Peter Geoghegan 3 Tomas Vondra 4 Jeff Davis 4 Pavel Stehule 4 Robert Haas 6 Amit Kapila 9 Peter Eisentraut 10 Nobody I haven't looked too carefully, but it seems to me that the people who reviewed patches reviewed more than the number they submitted, but also many people didn't review anything. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
2013-01-16 14:18 keltezéssel, Abhijit Menon-Sen írta: At 2013-01-16 13:08:27 +0100, mag...@hagander.net wrote: One start might be to actually start having commitfest managers. (I'm skipping over this point, since Craig's nomination as CF manager is being discussed elsewhere in this thread.) As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. It may be best to return the patch with feedback in that case (i.e. fail fast) rather than fix the problem. With some advice about the right way to do things, the original author or someone else can do the remaining work before a committer looks at it again. If that works, then everyone involved will have a better idea of where each patch stands, rather than having the opaque backlog we do now. In the best case, more people can get some hands-on experience in what it takes to prepare properly committable patches, and thus improve any reviews they do in future. I can only speak for myself, but I would be happy to help polish up patches like this. I don't know if this will help (and I haven't checked to see if it's been discussed before), but perhaps it's worth trying? Right now there seems to be very little else we non-committers can do to help directly. Aside: I thought about importing all CF patches into separate branches in one git repository, so that one doesn't have to play hunt-the-wumpus to find the latest version of a patch, and it's easier to submit small fixes, easier to reject or fix patches that don't apply, easier to fix whitespace errors once and for all (or even run pg_indent), etc. If any committers think it would be helpful, please let me know. We also talked about the one-patch-one-review. Did someone ever check if that worked out - did we get that spread, or did we end up with the same ratio as last time? Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). First, authors: 1 Abhijit Menon-Sen 1 Alexander Korotkov, Oleg Bartunov 1 Alexander Lakhin 1 Amit Kapila, Hari Babu 1 Asif Rehman 1 Atri Sharma 1 cjk 1 Dean Rasheed 1 Etsuro Fujita 1 Greg Smith 1 Heikki Linnakangas 1 Jameison Martin 1 Jan Wieck 1 Jeff Janes 1 Jeremy Evans 1 Joachim Wieland 1 Josh Kupershmidt 1 Kevin Grittner 1 Kyotaro Horiguichi 1 Lars Kanis 1 Marko Tiikkaja, Joel Jacobson 1 Oskari Saarenmaa 1 Phil Sorber 1 Robert Haas 1 Satoshi Nagayasu 1 Vik Reykja 1 Will Leinweber 2 Álvaro Herrera 2 Andres Freund 2 Fujii Masao 2 Kyotaro Horiguchi 2 Peter Eisentraut 3 Amit Kapila 3 Dimitri Fontaine 3 Pavel Stehule 3 Tomas Vondra 3 Zoltán Böszörményi It should be 2 submitted patches. The PQconninfo() was a subfeature of the pg_basebackup patch, Magnus added it to the commitfest separately. 7 Alexander Korotkov 13 Karl O. Pinc Second, reviewers: 1 Alastair Turner 1 Ali Dar 1 Amit Khandekar 1 Andrew Dunstan 1 Asif Rehman 1 Dimitri Fontaine 1 Etsuro Fujita 1 Hari Babu 1 Heikki Linnakangas 1 Ibrar Ahmed 1 Jeevan Chalke 1 Josh Berkus 1 Josh Kupershmidt 1 KaiGai Kohei 1 Kevin Grittner 1 Kyotaro Horiguchi 1 Magnus Hagander 1 Michael Paquier 1 Muhammad Usama 1 Piyush Newe 1 Simon Riggs 1 Vik Reykja 1 Zoltán Böszörményi I reviewed 2 patches: https://commitfest.postgresql.org/action/patch_view?id=1005 https://commitfest.postgresql.org/action/patch_view?id=1006 2 Abhijit Menon-Sen 2 Andres Freund 2 Fujii Masao 2 Karl O. Pinc 2 Marko Tiikkaja 2 Noah Misch 2 Shigeru Hanada 2 Tom Lane 3 Peter Geoghegan 3 Tomas Vondra 4 Jeff Davis 4 Pavel Stehule 4 Robert Haas 6 Amit Kapila 9 Peter Eisentraut 10 Nobody I haven't looked too carefully, but it seems to me that the people who reviewed patches reviewed more than the number they submitted, but also many people didn't review anything. -- Abhijit -- -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Simon Riggs escribió: On 16 January 2013 08:21, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Totally lost control is an overstatement. The current situation is that there are clearly more patches than people working on them; a situation that we've been in, with various degrees for all of the last 9 years of my involvement with PostgreSQL. AFAIK there was no CF manager assigned to Nov2012 CF, so not surprising it is off track. Not sure we ever agreed a process for assigning CF mgrs, so again, not surprising it is off track. FWIW, the September 2012 commitfest also went without a manager *for a month*. I started doing the job at that point because it was pretty clear to me that there was a serious problem. And when I finally closed it (just before the Nov. 2012 one was about to start), it was my intention to manage the next one as well; however somebody else told me they wanted that job, so I stepped aside. That was, probably, a bad decision on my part, and I blame myself for not keeping a closer eye on things that I intuitively knew were going wrong. I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. Seconded. I particularly like the fact that Craig is not already a PG developer, so he's not going to be working on his own patches. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. Seconded. I particularly like the fact that Craig is not already a PG developer, so he's not going to be working on his own patches. So when can he start? :D Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] CF3+4
At 2013-01-16 09:02:45 -0500, sfr...@snowman.net wrote: So when can he start? :D Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. (If others agree, I can do some patch-moving, unless some special privileges are needed to do it.) -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/16/2013 10:02 PM, Stephen Frost wrote: * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. Seconded. I particularly like the fact that Craig is not already a PG developer, so he's not going to be working on his own patches. So when can he start? :D Hey, I've had a few toy patches accepted... but yeah, I'm more interested in build systems, QA, docs, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/16/2013 08:12 PM, Simon Riggs wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. I'm happy to step up and help out. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
* Abhijit Menon-Sen (a...@2ndquadrant.com) wrote: Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. I'd leave it up to him to decide, but I think the general thought was to move it all to one place. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 01:08:27PM +0100, Magnus Hagander wrote: On Wed, Jan 16, 2013 at 9:21 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Not sure. One start might be to actually start having commitfest managers. I haven't actually checked numbers, but my feeling is that last round most commitfests (maybe not all) had a commitfest manager that worked (usually hard) at keeping things moving, whereas this round most (or all?) commitfests have been unmanaged. I'm not sure if that's the reason for it, but it's a feeling I have.. Agreed. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? I'm not one of the committers that pick up the most patches, but from reading messages on the lists I think fairly often patches that are marked as ready, aren't. Sometimes they require a small change, which is fine, but more often than not once it hits a committer it ends up with a lot of feedback requiring rather extensive changes. That's an intrinsic hazard of non-committer reviews. All the people who are truly skilled at distinguishing ready patches from not-ready patches are already committers, but a non-committer review catching even one important problem adds value. I'd rather have more people getting involved in the process with high-effort reviews, however imperfect. As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. That's even harder, because it's subjective. Committers often disagree on such matters. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. +1, although I'll suggest that we should have *two* CF managers for this one to keep the workload manageable. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
Stephen Frost sfr...@snowman.net writes: * Abhijit Menon-Sen (a...@2ndquadrant.com) wrote: Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. I'd leave it up to him to decide, but I think the general thought was to move it all to one place. The original intention, per agreement at the last dev meeting, https://wiki.postgresql.org/wiki/PgCon_2012_Developer_Meeting#CommitFest_Schedule was that we'd have a triage discussion between CF3 and CF4 to try to figure out which remaining big patches had a realistic chance of getting committed during CF4. The ones that didn't could then be deferred to 9.4 without first sucking a lot of time away from the ones that could get in. If we decide to fold CF3 and CF4 together, either we lose that step (which would make me sad, it seems like a good idea) or we need to figure another way to work it into the process. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
If we decide to fold CF3 and CF4 together, either we lose that step (which would make me sad, it seems like a good idea) or we need to figure another way to work it into the process. Well, we should have the triage discussion ASAP then. We were really supposed to have it a week ago. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
Josh Berkus j...@agliodbs.com writes: If we decide to fold CF3 and CF4 together, either we lose that step (which would make me sad, it seems like a good idea) or we need to figure another way to work it into the process. Well, we should have the triage discussion ASAP then. We were really supposed to have it a week ago. Well, I don't want to just say hey, it's triage time! while we're still actively working on the CF. Personally, the reason I've accomplished nothing since early December on CF3 is lack of bandwidth --- bug fixing, email-answering, and non-PG Red Hat work have consumed all my time that wasn't eaten by holiday distractions. I'm hoping to get something done soon towards committing immediately-committable CF entries, but if we're instead focusing on patches that require triage discussions, that's likely to not happen. Looking back at the original schedule agreement, I see I misremembered it a bit; we actually had the idea for two rounds of which-patches-have- a-chance: CF1: June 15 - July 15 CF2: Sept 15 - Oct 15 CF3: Nov 15 - Dec 15 Planning Week - Dec 8-15 CF4.1: Jan 15 - Feb 15 Final Triage: Feb 1-7 but both of those discussion weeks were mid-CF, and I now think that that's got little chance of working because of bandwidth considerations. We certainly totally forgot about the first one. I think a realistic answer might be to admit that we've slipped quite a bit. Set the end date of CF3 to perhaps end of January, do triage the first week of February, and then start CF4 after that, about three or four weeks later than planned. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On Wed, Jan 16, 2013 at 03:13:50PM -0500, Tom Lane wrote: I think a realistic answer might be to admit that we've slipped quite a bit. Set the end date of CF3 to perhaps end of January, do triage the first week of February, and then start CF4 after that, about three or four weeks later than planned. This makes sense. Things have just not worked out as planned and we should adjust. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 12:36 PM, Josh Berkus j...@agliodbs.com wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. +1, although I'll suggest that we should have *two* CF managers for this one to keep the workload manageable. That has never worked before, so I'm reluctant to start now. Either one person does all the work and the other person still gets half the credit, or neither person does any work because they're hoping the other person has it covered. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On 16 January 2013 19:28, Tom Lane t...@sss.pgh.pa.us wrote: Stephen Frost sfr...@snowman.net writes: * Abhijit Menon-Sen (a...@2ndquadrant.com) wrote: Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. I'd leave it up to him to decide, but I think the general thought was to move it all to one place. The original intention, per agreement at the last dev meeting, https://wiki.postgresql.org/wiki/PgCon_2012_Developer_Meeting#CommitFest_Schedule was that we'd have a triage discussion between CF3 and CF4 to try to figure out which remaining big patches had a realistic chance of getting committed during CF4. The ones that didn't could then be deferred to 9.4 without first sucking a lot of time away from the ones that could get in. Sorry to correct you but that was not the original intention/agreement. Robert made that suggestion, and I opposed it, saying it was too early and suggesting triage week for first week of Feb instead. My recollection, sitting opposite you, was that you agreed, as did many others. I agree we need triage, and have no problem if you lead that. But lets wait until early Feb, please. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
Simon Riggs si...@2ndquadrant.com writes: I agree we need triage, and have no problem if you lead that. But lets wait until early Feb, please. See my later response in the thread, where I corrected my initial recollection (of the first proposal) to be what the meeting minutes say we agreed to. However, since we already missed the scheduling agreed to then, the question that's on the table now is what we should do instead. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/17/2013 06:01 AM, Robert Haas wrote: On Wed, Jan 16, 2013 at 12:36 PM, Josh Berkus j...@agliodbs.com wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. +1, although I'll suggest that we should have *two* CF managers for this one to keep the workload manageable. That has never worked before, so I'm reluctant to start now. Either one person does all the work and the other person still gets half the credit, or neither person does any work because they're hoping the other person has it covered. Alright - it sounds like there's a tentative agreement on having me help out and as time's a'-flowing I'll get to it. If anyone has anything they think needs specific attention please email me directly. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
At 2013-01-16 22:40:07 -0500, t...@sss.pgh.pa.us wrote: However, since we already missed the scheduling agreed to then, the question that's on the table now is what we should do instead. I suggest we close CF3 and bring the pending CF3 patches into CF4, but still have a triage of CF4 patches in early February. Until then, there are several patches marked Ready for committer for committers to look at when they have time (14 in CF3, 9 in CF4, with two weeks remaining to the beginning of February). Merging the two CFs will result in a huge number of patches, but I don't think we can treat CF4 as being any less in progress than CF3 at this point. People have already started reviewing those patches, etc. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On Wed, 2013-01-16 at 15:13 -0500, Tom Lane wrote: I think a realistic answer might be to admit that we've slipped quite a bit. Set the end date of CF3 to perhaps end of January, do triage the first week of February, and then start CF4 after that, about three or four weeks later than planned. I'd suggest moving all the open items from CF3 into CF4 and start CF4 right away. If we did that, the number of patches in CF4 will be about the same as in the last commit fests of the previous years, so it wouldn't be completely off track. If you postpone the start of the last commit fest to mid-February, I'd expect that we will have a much larger number of patches. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
Peter Eisentraut pete...@gmx.net writes: If you postpone the start of the last commit fest to mid-February, I'd expect that we will have a much larger number of patches. Since some people seem to be under the impression that new work is still okay to submit now, you have a point ... we are definitely past what was envisioned as the deadline for new 9.3 patches. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On 01/17/2013 12:48 PM, Abhijit Menon-Sen wrote: At 2013-01-16 22:40:07 -0500, t...@sss.pgh.pa.us wrote: However, since we already missed the scheduling agreed to then, the question that's on the table now is what we should do instead. I suggest we close CF3 and bring the pending CF3 patches into CF4, but still have a triage of CF4 patches in early February. Until then, there are several patches marked Ready for committer for committers to look at when they have time (14 in CF3, 9 in CF4, with two weeks remaining to the beginning of February). This seems sensible to me. 2012-11 is gone, whether truly finished or not, and if everyone's OK with it I'd like to move all open work into 2013-01, close 2012-11, and open 2013-03 for post-9.3 work. That'll at least provide a place for post-9.3 patches and consolodate everything for somewhat easier tracking. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On Thu, Jan 17, 2013 at 3:52 PM, Craig Ringer cr...@2ndquadrant.com wrote: This seems sensible to me. 2012-11 is gone, whether truly finished or not, and if everyone's OK with it I'd like to move all open work into 2013-01, close 2012-11, and open 2013-03 for post-9.3 work. That'll at least provide a place for post-9.3 patches and consolodate everything for somewhat easier tracking. Is it really necessary to create a new commit fest just to move the items? Marking the patches that are considered as being too late for 9.3 should be just returned with feedback. The former patch writers, or people who want to pick up the old patches and resubmit them, will just need to move the items to the 9.4 commit fests once they are officially created. -- Michael Paquier http://michael.otacoo.com
Re: [HACKERS] CF3+4
At 2013-01-17 16:05:05 +0900, michael.paqu...@gmail.com wrote: Is it really necessary to create a new commit fest just to move the items? Marking the patches that are considered as being too late for 9.3 should be just returned with feedback. Opening 2013-03 is not so much to move existing patches, but to give people a place to submit *new* post-9.3 patches without interrupting the 2013-01 CF. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 1:51 PM, Abhijit Menon-Sen a...@2ndquadrant.comwrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? I know various people (myself included) have been trying to keep CF3 moving, e.g. sending followup mail, adjusting patch status, etc. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? ISTM that even committers are often overwhelmed with the work, their own as well as that of reviewing other's patches and committing them. Especially when a patch is large or touches core areas, I can feel the significant work that the committer has to do even after some help and initial review from CF reviewers. On the other hand, if the patches are not committed in time, we loose context, interest dies down and when the patch is actually picked up by a committer who is often not involved in the original discussion, many points need to be revisited and reworked. Would it help to step up a few developers and create a second line of committers ? The commits by the second line committers will still be reviewed by the first line committers before they make into the product, but may be at later stage or when we are in beta. I thought of even suggesting that the master branch will contain only commits by the first line committers. We then maintain a secondary branch which also have commits from the second line committers in addition to all commits from the master branch. The first line committers can then cherry pick from the secondary branch at some later stage. But not sure if this will add more overhead and defeat the problem at hand. My 2 cents on this difficult topic. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee
