Re: [HACKERS] 8.4 release planning
Hi, On Tue, Feb 3, 2009 at 5:08 AM, Bruce Momjian br...@momjian.us wrote: o Others We will focus on all the other items on the commit fest page, and that will determine our time-line for 8.4 beta, i.e. the first three items will not delay our beta release. Simon is assigned as reviewer of PITR performance improvement patch, but I think that he is too busy with HS to check the code. More reviewer should be assigned to that? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
I understand Simon is extremely busy on his own patch. I appreciate if anyone help the review. 2009/2/6 Fujii Masao masao.fu...@gmail.com: Hi, On Tue, Feb 3, 2009 at 5:08 AM, Bruce Momjian br...@momjian.us wrote: o Others We will focus on all the other items on the commit fest page, and that will determine our time-line for 8.4 beta, i.e. the first three items will not delay our beta release. Simon is assigned as reviewer of PITR performance improvement patch, but I think that he is too busy with HS to check the code. More reviewer should be assigned to that? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- -- Koichi Suzuki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
To summarize where I think we are, release-wise: o Log streaming hold for 8.5 o Hot standby if committable for 8.4, fine, if not, 8.5, Heikki decides o SE-PostgreSQL no row-level security, if committable for 8.4, fine, if not, 8.5 o Others We will focus on all the other items on the commit fest page, and that will determine our time-line for 8.4 beta, i.e. the first three items will not delay our beta release. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Peter Eisentraut wrote: On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote: well from a quick glance there is the bugzilla demo install as well as pieces of reviewboard and patchwork on the trackerdemo jail. So what's the URL and where can we sign up? resurrected the install and subscribed it to pgsql-hackers: http://trackerdemo.postgresql.org however it seems that it won't deal with patches that just have Content-Type: text/plain (like: http://archives.postgresql.org/pgsql-hackers/2009-01/msg02586.php) - seems not to hard to fix from a quick glance at the code however. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Stefan Kaltenbrunner píše v čt 29. 01. 2009 v 18:29 +0100: Peter Eisentraut wrote: On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote: well from a quick glance there is the bugzilla demo install as well as pieces of reviewboard and patchwork on the trackerdemo jail. So what's the URL and where can we sign up? note the pieces part of my mail :-) As far as I recall the patchworks install somehow collided with the reviewboard one so it was disabled because Zdenek was still actively using reviewboard. I don't use it at this moment. You can disable reviewboard if you want. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Thursday 29 January 2009 12:03:45 Robert Haas wrote: I don't believe that you can speed a project up much by adjusting the length of the release cycle, but it is *sometimes* possible to speed up a project by dividing up the work over more people. This is interesting. We had a problem in 8.3 (and most of the releases before that) of too many patches in the queue at the end of the development cycle. Most everyone agreed that more reviewers/committers would help, but given no way to conjure them up, they realized that wasn't a solution. Instead, we went to a tighter development cycle, with one month of dev and then a commifest. This allowed us to better parralelize both reviews and commits, allowed a number of patches to get bumped through multiple fests with relatively few compliants (after all, the next fest was just a month down the line), keep the patch queue pretty manageable (right up untill the end, when we stopped the cycle), and also delivered us some really big features along the way. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Magnus Hagander wrote: On 29 jan 2009, at 05.35, Bruce Momjian br...@momjian.us wrote: Peter Eisentraut wrote: On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote: Marko Kreen wrote: On 1/27/09, Peter Eisentraut pete...@gmx.net wrote: On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: Such app already exists: http://ozlabs.org/~jk/projects/patchwork/ So it's a matter of just setting it up. I was in fact in the process of setting that up just now. :-) Nice to know. :) I feel that even if we decide to do our own solution it would be good to try existing solution first. IIRC, we already installed and tried this a while ago. I don't remember exactly what it failed on, but there was something pretty clear. But maybe it's been fixed by now. Details? I find no public record of this. I think it was Keystone; Marc set it up. Not at all. That was *ages* ago. This was recently, and I was part of setting it up myself... In fact we still have the jail running ... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Stefan Kaltenbrunner wrote: Magnus Hagander wrote: On 29 jan 2009, at 05.35, Bruce Momjian br...@momjian.us wrote: Peter Eisentraut wrote: On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote: Marko Kreen wrote: On 1/27/09, Peter Eisentraut pete...@gmx.net wrote: On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: Such app already exists: http://ozlabs.org/~jk/projects/patchwork/ So it's a matter of just setting it up. I was in fact in the process of setting that up just now. :-) Nice to know. :) I feel that even if we decide to do our own solution it would be good to try existing solution first. IIRC, we already installed and tried this a while ago. I don't remember exactly what it failed on, but there was something pretty clear. But maybe it's been fixed by now. Details? I find no public record of this. I think it was Keystone; Marc set it up. Not at all. That was *ages* ago. This was recently, and I was part of setting it up myself... In fact we still have the jail running ... That's reviewboard AFAIK, or do we also have one with patchwork? Which one? //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Magnus Hagander wrote: Stefan Kaltenbrunner wrote: Magnus Hagander wrote: On 29 jan 2009, at 05.35, Bruce Momjian br...@momjian.us wrote: Peter Eisentraut wrote: On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote: Marko Kreen wrote: On 1/27/09, Peter Eisentraut pete...@gmx.net wrote: On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: Such app already exists: http://ozlabs.org/~jk/projects/patchwork/ So it's a matter of just setting it up. I was in fact in the process of setting that up just now. :-) Nice to know. :) I feel that even if we decide to do our own solution it would be good to try existing solution first. IIRC, we already installed and tried this a while ago. I don't remember exactly what it failed on, but there was something pretty clear. But maybe it's been fixed by now. Details? I find no public record of this. I think it was Keystone; Marc set it up. Not at all. That was *ages* ago. This was recently, and I was part of setting it up myself... In fact we still have the jail running ... That's reviewboard AFAIK, or do we also have one with patchwork? Which one? well from a quick glance there is the bugzilla demo install as well as pieces of reviewboard and patchwork on the trackerdemo jail. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
read up-thread, i've already shown that this would not be the case. remember, we reduce the pressure from the large, complex patches that bottleneck the process, which allows more parralell review/commit. I read what you wrote - I just don't believe it. My own experience is that doing more releases is more work. Also, two commitfests per release means that if you can't get your patch up to snuff in two iterations, you're bumped. The diminished pain of being bumped will, I think, be more than balanced out by the increased frequency of bumps. Many patches needed 2 or 3 commitfests to get committed; all of the people who need 3 iterations, and anyone who needs 2 iterations and doesn't submit until the second commitfest of the cycle, will take two releases to get out the door. I don't believe you're ever going to make beta/release so low-impact that that won't be disruptive or irritating to people. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Haas robertmh...@gmail.com writes: read up-thread, i've already shown that this would not be the case. remember, we reduce the pressure from the large, complex patches that bottleneck the process, which allows more parralell review/commit. I read what you wrote - I just don't believe it. My own experience is that doing more releases is more work. Yeah, more releases than once a year is kind of crazy. Also, two commitfests per release means that if you can't get your patch up to snuff in two iterations, you're bumped. I wish we could get rid of the whole concept and stigma of being bumped your patch will be released in the next release after it's committed. What does it matter if there's been another release since you started or not? ISTM there are two ways to make the release step take less time. Either we sacrifice quality or we change the development process so more testing and review happens during development. I see the latter as realistic if we hold off committing (but not reviewing) any major patches submitted after commitfest#1 until the next commitfest#1. That means when it comes time to release there won't be any major changes in it that people haven't had months to experiment with already. I would still like an answer to my question about what steps there are that take so many months for a release, but I expect most of them boil down to (justified) paranoia about testing major features that people haven't already tested outside of development environments. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote: well from a quick glance there is the bugzilla demo install as well as pieces of reviewboard and patchwork on the trackerdemo jail. So what's the URL and where can we sign up? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Thursday 29 January 2009 08:39:48 Gregory Stark wrote: I wish we could get rid of the whole concept and stigma of being bumped your patch will be released in the next release after it's committed. What does it matter if there's been another release since you started or not? This is the whole point. It isn't that there is a stigma to getting bumped; It matters becase missing a release means 12-14 months before your feature will be released, even when it takes far less time than that to complete the work. That 12-14 month delay has real implications on a number of levels for users and developers. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
I wish we could get rid of the whole concept and stigma of being bumped your patch will be released in the next release after it's committed. What does it matter if there's been another release since you started or not? It matters because the intervening beta/release cycle is likely to sap some focus from the ongoing process of patch review. If nothing else, it's a longer period of time before the patch gets another look, and authors, reviewers, and committers have moved on to other things and forgotten details that then need to be rehashed. If we could have commitfests every two months regardless of the release schedule, then I think the timing of releases really wouldn't matter as much. But then we'd need multiple branches and I don't think Tom et. al. want to go that route. And I understand why. Merging in CVS is the suck, but even if you can make that aspect of things easier, it's still going to be some work, and you still have the problem that people might not do enough testing on release N if they're already in the midst of heavy development for release N+1. Linux solves this problem by having back-branch maintainers and subsystem maintainers who have roles that are intermediate between random patch submitter and full committer. We don't really have quite that much structure, maybe because we're a small project. But it's worth thinking about, because if Tom or Peter or Alvaro or Heikki called me up and said, If you agree to do be responsible for task X over the next year, which I estimate will save me 40 hours of work, I will agree to spend an additional 10 hours over that same time period reviewing and potentially committing your patches - I would probably take that deal. And if I didn't, I bet there are at least five other people who would be more than happy to get in line. Of course, making that work depends on one of those people having a well-defined task that they trust me (or whoever) to do and that can be severed from the rest of their work, and there may not be anything of that nature (or maybe it's at a higher increment, like 200 hours for 50 hours, in which case it would be more than I could take on, but someone else might be interested). But I think that's what we have to look for. I don't believe that you can speed a project up much by adjusting the length of the release cycle, but it is *sometimes* possible to speed up a project by dividing up the work over more people. I would still like an answer to my question about what steps there are that take so many months for a release, but I expect most of them boil down to (justified) paranoia about testing major features that people haven't already tested outside of development environments. That I'm not sure about. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Peter Eisentraut wrote: On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote: well from a quick glance there is the bugzilla demo install as well as pieces of reviewboard and patchwork on the trackerdemo jail. So what's the URL and where can we sign up? note the pieces part of my mail :-) As far as I recall the patchworks install somehow collided with the reviewboard one so it was disabled because Zdenek was still actively using reviewboard. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
All, Thing is, our review/commit process is so peculiar to our project that using *any* prebuilt solution would require us to change our process to support the tool. And I can't imagine this group doing that. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On Thu, 2009-01-29 at 10:18 -0800, Josh Berkus wrote: All, Thing is, our review/commit process is so peculiar to our project that using *any* prebuilt solution would require us to change our process to support the tool. And I can't imagine this group doing that. I am not sure I agree with this. Someone submits patch ticket is created reviewer takes ticket comments submitter takes ticket fixes based on comments review takes ticket approves if reviewer is a committers, he commits. if reviewer isn't he set the ticket to need final review tickets that are in that state are reviewed by commiters. Sounds like standard stuff to me. Joshua D. Drake --Josh -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Josh, Someone submits patch ticket is created reviewer takes ticket comments submitter takes ticket fixes based on comments review takes ticket approves if reviewer is a committers, he commits. if reviewer isn't he set the ticket to need final review tickets that are in that state are reviewed by commiters. Sounds like standard stuff to me. But that's *not* actually how we do things. So you're making my point. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Josh Berkus j...@agliodbs.com writes: But that's *not* actually how we do things. So you're making my point. Well, the stuff around the wiki status board is pretty new and I don't think anyone feels that it's set in stone yet. The thing we don't want to compromise on, IMHO, is that the long-term record of what's happened is in the mailing list archives and *not* in the internal state of some tool we happen to be using. (One obvious reason for not compromising on that is that we'd be locked into whatever tool we first pick.) But it doesn't really matter whether the tool thinks it has archival state, as long as we can make it link to the archives conveniently. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Greg Smith wrote: Where I suspect this is all is going to settle down into is that if 1) the SE GUC is on and 2) one of the tables in a join has rows filtered, then you can expect that a) it's possible that the result will leak information, which certainly need to be documented, As far as I can tell this is the case however you hide the information. If you implemented it with views you'll have the same issue. If you hide the existence of project p_id=TOPSECRET01 and people can run inserts then they can spot it. Likewise, it you have fkey references to the row then deletions can be used to spot it. -- Richard Huxton Archonet Ltd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Greg Smith wrote: PostgreSQL advocacy point, one of the questions Tom asked about a bit upthread is still a bit hazy here. There are commercial database offerings selling into the trusted space already. While the use-cases you describe make perfect sense, I don't think it's clear to everyone yet if there's a unique draw to a PostgreSQL + selinux solution that the class of customers you're talking about would prefer it to purchasing one of those products. Is the cost savings the main driver here, or is there something else about a secure LAPP stack that makes it particularly compelling? According to the data available to me, it is a combination of doing it better than the other guys (e.g., a SELinux type interface instead of something handcrafted) and the usual cost savings. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Peter Eisentraut wrote: On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote: Marko Kreen wrote: On 1/27/09, Peter Eisentraut pete...@gmx.net wrote: On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: Such app already exists: http://ozlabs.org/~jk/projects/patchwork/ So it's a matter of just setting it up. I was in fact in the process of setting that up just now. :-) Nice to know. :) I feel that even if we decide to do our own solution it would be good to try existing solution first. IIRC, we already installed and tried this a while ago. I don't remember exactly what it failed on, but there was something pretty clear. But maybe it's been fixed by now. Details? I find no public record of this. I don't recall specifically :-( Which in itself might mean it's worthwhile to make another try. But i recall trying that one and reviewboard, and none of them was what we needed. If you look at Berkus' list of required features (if you haven't seen it, I'm sure he'll be happy to send you a copy), you will see that it doesn't come close. We can always argue if his list is reasonable :-), but that's just a fact. It has nothing about round-robin reviewers. It has no keep-track-of-nagging features. It has no integration with our mail archives. At least it didn't then - it also appears to have no online documentation, so I can't easily check now :-P //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Magnus Hagander a écrit : Peter Eisentraut wrote: On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote: Marko Kreen wrote: On 1/27/09, Peter Eisentraut pete...@gmx.net wrote: On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: Such app already exists: http://ozlabs.org/~jk/projects/patchwork/ So it's a matter of just setting it up. I was in fact in the process of setting that up just now. :-) Nice to know. :) I feel that even if we decide to do our own solution it would be good to try existing solution first. IIRC, we already installed and tried this a while ago. I don't remember exactly what it failed on, but there was something pretty clear. But maybe it's been fixed by now. Details? I find no public record of this. I don't recall specifically :-( Which in itself might mean it's worthwhile to make another try. But i recall trying that one and reviewboard, and none of them was what we needed. If you look at Berkus' list of required features (if you haven't seen it, I'm sure he'll be happy to send you a copy), Josh, can you please give the link to this list of feature ? you will see that it doesn't come close. We can always argue if his list is reasonable :-), but that's just a fact. It has nothing about round-robin reviewers. It has no keep-track-of-nagging features. It has no integration with our mail archives. At least it didn't then - it also appears to have no online documentation, so I can't easily check now :-P //Magnus - -- Cédric Villemain Administrateur de Base de Données Cel: +33 (0)6 74 15 56 53 http://dalibo.com - http://dalibo.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkmAOAMACgkQo/dppWjpEvxn6ACg2F5to39Q9fW9vvm25E9fW2Zl GAAAoOP9yMO3WuT5Rj98s7OyHhDYK4Ui =rP62 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Richard Huxton wrote: Greg Smith wrote: Where I suspect this is all is going to settle down into is that if 1) the SE GUC is on and 2) one of the tables in a join has rows filtered, then you can expect that a) it's possible that the result will leak information, which certainly need to be documented, As far as I can tell this is the case however you hide the information. If you implemented it with views you'll have the same issue. If you hide the existence of project p_id=TOPSECRET01 and people can run inserts then they can spot it. Likewise, it you have fkey references to the row then deletions can be used to spot it. It is a covert channel discussion. At least, SE-PostgreSQL does not care about hiding its existence, so it does not prevent user to infer the existence of a tuple with same key value, using PK confliction. (Please note that he must have a info about PK value or lucky to make a key confliction.) But, it enables to prevent unclassified user to read the tuple, and him to know an info the tuple contains p_id=TOPSECRET01 as a result of this read action. Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wed, Jan 28, 2009 at 4:28 AM, Peter Eisentraut pete...@gmx.net wrote: Greg Smith wrote: PostgreSQL advocacy point, one of the questions Tom asked about a bit upthread is still a bit hazy here. There are commercial database offerings selling into the trusted space already. While the use-cases you describe make perfect sense, I don't think it's clear to everyone yet if there's a unique draw to a PostgreSQL + selinux solution that the class of customers you're talking about would prefer it to purchasing one of those products. Is the cost savings the main driver here, or is there something else about a secure LAPP stack that makes it particularly compelling? According to the data available to me, it is a combination of doing it better than the other guys (e.g., a SELinux type interface instead of something handcrafted) and the usual cost savings. I don't know about better, but I would definitely say that it's a more integrated (with the OS) solution. Can you get Oracle to use SELinux policies? Sure. But it would take a combination of Label Security, Fine Grained Access Control tweaks, custom C functions, and custom policies to handle the access control. And, it would cost a helluva lot of money. In short, this would make Postgres quite a bit more appetizing to those who need this functionality, those who prefer SELinux-based policies, and those who don't have the time/money to do it in systems like Oracle. How many people is that? Based on my consulting experience and questions from DoD/DoE people specifically, I think the number of people needing this feature is fairly small right now. But, it wouldn't hurt us to have it. Just to make it clear, this feature wouldn't make Postgres a trusted database in any certification sense. So, using that term would likely cause confusion and get people who used it thinking it had an EAL certification into trouble. -- Jonah H. Harris, Senior DBA myYearbook.com
Re: [HACKERS] 8.4 release planning
Robert Treat wrote: The problem is that the pain point is extremely high for missing a given release cycle. If you don't make a specific release, you have a 12-14 month wait for feature arrival. Given that choice, someone who deperately need (aka wants) HS/SEPostgres/Win32/HOT/IPU/etc... will likely be willing to push a release 3-6 months for that one feature. There will always be some features that people are willing to push a release for, if it's a feature they want. At least I hope so - because that means we keep adding new features that people want. But as long as there is some overlap in development timelines - which there will *always* be - there will always be some patch that is not quite ready on time. If the release is pushed back, maybe some other patch could also have been finished by the new deadline - should that also be included? What about a completely new feature that isn't even started yet, but that could easily be finished before the new deadline? What makes those less worthy? The question is, how long do we make people wait for *other* features. It delays this version *and* the next. Incidentally, this is probably why people didnt worry about making a given commitfest. The pain point was low, so there was no percieved need to rework a patch for a specific commit, since there was another one just a couple months away. However, we still see a rush of patches at the final freeze because people know that there is no tommorrow at that point. A problem at this point is that we are basically serializing the project over one or a couple of patches. All those people who aren't qualified to review/work on HS or SEPG are left in a position where they can't get anything done. Sure, they can work on patches offline, and add them to a hypothetical future commitfest that they have no clue when it's going to happen, so they don't know when they need to be available to deal with feedback. And we're back to ending up with a lot of conflicting patches simply because they sit in the queue for too long. That's a lot of developer talent wasted. The commitfests were designed in part to get around this - to get developers quick feedback so they can get on with more features. The final commitfest being much longer than the others by design already makes this hard. Dragging it out even longer makes it an even bigger failure in this way. We can't just say that everybody should help with these patches. Not everybody is qualified to do so, or has an interest to, while they're still both qualified and interested in working on other things for 8.5. In the third place, unless we get an upgrade-in-place process that works all the time, we would be looking at maintaining twice as many back branches in order to provide the same kind of release lifespan we have today. We are at the limit of what we can realistically do in back-branch maintenance already :-( Yeah, I can't argue with that. I'm not sure it's an unsolvable problem though; if odd/even release maintenance doesn't sound good, we could do something like ubuntus LTS, where we pick 1 release every 3 years to make a long-term support commitment (I think 5 years is our current max), and keep other releases on a shorter lifespan (1 or 2 years). Certainly having IPU (or is that UIP?) would make that an easier decision. We're still going to have to pay the full cost of doing a release every time. With beta/rc management, release notes, announcements, postings, packaging and all those things. The PostgreSQL tree tends to be a lot more stable than others. In many cases, you can just snapshot CVS HEAD and get more or less the same things. Perhaps if someone were to simply maintain a bunch of tags or branches against known feature-points in the system in a separate SCM somewhere that'd be enough for people who desperately need the features - or who just want to test them out earlier. That would be close to zero cost for the core project to maintain. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus wrote: That's modest. I've talked to several oracle and db2 shops that want a standby for reporting that has relatively easy setup/maintenance (handling ddl is a big part of this) and the HS feature your working on will give them something as good as what they are getting now. So yeah, HS appeals to future users as well. I've talked to some of my clients, and while they *want* synch or near-synch HS, even slow HS is useful to them *now*. One client is planning on deploying a rather complex FS cloning infrastructure just to have a bunch of reporting, testing and read-only search databases they need. They'd be thrilled with an HS feature which produced DBs which were an hour out of date (or even 6 hours out of date), but ran read-only queries. I have a lot of clients who would be thrilled to have stuff that's been in our tree for half a year by now, and they'd be thrilled to have it *now*. How much extra should we have them wait for the needs of your clients? (Yes, I have clients now who would very much like HS as well, of course, but that's not the point) //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Haas wrote: I think the best thing we could do overall is to set release dates and stick to them. If your patch is not ready, well, at least it will get out in a defined amount of time. Right now, the *real* problem with it being pushed to the next release is you don't know how successful some other guy will be at persuading us to delay the next release. +1, LOL. Let's not forget that we've already got CTE, window functions, partial vacuums, and column-level permissions, all of which are major features that should benefit a lot of people. I hope Hot Standby gets committed but even if it doesn't, I'm still going to get a lot of benefit out of this release, so I'd like it to happen on some sort of reasonable time scale. Agreed. Even without HS, this will be one of the biggest releases we've had in a while, IMHO. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Magnus Hagander mag...@hagander.net writes: Josh Berkus wrote: One client is planning on deploying a rather complex FS cloning infrastructure just to have a bunch of reporting, testing and read-only search databases they need. They'd be thrilled with an HS feature which produced DBs which were an hour out of date (or even 6 hours out of date), but ran read-only queries. I have a lot of clients who would be thrilled to have stuff that's been in our tree for half a year by now, and they'd be thrilled to have it *now*. How much extra should we have them wait for the needs of your clients? I really am unconvinced by the argument that delaying existing features is a big deal. Logically it's less of a big deal than delaying HS a whole release cycle which I already said I think isn't a big deal either. This is purely a question of latency between development and release; we still get just as much in each release, it's just 6-12 months later than it might have been. What bothers me is delaying work on things like Bitmap Indexes which won't really start in earnest until Gianni can get feedback from the lists after the release. Or Join Removal which Simon isn't going to look at until after HS is committed (not *released* -- once it's *committed* he'll be free to go on to other things). This would impact *bandwidth* of development which I think is a much bigger deal. It reduces the amount of new features in each release, not just which release they fall in. I'm a bit shocked by how long Tom expects the release cycle to take even if we froze the code today. I guess I forget how long it takes and how many steps there are from past releases. If it's going to be 9+ months between Nov 1st and the first commitfest I'm worried about how many patches will be languishing in the queue with their authors having moved on to other more fruitful pastures in the mean time. If we delay further we're talking about close to a year with developers left hanging. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wed, Jan 28, 2009 at 11:31:25AM +0900, KaiGai Kohei wrote: As I noted before, there is a symmetrical structure between OS and DBMS. Well, you said that before. I think your analogy is awful. I don't think the similarities are nearly as great as you think, and I also think there are significant differences that _make_ the difference in these cases. In particular, In operating system, a process accesses objects (like file, socket, ...) managed by operating system via system calls. Its security system (filesystem permission, SELinux, ...) acquires the request and applies its access control rules. In DBMS, a client accesses database objects managed by DBMS via SQL queries. Its security system (Database ACL, SE-PostgreSQL, ...) aquires the request and applies its access control rules. the difference here is that in the OS, a process accessing the object has few to no guarantees about concurrency. In RDBMS, a very significant reason even to use the DBMS is the ACID guarantees, which make a number of claims about concurrency that simply aren't there in most filesystems. It's at exactly this architectural point that most of the in-principle design questions have been aimed. My personal view is that those questions haven't really been answered very well, but as I've already said I mostly stopped paying attention to this work several months ago; so maybe I overlooked something. I note that Peter and Bruce seem to have been satisfied, so maybe they understood something I don't (that's quite likely). The most significant feature is centralized access control policy between OS and DBMS. Right, I get that; but all the discussion I've seen on this suggest that, to get the benefit of the centralised access control, I trade away certain well-understood assumptions of a relational environment, but without much indication that I've done so. I talked here we should consider the value of information asset is independent from the way to store them. Yes, I know that was your premise. I am not entirely sure I agree with it, is the thing. Needless to say, the value of information asset is decided by its contents. Nonsense. The value of an information asset is determined only partly by its contents. I'd argue that the value of an information asset is a function of its use-value. If the information asset is completely unusable, then it isn't worth anything at all. If your credit card number is recorded on a paper, do you think it has lesser value than recorded on database? Yes. The database makes the credit card number available to other applications, which can then use that data to charge the credit card with other purchases. For me, therefore, the piece of paper, correctly handled, imposes less risk than the database; in addition, the piece of paper offers a smaller advantage, because it cannot be leveraged to make other interactions more convenient. Finally, the piece of paper offers a different kind of risk, because if it is mishandled and then becomes the basis on which the number ends up in a database, I have a new problem for which I was not prepared. I believe my fundamental objection was that, as far as I was able to tell, SELinux simply didn't have anything useful to say about concurrent actions on data under SE controls; that's because it was aimed at a fairly primitive database (a filesystem) without the rich concurrency support of RDBMS. I still don't see anywhere in your discussions an extension of the SELinux model to account for that concurrency richness, so I think there's something wrong with the principles from which your're starting. I'm totally prepared to admit I've missed something, however. Also, since this isn't really my problem any more, I'm unlikely to spend much time reading more design notes or anything of the sort. Finally, It finally enables to apply centralized access control policy on whole of application stack. Please note that 95% of attacks in 2008 targeted to web system, so it gives a nightmare for security folks. this argument gets to the heart of what you seem to want, which is a centralized system that guarantees the controls you want. I'm actually dubious that such centralization is actually the benefit that its proponents seem to think it is; but if it is, then the centralised system needs to be exactly as rich as the richest system under control. By starting with SELinux, I argue that the approach starts with a too-poor model. (See above.) More fundamentally, the premise that the database is just a part of an application stack is, in my view, exactly _why_ these systems are so vulnerable to attack. Database management systems are not designed to be dumb storage for a single application, and they're actually very poorly adapted to such a role. My impression is that SEPostgres is an attempt to finally force the database system under such controls, as though it were a glorified filesystem. I have no idea whether it
Re: [HACKERS] 8.4 release planning
On Wed, 2009-01-28 at 14:55 +0100, Magnus Hagander wrote: If the release is pushed back, maybe some other patch could also have been finished by the new deadline - should that also be included? What about a completely new feature that isn't even started yet, but that could easily be finished before the new deadline? What makes those less worthy? Committers have always added features after freeze... For example, Virtual TransactionIds were added to 8.3 almost exactly 5 months after feature freeze. Not even suggested until about 5 months after, in fact. I argued against such a change late in the cycle, but we did it. It's a great feature and I'm glad we did. If we try to shorten the release cycle, we just end up missing out on tuning opportunities that emerge in beta. IIRC 8.2 was delayed while we changed index cost models. Thankfully. 8.0 was shipped with a completely ineffective bgwriter, so the above changes seem like common sense in comparison. The only way to keep the dev window open longer is to overlap the start of the next cycle with the previous one. i.e. branch new version before final release. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wednesday 28 January 2009 08:55:56 Magnus Hagander wrote: We're still going to have to pay the full cost of doing a release every time. With beta/rc management, release notes, announcements, postings, packaging and all those things. As I pointed out to Tom, by percentage the additional beta/release cycles wouldn't be very different than what we have now; the more churn you have during development, the longer it takes to beta/release. I'm pretty sure that if we had pushed everything not committed on December 1st, we would be very close to release right now, and that's with more dev cycles than I'm talking about for 8.5. And I think most people (aka not the patch authors :-) would have been willing to push the stuff we're dealing with now if they knew the next release would be closer to 6 months than 14 months. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat xzi...@users.sourceforge.net writes: On Wednesday 28 January 2009 08:55:56 Magnus Hagander wrote: We're still going to have to pay the full cost of doing a release every time. With beta/rc management, release notes, announcements, postings, packaging and all those things. As I pointed out to Tom, by percentage the additional beta/release cycles wouldn't be very different than what we have now; the more churn you have during development, the longer it takes to beta/release. I don't believe that thesis in itself, because it ignores economies of scale and parallelism for beta testing. And in any case it's complete nonsense in respect to back-branch maintenance costs. If we double the frequency of releases we are going to be pretty much forced to halve the support lifetime, and ain't nobody going to be happy with us. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Le 28 janv. 09 à 16:22, Simon Riggs si...@2ndquadrant.com a écrit : The only way to keep the dev window open longer is to overlap the start of the next cycle with the previous one. i.e. branch new version before final release. This is the second time the idea is raised and I like it. Do we have anywhere near enough resources for this to happen? That would mean first 8.5 commit-fest begins e.g. April 1st, while hopefully 8.4 enters beta or get ready for it. If no commiter is available for reviewed patch they just get postponed as ready to commit on next commit fest. This way our Round Robin Reviewers team is still at work while in beta, and more importantly developpers still get feedback. -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Dimitri Fontaine dfonta...@hi-media.com writes: Le 28 janv. 09 à 16:22, Simon Riggs si...@2ndquadrant.com a écrit : The only way to keep the dev window open longer is to overlap the start of the next cycle with the previous one. i.e. branch new version before final release. This is the second time the idea is raised and I like it. Do we have anywhere near enough resources for this to happen? No. The key committers are overstressed already. It would also pretty much guarantee that we get *no* help during review and beta, because everyone else will find it more interesting/fun to work on new patches instead. The current system at least gives non-committer developers some motivation to help with that stuff, because they know their patches won't be looked at until beta is over. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Zdenek Kotala wrote: Bruce Momjian p??e v po 26. 01. 2009 v 23:02 -0500: OK, time for me to chime in. I think the outstanding commit-fest items can be broken down into four sections: o Log streaming o Hot standby o SE-PostgreSQL o Others You omit pg_upgrade. Does it mean that this project is already killed for 8.4? I considered pg_upgrade one of the others on the list; it is not as complex as the previous three. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
I considered pg_upgrade one of the others on the list; it is not as complex as the previous three. LOL. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wednesday 28 January 2009 12:35:42 Tom Lane wrote: Robert Treat xzi...@users.sourceforge.net writes: On Wednesday 28 January 2009 08:55:56 Magnus Hagander wrote: We're still going to have to pay the full cost of doing a release every time. With beta/rc management, release notes, announcements, postings, packaging and all those things. As I pointed out to Tom, by percentage the additional beta/release cycles wouldn't be very different than what we have now; the more churn you have during development, the longer it takes to beta/release. I don't believe that thesis in itself, because it ignores economies of scale and parallelism for beta testing. And in any case it's complete nonsense in respect to back-branch maintenance costs. If we double the frequency of releases we are going to be pretty much forced to halve the support lifetime, and ain't nobody going to be happy with us. Yes, back branch maintanance is an issue, but I'd bet that as long as we occasionally designate specific releases as long term support releases (my guess is 1 every 4 releases, though I haven't done the math), people would be comfortable with this. We've already had short maintainance windows for win32 support, and that has gone over without significant uproar. Also other projects (some much larger than ours) have implemented similar schemes, and it's been fairly well recieved. People understand the trade-offs of new features verses stability, and as long as you give them a long term option, they're happy. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 27, 2009 at 12:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Right, but you expect that to be a small and predictable cost, say in the single-digits-percentage range. Plan optimizations that suddenly stop happening can cost you multiple orders of magnitude. Well, look at it another way. If we don't accept row-level security into PostgreSQL, then people will have to implement it themselves. In fact, I currently have a real application that does exactly this. The row-filtering is done, in essence, by having the web application add certain conditions to the WHERE clause of certain queries depending on which user is making the request. And if those WHERE clauses happen to mention columns from table X, then table X won't be a candidate for join removal. The only difference is that the logic is in my app rather than in the database itself. To put that another way, row-level permissions are just another attribute of a table that could potentially affect the query result, and the impact of referring to that attribute will be exactly the same as the impact of referring to any other attribute in that table. The flaw in that argument is that as you are doing it, the de-optimization only happens on queries that actually need the behavior. As the SEPostgres patch is constructed, the planner could *never* trust an FK for optimization since it would have no way to know whether row level permissions might be present (perhaps only for some rows) at execution time. You could only get back the optimization in builds with SEPostgres compiled out. That's pretty nasty, especially for packagers who have to decide which build setting will displease fewer users. I am afraid that SQL-level row permissions would also cause that problem, and I thought they were enabled by default. (The configure flag --enable-selinux only controls SE-Linux support.) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Simon Riggs wrote: On Tue, 2009-01-27 at 14:18 -0500, Tom Lane wrote: Joshua Brindle met...@manicmethod.com writes: Tom Lane wrote: Right, which is why it's bad for something like a foreign key constraint to expose the fact that the row does exist after all. Once again, this is not an issue for us. Yes it is an issue The question of whether there is a covert channel is only a small part of my complaint here. If it's the judgement of security experts that that's an acceptable thing, that's fine, it's their turf. But SQL behavior is my turf, and I'm not happy with discarding fundamental semantic properties. Why did we bother to invite Joshua here if we aren't going to listen to him? Thanks for coming to help Joshua, much appreciated. I agree. This is exactly the type of feedback I was hoping for. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Haas wrote: The flaw in that argument is that as you are doing it, the de-optimization only happens on queries that actually need the behavior. As the SEPostgres patch is constructed, the planner could *never* trust an FK for optimization since it would have no way to know whether row level permissions might be present (perhaps only for some rows) at execution time. You could only get back the optimization in builds with SEPostgres compiled out. That's pretty nasty, especially for packagers who have to decide which build setting will displease fewer users. OK, I think I am starting to understand your concern now. My understanding of how the world works is SE-PostgreSQL would always be compiled in but could be turned off at run-time with a GUC. I know that the original design called for a compile-time switch, but everyone hated it and I am pretty sure KaiGai changed it. If he hasn't, he will. :-) There was also talk of having a table-level option to include/exclude the security ID (I'm not sure if it's currently implemented that way). Obviously that wouldn't be relevant for row-level MAC (because presumably you would need/want that turned on for all tables) but it would be very relevant for row-level DAC (because it's easy, at least for me, to imagine that you would only turn this on for a subset, possibly quite a small subset, of your tables where you knew that it was really needed). If, by default, we make sepostgresql disabled, MAC security IDs on newly created tables off, and DAC security IDs on newly created tables off, then the pain will be confined to people who explicitly request sepostgresql or row-level DAC. Yes, if there is concern about row-level security turning off optimizations, some flag would have to be checked so the optimization would be possible for sites not using row-level security; ideally there would be a table-level flag, and I think the current patch implements it that way. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Bruce Momjian br...@momjian.us writes: Tom Lane wrote: As the SEPostgres patch is constructed, the planner could *never* trust an FK for optimization since it would have no way to know whether row level permissions might be present (perhaps only for some rows) at execution time. You could only get back the optimization in builds with SEPostgres compiled out. That's pretty nasty, especially for packagers who have to decide which build setting will displease fewer users. I am afraid that SQL-level row permissions would also cause that problem, and I thought they were enabled by default. (The configure flag --enable-selinux only controls SE-Linux support.) So they would. However, I've already determined that I'm against row-level permissions of either flavor ;-) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Peter Eisentraut wrote: On Tuesday 27 January 2009 16:36:50 Stephen Frost wrote: * Peter Eisentraut (pete...@gmx.net) wrote: As one of the earlier reviewers, I think the design is OK, but the way the implementation is presented was not acceptable, and very little has been accomplished in terms of reacting to our comments. For example, where is the SQL row security feature, which should have been designed, implemented, and committed separately, in the opinion of most commentaries. Eh? Are you thinking of column-level privileges, which was committed last week? No. The point is we would have preferred to see SQL-level row permissions as a separate patch first; that just didn't happen in this case, and it makes the discussion a little more complex. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Simon Riggs wrote: On Tue, 2009-01-27 at 15:46 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: Not to pick on you personally, but this is the kind of review that should have happened six months ago, not during a why is our development process inadequate discussion on the eve of beta. Right now, today, in this thread, is the first time that we've had any opportunity to debate the design of SEPostgres with knowledgeable people other than KaiGai-san. It would likely be better if we started a new thread with a more appropriate title, but I see nothing wrong with asking pretty fundamental questions. Except that Bruce and I already checked detailed documentation references on this very topic months ago. Check with Bruce; he was careful to point those things out to me, so I'm sure he'll do the same for you. I'm satisfied, on this point. Sure, here is the email: http://archives.postgresql.org/pgsql-hackers/2008-09/msg01750.php -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wed, 2009-01-28 at 17:04 -0500, Tom Lane wrote: The key committers are overstressed already. Some developers are too. I'm sure there's a way to avoid it being a zero-sum game. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat wrote: The revisionism was that of remarkable failure. That was our shortest release cycle in the modern era. And it didn't have the advantage of the commitfest process. But I think what is important here is to recognize why it didn't work. Once again we ended up with large, complex features (HOT, tsearch) that people didn't want to wait 14 months to see if they missed the 8.3 release. And yes, most of these same arguements were raised then... full text search is killer feature, whole applications are waiting for in-core full text search, hot will give allow existing customers to use postgres on a whole new level, not fair to push back patches so long when developers followed the rules, sponsors wont want to pay for features they wont see for years, developers dont want to wait so long to see features committed, and on and on... I think the big reminder for me from above is that we will always have big stuff that doesn't make a certain major release, and trying to circumvent our existing process is usually a mistake. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat wrote: We are going to have exactly no credibility if we tell Simon et al we're pushing these patches to 8.5, but don't worry, it'll be a short release cycle. The other options being we stall 8.4 indefinatly waiting for HS (which, honestly I am comfortable with), or his patches get pushed and he doesnt get them for another 14 months. Seems to me our credibility isn't really even a factor here. Right now I'm really trying to figure out how to solve this problem for the long term. If we say up front now that the next 2 cycles are short cycles, then I think people will be more willing to push patches come end-of-8.5 (and let's not pretend we're not going to have this same argument over streaming replication or synchronous replay or merge command or whatever hot feature is almost ready at that time) You want the fix --- have people complete their patches long before the last commit fest --- no matter of adjustment is going to fix that. If they can't complete it early, fine, but don't expect we can reach some ideal development schedule by adjusting things --- sometimes ideal just isn't possible. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Gregory Stark wrote: I'm a bit shocked by how long Tom expects the release cycle to take even if we froze the code today. I guess I forget how long it takes and how many steps there are from past releases. If it's going to be 9+ months between Nov 1st and the first commitfest I'm worried about how many patches will be languishing in the queue with their authors having moved on to other more fruitful pastures in the mean time. If we delay further we're talking about close to a year with developers left hanging. What makes the period long are the many things we don't know are broken in CVS, but we know we will find before the final release. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wednesday 28 January 2009 20:12:40 Bruce Momjian wrote: Robert Treat wrote: The revisionism was that of remarkable failure. That was our shortest release cycle in the modern era. And it didn't have the advantage of the commitfest process. But I think what is important here is to recognize why it didn't work. Once again we ended up with large, complex features (HOT, tsearch) that people didn't want to wait 14 months to see if they missed the 8.3 release. And yes, most of these same arguements were raised then... full text search is killer feature, whole applications are waiting for in-core full text search, hot will give allow existing customers to use postgres on a whole new level, not fair to push back patches so long when developers followed the rules, sponsors wont want to pay for features they wont see for years, developers dont want to wait so long to see features committed, and on and on... I think the big reminder for me from above is that we will always have big stuff that doesn't make a certain major release, and trying to circumvent our existing process is usually a mistake. Our usual process *is* to try and circumvent our usual process. And I believe it will continue to be that way until we lower the incentive to lobby for circumvention. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat wrote: On Wednesday 28 January 2009 20:12:40 Bruce Momjian wrote: Robert Treat wrote: The revisionism was that of remarkable failure. That was our shortest release cycle in the modern era. And it didn't have the advantage of the commitfest process. But I think what is important here is to recognize why it didn't work. Once again we ended up with large, complex features (HOT, tsearch) that people didn't want to wait 14 months to see if they missed the 8.3 release. And yes, most of these same arguements were raised then... full text search is killer feature, whole applications are waiting for in-core full text search, hot will give allow existing customers to use postgres on a whole new level, not fair to push back patches so long when developers followed the rules, sponsors wont want to pay for features they wont see for years, developers dont want to wait so long to see features committed, and on and on... I think the big reminder for me from above is that we will always have big stuff that doesn't make a certain major release, and trying to circumvent our existing process is usually a mistake. Our usual process *is* to try and circumvent our usual process. And I believe it will continue to be that way until we lower the incentive to lobby for circumvention. Anybody lobbying to get the process circumvented gets their feature reverted? :-) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Our usual process *is* to try and circumvent our usual process. And I believe it will continue to be that way until we lower the incentive to lobby for circumvention. I think Tom and Bruce have both pretty much stated that they're not keen on a shorter release cycle, and they're the ones who would have to do the work, so I think this argument is going nowhere. Moreover, I agree with them. Having short release cycles would probably be a good idea if we had a larger community with more patch authors, more reviewers, and more committers. As it is, I think it would simply mean that the committers would spend more time doing releases and back-branch maintenance, and correspondingly less time to do what we really want them to do: review and commit patches. That problem is already pretty severe, and it would be a bad thing if it got worse. If anyone really can't wait a year for a new feature, they can backport it to the previous release, or pay the patch author to do it. If they were paying the patch author to develop the feature in the first place, it shouldn't be a horribly expensive proposition. At the moment, what we really should be doing is conducting final reviews of as many patches as possible and trying to make sure that they are in good shape to be committed so that the people who have put in hard work for THIS release have a chance to see that work go out the door in a somewhat timely fashion. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
KaiGai Kohei wrote: Bruce Momjian wrote: OK, time for me to chime in. I think the outstanding commit-fest items can be broken down into four sections: o Log streaming o Hot standby o SE-PostgreSQL o Others - snip - SE-PostgreSQL has been in steady development for a year so this is the time to decide about it. My feeling is if we don't accept it now, we are never going to have SE-Linux or row-level security. The next week should show us the right direction when we start discussion on Wednesday, noon GMT. Hmm...? This conditional punishment of death seems to me not a reasonable one. If we can found a matter as a result of discussion, which is impossible to fix within reasonable term, I'll agree it being postponed to v8.5. However, why is the punishment of death necessary here? What I meant was that this features is not going to get any better than the work you have done, so if we reject it odds are we will never accept it. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wednesday 28 January 2009 23:42:11 Robert Haas wrote: Our usual process *is* to try and circumvent our usual process. And I believe it will continue to be that way until we lower the incentive to lobby for circumvention. I think Tom and Bruce have both pretty much stated that they're not keen on a shorter release cycle, and they're the ones who would have to do the work, so I think this argument is going nowhere. Moreover, I agree with them. Having short release cycles would probably be a good idea if we had a larger community with more patch authors, more reviewers, and more committers. more reviewers and more committers would actually be an argument against shorter release cycles, since we'd have a better shot at actually getting all patches in in a timely fasion. we dont, and we cant change that. again, thats the whole point of this... look at the variables and see which ones we can and cant change, and if those changes would address the causes. As it is, I think it would simply mean that the committers would spend more time doing releases and back-branch maintenance, and correspondingly less time to do what we really want them to do: review and commit patches. That problem is already pretty severe, and it would be a bad thing if it got worse. read up-thread, i've already shown that this would not be the case. remember, we reduce the pressure from the large, complex patches that bottleneck the process, which allows more parralell review/commit. If anyone really can't wait a year for a new feature, they can backport it to the previous release, or pay the patch author to do it. If they were paying the patch author to develop the feature in the first place, it shouldn't be a horribly expensive proposition. i dont think we as a community should encourage people to pay for private releases. that is a *really* bad idea. At the moment, what we really should be doing is conducting final reviews of as many patches as possible and trying to make sure that they are in good shape to be committed so that the people who have put in hard work for THIS release have a chance to see that work go out the door in a somewhat timely fashion. not that i disagree, but if we can improve things for the people working on the next release, well, I think that's a good idea too, and I dont see how doing nothing is going to help. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. I don't buy that. Sure, sync-rep would be the icing on the cake, but HS with a small archive_timeout (even of the order of 10 or 15 minutes) would have been extremely useful on a number of systems I used to run. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote: Then why has *nobody* stepped up to review the design, much less the whole patch? The plain truth is that no one appears to care enough to expend any real effort. I've spent some time looking at it and have made all the comments I wished to make. The design seems clear and fit for purpose, having read KaiGai's excellent Wiki description of how it all fits together and also read some PDF links Bruce sent out. But I've not had time to look at the whole patch and my contacts have not had sufficient time to do anything meaningful with it either. If we can minimise the impact on normal running and it doesn't have any implications for robustness, it should be OK. Surely we should give it a quick review to see if it has any gotchas. If not, and KaiGai is willing to commit to supporting it, then should be good to go. KaiGai isn't a home hacker, he's a lead developer for a major multinational, so we should be able to take his word if he says he will continue to contribute fixes if problems are found. If we don't commit to him and his company then they won't commit to us either. The process works like this: software gets developed, then it gets certified. If its not certified, then Undercover Elephant will not be used by the secret people. We can't answer the will it be certified? question objectively yet. If we have someone willing to write the software and put it forward for certification then we should trust that it probably will pass certification and if it doesn't we will see further patches to allow that to happen. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Dave Page wrote: On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. I don't buy that. Sure, sync-rep would be the icing on the cake, but HS with a small archive_timeout (even of the order of 10 or 15 minutes) would have been extremely useful on a number of systems I used to run. +1 I have customers who want exactly this - a simple to administer, query-able slave that does DDL transparently and is up to date within a controllable time frame. Bluntly, it looks like a killer feature. regards Mark -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Jan 27, 2009, at 2:41 AM, Mark Kirkwood wrote: Dave Page wrote: On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. I don't buy that. Sure, sync-rep would be the icing on the cake, but HS with a small archive_timeout (even of the order of 10 or 15 minutes) would have been extremely useful on a number of systems I used to run. +1 I have customers who want exactly this - a simple to administer, query-able slave that does DDL transparently and is up to date within a controllable time frame. Bluntly, it looks like a killer feature. regards +1 So, I am just a lurker here. I mostly follow hackers to find out if any new features are coming out that will make it worth upgrading, and to keep up on any backwards compatibly changes that I should be aware of. I am on 8.1 and it performs well and no features added since then have seemed worth downing the whole system to do the upgrade for. However, a simple to administer, query-able slave that does DDL transparently and is up to date within a controllable time frame is something that would undoubtably make it worth the upgrade. Whatever version this feature makes it into will probably be the one I will upgrade to. Of course this is just one developer giving you anecdotal evidence and there are obviously many concerns other than just how in demand it is, but I just wanted to register my vote that this is a very sought after feature and it is hard for me to imagine a situation (especially for a 24x7 web application) where having an easy to admin hot standby server wouldn't help your local DBA sleep better at night. Rick -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Tom Lane (t...@sss.pgh.pa.us) wrote: SEPostgres seems qualitatively different to me, though. I think PG people have avoided reviewing it because (a) they weren't interested in it and (b) they knew they were unqualified to review it. Meanwhile it's emerging that the selinux people don't feel qualified to review it either. I'm not quite sure what to do about that. But throw it in there on faith doesn't sound like an appealing answer, and I've got no idea how long it will take to work out a non-faith-based answer. Erm, I have to say here that this strikes me as rather unfair. Perhaps I'm wrong, but I suspect KaiGai feels pretty good about the patch and his qualifications in both the PG realm and the SELinux realm. He's asking the PG folks to review it because that's the process that the PG community (through the CommitFest, etc) has laid out for getting a patch included upstream. I'm confident KaiGai isn't going to just disappear into the ether if the patch is committed. Sure, it'd be nice if 4 or 5 other SELinux developers got in and understood the PG code well enough to implement such a patch, but I think the combination of KaiGai (overall), a seperate SELinux hacker (for the security design and SELinux side of it), and a PG committer (for where the hooks are placed and how), reviewing the patch and being comfortable with it is quite sufficient for a high quality result. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Simon Riggs wrote: On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote: Then why has *nobody* stepped up to review the design, much less the whole patch? The plain truth is that no one appears to care enough to expend any real effort. I've spent some time looking at it and have made all the comments I wished to make. The design seems clear and fit for purpose, having read KaiGai's excellent Wiki description of how it all fits together and also read some PDF links Bruce sent out. Thanks for your comment, although you also have a tough work. But I've not had time to look at the whole patch and my contacts have not had sufficient time to do anything meaningful with it either. If we can minimise the impact on normal running and it doesn't have any implications for robustness, it should be OK. Surely we should give it a quick review to see if it has any gotchas. If not, and KaiGai is willing to commit to supporting it, then should be good to go. KaiGai isn't a home hacker, he's a lead developer for a major multinational, so we should be able to take his word if he says he will continue to contribute fixes if problems are found. If we don't commit to him and his company then they won't commit to us either. Needless to say, I will continue to support the feature. I cannot understand why is it necessary to disappear from here. At least, a binary with --enable-selinux passes all regression test with/without pgace_feature=selinux. The benchmark results I have is a bit legacy, so it is necessary to record it again, but I don't think it gives significant implications on normal running (pgace_feature=none). (Yes, it indeed gives us performance loss with selinux-enabled, but we assume performance is not the first priority in this case.) The process works like this: software gets developed, then it gets certified. If its not certified, then Undercover Elephant will not be used by the secret people. We can't answer the will it be certified? question objectively yet. If we have someone willing to write the software and put it forward for certification then we should trust that it probably will pass certification and if it doesn't we will see further patches to allow that to happen. -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 00:42:32 Ron Mayer wrote: If it were just as easy for us to pull from a all 'pending-patches' for-commit-fest-nov that pass regression tests branch, I'd happily pull from that instead. Considering that most patches don't come with regression tests, this would accomplish very little. And even those patches that did come with regression tests (e.g., updatable views) need a design analysis much more than running an automated test suite. Ultimately, it does come down to human work. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Robert Haas escribió: I think that it would probably be pretty easy to write a webapp to replace the CommitFest web page that basically did the same thing but with a bit more structure around it - with database tables like commitfest, patch, patch_version, patch_comment, and patch_review. I think I might even be willing to write such a webapp if someone would be willing to provide the infrastructure. The CommitFest web page was really useful this time around, but it's not conducive to any kind of automated pull. Hey, if you're willing to do it, we're certainly accepting proposals. The current wiki-based CommitFest is supposed to be just a stop-gap. It was started not only to support 8.4 development, but also as a test of the Commitfest idea itself. This has proven so successful that it's clear we should be going somewhere with it. As for somewhere to host it, we certainly have some servers; not tons, but probably enough. Some of them even have Postgres running on it. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On Tue, Jan 27, 2009 at 1:42 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: As for somewhere to host it, we certainly have some servers; not tons, but probably enough. Some of them even have Postgres running on it. We can certainly host an app under postgresql.org. The bigger issue will be speccing it to meet the requirements of the community without getting bogged down in bike shedding. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Dave Page wrote: On Tue, Jan 27, 2009 at 1:42 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: As for somewhere to host it, we certainly have some servers; not tons, but probably enough. Some of them even have Postgres running on it. We can certainly host an app under postgresql.org. The bigger issue will be speccing it to meet the requirements of the community without getting bogged down in bike shedding. I have started some very trivial work around this a while ago with the intent to get something simple up and working before too much bike shedding is done. I'll contact Robert off-list to discuss that. If somebody else - who actively works with what we have now!! - is interested in that discussion, let me know. Will obviously take it on-list before any decisions are made. So far I'm just talking about discussing a prototype. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On 1/27/09, Alvaro Herrera alvhe...@commandprompt.com wrote: Robert Haas escribió: I think that it would probably be pretty easy to write a webapp to replace the CommitFest web page that basically did the same thing but with a bit more structure around it - with database tables like commitfest, patch, patch_version, patch_comment, and patch_review. I think I might even be willing to write such a webapp if someone would be willing to provide the infrastructure. The CommitFest web page was really useful this time around, but it's not conducive to any kind of automated pull. Hey, if you're willing to do it, we're certainly accepting proposals. The current wiki-based CommitFest is supposed to be just a stop-gap. It was started not only to support 8.4 development, but also as a test of the Commitfest idea itself. This has proven so successful that it's clear we should be going somewhere with it. As for somewhere to host it, we certainly have some servers; not tons, but probably enough. Some of them even have Postgres running on it. Such app already exists: http://ozlabs.org/~jk/projects/patchwork/ So it's a matter of just setting it up. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Bruce Momjian píše v po 26. 01. 2009 v 23:02 -0500: OK, time for me to chime in. I think the outstanding commit-fest items can be broken down into four sections: o Log streaming o Hot standby o SE-PostgreSQL o Others You omit pg_upgrade. Does it mean that this project is already killed for 8.4? Thanks Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 02:21:41 Tom Lane wrote: Then why has *nobody* stepped up to review the design, much less the whole patch? The plain truth is that no one appears to care enough to expend any real effort. But this patch is far too large and invasive to accept on the basis that only one guy understands it and will/might continue to maintain it. As one of the earlier reviewers, I think the design is OK, but the way the implementation is presented was not acceptable, and very little has been accomplished in terms of reacting to our comments. For example, where is the SQL row security feature, which should have been designed, implemented, and committed separately, in the opinion of most commentaries. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Simon Riggs wrote: The process works like this: software gets developed, then it gets certified. If its not certified, then Undercover Elephant will not be used by the secret people. We can't answer the will it be certified? question objectively yet. If we have someone willing to write the software and put it forward for certification then we should trust that it probably will pass certification and if it doesn't we will see further patches to allow that to happen. For what it's worth, we can see that there are indeed Postgres forks on the Common Criteria certified list. http://www.commoncriteriaportal.org/products_DB.html PostgreSQL Certified Version V8.1.5 for Linux ManufacturerAssurance level Certification date NTT DATA CORPORATIONEAL122-MAR-07 Certification report c0089_ecvr.pdf http://www.commoncriteriaportal.org/files/epfiles/c0089_ecvr.pdf though at EAL1 they're quite far from the EAL4+ that DB2, Oracle, etc get. That someone went through the effort suggests that there's at least some interest in getting security certifications for postgres. It'd be interesting to hear from whomever at NTT was involved with that certification, if SEPostgreSQL would have either made that process easier or help postgres achieve a higher level. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
I have started some very trivial work around this a while ago with the intent to get something simple up and working before too much bike shedding is done. I'll contact Robert off-list to discuss that. If somebody else - who actively works with what we have now!! - is interested in that discussion, let me know. Will obviously take it on-list before any decisions are made. So far I'm just talking about discussing a prototype. Sounds good. I think we will have the best chance of success if we keep it real simple. I don't want this to turn into a propaganda war about using everyone's favorite tool. I just want to write down a database schema that mimics the organization of the existing wiki page, put a thin web interface around it, and call it a day. It will take longer to analyze whether some other tool is sufficiently close to that than it will to write a tool that is exactly that. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Peter Eisentraut wrote: On Tuesday 27 January 2009 00:42:32 Ron Mayer wrote: If it were just as easy for us to pull from a all 'pending-patches' for-commit-fest-nov that pass regression tests branch, I'd happily pull from that instead. Considering that most patches don't come with regression tests, this would accomplish very little. And even those patches that did come with regression tests (e.g., updatable views) need a design analysis much more than running an automated test suite. Ultimately, it does come down to human work. So long as the patch passes the pre-existing regression tests, it's likely to be stable enough to run on some of our development instances. I certainly don't suggest that this is a substitute for reviews. Just that more testing of patches might happen incidentally (by people who currently test their own software against CVS head) if all the pending patches for a commit fest were as easy to pull as CVS head. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Peter, * Peter Eisentraut (pete...@gmx.net) wrote: As one of the earlier reviewers, I think the design is OK, but the way the implementation is presented was not acceptable, and very little has been accomplished in terms of reacting to our comments. For example, where is the SQL row security feature, which should have been designed, implemented, and committed separately, in the opinion of most commentaries. Eh? Are you thinking of column-level privileges, which was committed last week? The SQL spec doesn't define row-level security, and coming up with something willy-nilly on our own doesn't really strike me as the best approach. Oracle, SQL Server, etc, also use the security labels concept that the SE-PostgreSQL patch implements. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 06:20:41AM -0800, Ron Mayer wrote: For what it's worth, we can see that there are indeed Postgres forks on the Common Criteria certified list. http://www.commoncriteriaportal.org/products_DB.html PostgreSQL Certified Version V8.1.5 for Linux Manufacturer Assurance level Certification date NTT DATA CORPORATION EAL122-MAR-07 Certification report c0089_ecvr.pdf http://www.commoncriteriaportal.org/files/epfiles/c0089_ecvr.pdf though at EAL1 they're quite far from the EAL4+ that DB2, Oracle, etc get. As far as I understand, the different levels are about assuring a set of code/features to some assurance level. The Wikipedia page[1] gives a reasonable overview of the levels, but basically EAL1 says that a limited amount of effort (in practical terms, several person months/years of time for something like PG) was put in, EAL4 is the highest level before things start getting formal (i.e. you actually have to start doing some mathematical proofs about the design) and EAL7 has barely started, but says that the design is formally verified but the code isn't (as far as I understand). Research groups are suggesting that there should also be levels above EAL7 as we are *starting* to know how to verify code well enough that the code, as well as the design, can now be formally verified (e.g. [2]). Equally important as the assurance level are the actual feature set (there are technical names for this that I know very little about) that was actually tested for. For example, it would be comparatively easy to get PG certified saying that it loads and could be killed, but much harder to get it certified as complying with the complete SQL spec. -- Sam http://samason.me.uk/ [1] http://en.wikipedia.org/wiki/Evaluation_Assurance_Level [2] http://ertos.nicta.com.au/research/l4.verified/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Joshua Brindle met...@manicmethod.com writes: http://marc.info/?l=selinuxm=115762285013528w=2 Is the original discussion thread for the security model used in the sepostgresql work. Hopefully you'll see some of the evidence you speak of there. Thanks for the link. I took a look through that thread and saw a lot of discussion about issues like how to relate the database-side and client-side permissions, which is all good stuff but mostly outside my purview as a database geek. I didn't find anything about the stuff that is really bothering me, which I think can be broken down into two main categories: 1. Silently filtering out rows according to an arbitrary security policy can break a bunch of fundamental SQL semantics, the most obvious being foreign key constraints --- an application might be able to see a This is correct. Strange error conditions can happen when you are using mandatory access controls. The same thing happened in linux when selinux was introduced. There was plenty of code out there that assumed if it was running as root it could do anything. Lots of it didn't even check for error conditions. The existence of poorly written applications should never be an argument against adding security. dependent row that apparently has no referenced row, or might get an update or delete failure for a row that is unreferenced as far as it can see. Things get worse if an application can insert, update or delete rows that it can't select. The only answer I've been able to get about Because type enforcement (the primary mechanism behind selinux) is very flexible it is true that policy writers have plenty of rope to hang themselves with. We can only attempt to educate and document these issues, blocking out security is not a satisfactory answer. what SEPostgres will do about that is we really don't care that we're breaking SQL semantics. I don't find that to be a satisfactory answer. Plenty of people feel the same way about SELinux (or any mandatory access controls). That is why there are options, if you want this security and don't care if your applications puke then enable it, else disable it. Noone is going to force you to use this, right? The security-geek reason why not is that it represents a huge information leak. The database-geek reason why not is that this will The great thing about security is that, by itself, it actually doesn't mean anything. Security is where you are willing to balance between stopping people from getting something done and letting them get something done. In this case, removing all covert channels would not only be impossible but it would make an unusable database system. In SELinux we didn't worry about covert flows, (actually we ignored/documented plenty of overt flows as well). With something as complex as the Linux kernel or an enterprise rdbms it is nearly impossible to eliminate such things. People who need absolute separation of information already have options, multiple server processes, polyinstanciated views, etc. For people that don't care if someone can see that you've inserted a couple rows since the primary key got larger, or can tell that an associated row that isn't visible exists in another table a more flexible, yet still mandatory system like sepostgresql is the answer. The great thing about this work is that, in many cases, a well designed system (that is, well designed for the security policy it is going to be constrained under) should not be impacted greatly by these issues. If a client needs information where they can't see all of the associated rows you can have trusted stored procedures (which run in a different selinux context, as defined by a type transition from the client context) that do the work and return the appropriate results. I know you can't use stored procedures for everything but they'd go a long way in binding queries we trust to the data they expose (just like in SELinux we bind binary code on the filesystem to domains that code can be used to enter) permanently destroy our ability to do a lot of important optimizations, eg join removal on the basis of foreign key constraints. (There are probably other good reasons, but that one will do for starters.) Perhaps this is fixable by constraining what a security policy is allowed to do, or in some other way that I don't know about, but I've seen no serious discussion about that. 2. I don't understand where to draw the dividing line between database system accesses (which can't be security constrained, at least not without breaking things entirely --- eg it will do you little good to imagine that you can hide rows in pg_security from the security-enforcement code ;-)) and user accesses that should be security-constrained. I am certain that the line is muddied beyond usability in the current system; there are a lot of user-exposed functions that are making use of the same infrastructure that core system
Re: [HACKERS] 8.4 release planning
Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: SEPostgres seems qualitatively different to me, though. I think PG people have avoided reviewing it because (a) they weren't interested in it and (b) they knew they were unqualified to review it. Meanwhile it's emerging that the selinux people don't feel qualified to review it either. I'm not quite sure what to do about that. But throw it in there on faith doesn't sound like an appealing answer, and I've got no idea how long it will take to work out a non-faith-based answer. Erm, I have to say here that this strikes me as rather unfair. Perhaps I'm wrong, but I suspect KaiGai feels pretty good about the patch and his qualifications in both the PG realm and the SELinux realm. He's Not only that but he's had many discussions with us about sepostgres, from the security model to his reimplementation of the access vector cache. Just because we haven't been on this list doesn't mean we haven't been watching the work. asking the PG folks to review it because that's the process that the PG community (through the CommitFest, etc) has laid out for getting a patch included upstream. I'm confident KaiGai isn't going to just disappear into the ether if the patch is committed. He hasn't disappeared yet, that is probably a good sign :) Sure, it'd be nice if 4 or 5 other SELinux developers got in and understood the PG code well enough to implement such a patch, but I We aren't a huge community and because of the nature of SELinux we have people spread out over many different projects (X, dbus, NFS, distributions, ipsec/networking, solaris fmac), etc. I'm probably more familiar with databases than the others so I'm here to help (though my time is also spread over many other things). think the combination of KaiGai (overall), a seperate SELinux hacker (for the security design and SELinux side of it), and a PG committer (for where the hooks are placed and how), reviewing the patch and being comfortable with it is quite sufficient for a high quality result. That is all I asked for. No matter how familiar I become with the pgsql code I'll never be as qualified as you guys for identifying security hook call sites that are missing/misplaced. Assuming I think the security backend is correct then it shouldn't be hard for you guys to look at the docs, see that permissions x, y and z are required for operation foo, and know where the possible codepaths for operation foo are and check that the hooks for x, y and z are called. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On Wed, Jan 28, 2009 at 1:35 AM, Robert Haas robertmh...@gmail.com wrote: I have started some very trivial work around this a while ago with the intent to get something simple up and working before too much bike shedding is done. I'll contact Robert off-list to discuss that. If somebody else - who actively works with what we have now!! - is interested in that discussion, let me know. I'm very interested in that discussion. I don't know whether I am actively working with what we have now, but that's because since I wrote the original template structure, it hasn't changed a whole lot. Most of the tweaking has had to do with presentation, and massaging mediawiki to do what we wanted. As Alvaro points out, the wiki approach was intended to provide a stop-gap solution to patch tracking, and also to help us identify what we actually needed from a patch tracker, so that we could make a sensible decision about which tool to use when we did eventually move forward. Will obviously take it on-list before any decisions are made. So far I'm just talking about discussing a prototype. Sounds good. I think we will have the best chance of success if we keep it real simple. I don't want this to turn into a propaganda war about using everyone's favorite tool. I just want to write down a database schema that mimics the organization of the existing wiki page, put a thin web interface around it, and call it a day. It will take longer to analyze whether some other tool is sufficiently close to that than it will to write a tool that is exactly that. I can understand the desire to avoid a propaganda war. These discussions have borne little fruit previously, in part because we haven't had a clear idea of what was actually required from the tool. I think the picture has started to become more clear during the 8.4 dev cycle. Most importantly, there was much ado made about the need for powerful email integration features in previous discussions. This severely restricted our choices (possibly to zero?). I feel that the commitfest wiki has demonstrated that no such integration is required. Everyone wants to keep on using the mailing list for discussion, but we need somewhere else to keep track of patches and their status. To my knowledge, authors have been happy to add patches to the wiki and reviewers have been happy to update their status with no email integration whatsoever. We've continued to discuss things on the lists, while updating the wiki as required. If we forget about trying to integrate with email, the field opens right up and we can use pretty much any just-install-the-package tracking software out there and it will get the job done. For the sake of not advocating my favourite tool, I won't name any particular software, but I can think of several off the top of my head that could mirror the structure we currently have on the wiki without stretching. I think it's possible to skip the roll our own step in all of this and just move on to using a ready-made solution. In reality our requirements are very simple. Writing a low-fi version of the wiki would be pretty easy, but just dropping the patch data we already have into a patch tracker would be even easier. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
I think it's possible to skip the roll our own step in all of this and just move on to using a ready-made solution. In reality our requirements are very simple. Writing a low-fi version of the wiki would be pretty easy, but just dropping the patch data we already have into a patch tracker would be even easier. Well, if you're volunteering to set something up... great. We'll take a look at it when you have it working. That's not what I'm volunteering to do, though. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 06:40 +0100, Pavel Stehule wrote: 8.4-stable 8.4-experimental stable is everything that stable is. PostgreSQL at its best. I dislike this idea - it's same like short processed 8.5 - Actually it isn't because we wouldn't accept features into 8.4-experimental. The only thing we would accept into 8.4-experimental would be bug fixes that would automatically be ported up to 8.5 (or perhaps the other way around). We would still continue to build 8.5 as normal. that is more simple. We have tried the short release cycle before, it was called 8.2. It fails, remarkably. regards Pavel Stehule Well like I said, its just an idea :) Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 00:58 -0500, Jaime Casanova wrote: On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule pavel.steh...@gmail.com wrote: so it could be released. 8.5 should be implemented in shorted cycle - only one commitfest, that is enough (+3 month) for well completing SE and replication patches. we tried this before (8.2 to 8.3 i think), the idea was that the next release should be in 6 months... we release at least 6 months later... 8.2 was a short cycle release that lasted just as long as a normal release cycle :) Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: So, some feedback to make this decision more difficult: Users: care about HS more than anything else in the world. I don't think this is correct. I do. People literally grab my shoulder and ask when we'll have it. I've never seen anything like the interest in this for any database feature, including the fairies-and-unicorns multi-master replication people imagine will scale linearly in every dimension by plugging in nodes. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Dave Page dp...@pgadmin.org writes: On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Josh Berkus j...@agliodbs.com writes: Users: care about HS more than anything else in the world. I don't think this is correct. There are certainly a lot of users who would like an in-core replication solution, but HS by itself is not that --- you also need (near) real-time log shipping, which we have already decided to punt to 8.5. That being the case, I think the argument that HS is a must-have feature for 8.4 is actually rather weak. I don't buy that. Sure, sync-rep would be the icing on the cake, but HS with a small archive_timeout (even of the order of 10 or 15 minutes) would have been extremely useful on a number of systems I used to run. Sure, I don't deny that HS by itself would have significant use cases. But what those zillions of users want is easy-to-set-up replication (think mysql). Without an integrated and fairly high-performance log shipping capability, they are not going to find HS very compelling. Claiming otherwise is just wishful thinking. My own feeling about it is that once we have both HS and log shipping integrated and reasonably well polished, we'd have something that deserved the fabled 9.0 version number. But that's probably a year away, and we are not doing anyone a favor by not putting out 8.4 in the meantime. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 22:55 -0500, Tom Lane wrote: Silently filtering out rows according to an arbitrary security policy can break a bunch of fundamental SQL semantics, the most obvious being foreign key constraints That was exactly my reaction when I read the way it worked and I was ready to reject the patch as a result. Bruce and KaiGai provided documents that discuss the problem and it's a clearly a known issue in the security community. Specifically, it hasn't prevented Oracle from gaining security Certification and it shouldn't prevent us either. In the end it's the certification that matters here, rather than a general review of what database security is, or could be. I've seen enough to be happy that KaiGai has done a thorough job on *attempting* to address the needs of the security people. Passing security audit is the real test and I won't be beating him up if we do miss slightly. We have to try, otherwise we'll never know. My concerns are all about what it does to our code and the impacts of that. These are things we know how to check. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 10:51 -0500, Tom Lane wrote: Without an integrated and fairly high-performance log shipping capability, they are not going to find HS very compelling. Claiming otherwise is just wishful thinking. What HS will give us is same or better than the equivalent feature in latest release of Oracle. Oracle requires you to manually freeze/unfreeze the standby and so the data is never even close to being current. For us, it might be better if it was streamed, but its not a critically important issue for most users. An idle server is. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Simon Riggs si...@2ndquadrant.com writes: On Mon, 2009-01-26 at 22:55 -0500, Tom Lane wrote: Silently filtering out rows according to an arbitrary security policy can break a bunch of fundamental SQL semantics, the most obvious being foreign key constraints That was exactly my reaction when I read the way it worked and I was ready to reject the patch as a result. Bruce and KaiGai provided documents that discuss the problem and it's a clearly a known issue in the security community. Specifically, it hasn't prevented Oracle from gaining security Certification and it shouldn't prevent us either. In the end it's the certification that matters here, rather than a general review of what database security is, or could be. Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? I read them, but I suspect they are issues that can be addressed. How would any of this affect join removal, anyway? At most it would affect join removal WHEN USING SE-PostgreSQL, but I don't even see why it would affect that. We've already decided we're not overly concerned with covert channels, and the user being able to discern that a join got removed is surely no more than that. Furthermore, as covert channels go, it seems unlikely to be the one that breaks the bank. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
David Fetter da...@fetter.org writes: On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote: I don't think this is correct. I do. People literally grab my shoulder and ask when we'll have it. Do these people understand the difference between HS and a complete replication solution? Are they still as excited after you explain the difference? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 11:36 -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote: I don't think this is correct. I do. People literally grab my shoulder and ask when we'll have it. Do these people understand the difference between HS and a complete replication solution? Are they still as excited after you explain the difference? No. Because everyone I see rambling about it don't realize there is a difference. Joshua D. Drake regards, tom lane -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 11:26 -0500, Tom Lane wrote: Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? Yes, I'm sorry, I did. Will read on. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Haas robertmh...@gmail.com writes: Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? I read them, but I suspect they are issues that can be addressed. How would any of this affect join removal, anyway? It would prevent us from making optimizations that assume foreign key constraints hold; which is a performance issue not a covert-channel issue. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 11:36 -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote: I don't think this is correct. I do. People literally grab my shoulder and ask when we'll have it. Do these people understand the difference between HS and a complete replication solution? Are they still as excited after you explain the difference? Yes, I think they do. http://www.postgresql.org/community/survey.55 These people seem to understand also. Sync rep *is* important, but it opens up new classes of applications for us. As does SEP. Both of those are more speculative and harder to measure, but we've seen big impact before from this type of new feature. HS appeals to current users. Current users aren't so worried about new applications, they look forward to being able to run queries on their currently idle standby servers. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Joshua D. Drake j...@commandprompt.com writes: This is my take as well. This is very real, very scary things that are being worked on. HS should only ship after a very, very long non change cycle (meaning no significant bugs (or refactoring) found in HS patch for X period of time)... say after a full 8.5 dev cycle. I do not want to commit this patch and then have to yank it out 3 months from now. In general I'm for planning large features with the potential to break existing functionality going in the beginning of cycles. I don't think that's the same as no changes though. The reason we make changes is because they're believed to be for the better. The point in my mind is to get more people playing with the new feature in contexts that *aren't* expected by the developers. Developers are notoriously bad at testing their work no matter how diligent they are they just don't think of things they didn't anticipate when they're coding. (Which is only logical -- surely they would have just written it right the first time if they anticipated the problems...) Lastly, the last time a developer told me two weeks it was 3 months. Unless we get a written development plan that describes specifically what, when, why and how long I am severely suspect that Heikki or Simon have a clue on an actual deliverable time line (no offense guys). Well, Simon's been pretty impressively bang-on with his estimates for his *development* projects going back at least to async-commit. The *review* process, however, is inherently hard to estimate though. I doubt anyone will give Tom better than even odds on his side bet, even if that's our best estimate. Simon has been refactoring and recoding based on Heikki's suggestions as fast as he's been proposing them though. It seems the question isn't how fast Simon will get the work done so much as how many items we'll want to change before committing it. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 11:36:02AM -0500, Tom Lane wrote: David Fetter da...@fetter.org writes: On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote: I don't think this is correct. I do. People literally grab my shoulder and ask when we'll have it. Do these people understand the difference between HS and a complete replication solution? Yes, and those who don't catch on quickly. The difference between warm standby and hot is the difference between an idle resource which only consumes money to mitigate risk and a revenue-generating one. The former is for fat organizations with money to throw around, and the latter is for anybody who needs to scale reads. Are they still as excited after you explain the difference? Yes. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Brendan Jurd dire...@gmail.com writes: I think the picture has started to become more clear during the 8.4 dev cycle. Most importantly, there was much ado made about the need for powerful email integration features in previous discussions. This severely restricted our choices (possibly to zero?). I feel that the commitfest wiki has demonstrated that no such integration is required. Hardly --- one of the most critical usability fixes for the wiki was to make it relatively painless to insert links to the mail list archives (even for messages that hadn't made it there yet!). We're still gonna need that. I agree that we found out that we don't need to be able to send mail directly to the patch tracker, although perhaps cc'ing it would be a nice way to get such links installed. The other thing that is commonly thought of as email integration is the ability to generate notification email, which AFAIK the wiki does have (I haven't felt a need for it, but other people might be using that). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, 2009-01-26 at 22:55 -0500, Tom Lane wrote: Even accepting such a restriction, there's too much code in core Postgres to let anyone feel very good about keeping the core free of security leaks I see what you're saying, but we're trying to pass certification, not provide security in all cases. The security policy its implementation is part of the wall, so its straightforward to say don't do those things. Since both backups and plugins are not typically managed by unprivileged users, that seems reasonable. (And anyway, they should be using PITR :-). I'd rather see it go in now. It needs to be audited, and it might fail. If we put it in 8.5 and it still fails, we'll be in 8.6, which is far, far away and we shouldn't expect NEC to fund such a long range mission. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane t...@sss.pgh.pa.us writes: Robert Haas robertmh...@gmail.com writes: Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? I read them, but I suspect they are issues that can be addressed. How would any of this affect join removal, anyway? It would prevent us from making optimizations that assume foreign key constraints hold; which is a performance issue not a covert-channel issue. It does seem weird to simply omit records rather than throw an error and require the user to use a where clause, even if it's something like WHERE pg_accessible(tab). I wonder if we need a special kind of relational integrity trigger which requires that the privileges on a source row be a superset of the privileges on the target row. Can you even test superset on these privileges? Or are they too general for that? And would you have trouble adjusting the privileges later because giving someone access to a label would require checking every row to see if they have access to every referenced row too? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 11:49 -0500, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? I read them, but I suspect they are issues that can be addressed. How would any of this affect join removal, anyway? It would prevent us from making optimizations that assume foreign key constraints hold; which is a performance issue not a covert-channel issue. Well, only when sepostgresql = on. I would imagine Oracle has exactly the same issue. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Gregory Stark (st...@enterprisedb.com) wrote: It does seem weird to simply omit records rather than throw an error and require the user to use a where clause, even if it's something like WHERE pg_accessible(tab). It is weird from an SQL perspective, I agree with you there. On the other hand, it's what the security community is looking for, and is what's implemented by other databases (Oracle, SQL Server...) that do row-level security and security labels. Requiring a where clause or you throw an error would certainly make porting applications that depend on that mechanism somewhat difficult, and doesn't really seem like it'd gain you all that much... Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Robert Haas robertmh...@gmail.com writes: On Tue, Jan 27, 2009 at 11:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: It would prevent us from making optimizations that assume foreign key constraints hold; which is a performance issue not a covert-channel issue. Oh, I see now. That problem is going to be common to row-level DAC and SE-PostgreSQL proper. It would not surprise me if any sort of row-level access control turns out to be bad for performance, but mainly because the overhead of checking permissions on every tuple is bound to cost something. Right, but you expect that to be a small and predictable cost, say in the single-digits-percentage range. Plan optimizations that suddenly stop happening can cost you multiple orders of magnitude. And you won't soothe people by telling them that obsolete versions of Postgres would have been that slow all the time. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 11:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Yeah, people like certification, but they also like products that work. Did you stop reading before getting to my non-security-based complaints? I read them, but I suspect they are issues that can be addressed. How would any of this affect join removal, anyway? It would prevent us from making optimizations that assume foreign key constraints hold; which is a performance issue not a covert-channel issue. Oh, I see now. That problem is going to be common to row-level DAC and SE-PostgreSQL proper. It would not surprise me if any sort of row-level access control turns out to be bad for performance, but mainly because the overhead of checking permissions on every tuple is bound to cost something. If some day we have join removal and it has to be disabled when row-level access control is turned on, those users will be no worse off than they are today: no join removal. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 12:41:36PM -0500, Stephen Frost wrote: * Gregory Stark (st...@enterprisedb.com) wrote: It does seem weird to simply omit records rather than throw an error and require the user to use a where clause, even if it's something like WHERE pg_accessible(tab). […] do row-level security and security labels. Requiring a where clause or you throw an error would certainly make porting applications that depend on that mechanism somewhat difficult, and doesn't really seem like it'd gain you all that much... Throwing an error would entail a side-channel leak that would not be acceptable to the security community, I bet. That said, I have reservations, along the lines of Peter E's, that the early design-level objections to the approach were never answered. I certainly never got any real answer to questions I asked, for what it's worth. I will note that I tried to have a look at the literature on this topic. As I started to read, it became obvious that it was copious, but pretty well-determined. What bothered me most about the answers I got was that there never seemed to be an answer to please outline the design principles except for it's what SE-Linux does. The OS-level control rules seemed to me to be totally foreign to the database world, precisely because ACID is a problem in databases in a way it isn't for filesystems under the traditional UNIX model. I formed the impression -- only an impression, mind, that there was a poor fit between SE-Linux and database systems, and that the proponents had decided that enough caulk (in the form of don't do that) would seal the gap. I haven't (obviously) been paying much attention to this topic since, but I detect something of the same sort of response in the recent discussion. Maybe the goal isn't explicit enough. If the goal is compliance with some set of well-defined tests, what are they? If the goal is buzzword compliance, what are the tests of that (to the extent there ever are some)? If the goal is just security enhancement, I confess that I am still unable to understand the definitions of security and enhancement such that I think we have some operationalization of what the patch is supposed to provide. I know there are people who think this is cool. I guess, if I were running the circus, I'd want to know what's cool about it, and why. Then maybe the project would be in a position to understand whether that kind of cool is the way it wants to be. But without a clear problem statement, and a roadmap of how the patches solve the problem, I'm at a loss. And last I checked (which was, admittedly, not today), the project pages didn't have that information. A -- Andrew Sullivan a...@crankycanuck.ca -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Stephen Frost wrote: * Gregory Stark (st...@enterprisedb.com) wrote: It does seem weird to simply omit records rather than throw an error and require the user to use a where clause, even if it's something like WHERE pg_accessible(tab). The idea is for the level of informations security we're talking about, someone with limited permissions not only isn't allowed to know certain data, they're not allowed to know certain data *exists*. Within the SELinux framework, this is accomplished by hiding files you don't have permission to see, not merely denying access to them. The presumption is that if you know the data exists but can't access it directly, you'll use indirect methods to derive what it is. But if you don't even know it exists, then you won't look for it. There's a level above that which I don't think SEPostgres implements, which is data substitution, in which you see different data according to what security level you are. While this may seem insane for a business application, for military-support applications it makes some sense. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Josh Berkus j...@agliodbs.com writes: Stephen Frost wrote: It does seem weird to simply omit records rather than throw an error The presumption is that if you know the data exists but can't access it directly, you'll use indirect methods to derive what it is. But if you don't even know it exists, then you won't look for it. Right, which is why it's bad for something like a foreign key constraint to expose the fact that the row does exist after all. There's a level above that which I don't think SEPostgres implements, which is data substitution, in which you see different data according to what security level you are. While this may seem insane for a business application, for military-support applications it makes some sense. I think it might be possible to build such a thing using views, but I agree that the patch doesn't give it to you for free. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
