Re: [HACKERS] pg_upgrade project status
Tom Lane wrote: Robert Haas writes: That implies a fairly robust and configurable system for adding to and rewriting system catalogs, which today we haven't got. And we won't ever have, because it's unnecessary and would be impossibly complex. We know how to do the catalog update: basically, dump, initdb, reload, then move the data in. There are some corner case issues like how to preserve toast table OIDs, but the idea that we are going to invent a special process for each catalog change is just not reasonable. Right, the dump+initdb+reload approach works quite well in both pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is supporting dropped columns, and maybe there's something else, but it's fairly robust and works across any versions. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote: > Marko Kreen wrote: > > On 1/27/09, Peter Eisentraut wrote: > >> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: > >> > Such app already exists: > >> > > >> > http://ozlabs.org/~jk/projects/patchwork/ > >> > > >> > So it's a matter of just setting it up. > >> > >> I was in fact in the process of setting that up just now. :-) > > > > Nice to know. :) I feel that even if we decide to do our own > > solution it would be good to try existing solution first. > > IIRC, we already installed and tried this a while ago. I don't remember > exactly what it failed on, but there was something pretty clear. But > maybe it's been fixed by now. Details? I find no public record of this. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [OT] there is a way to extract a previously applied patch?
On Wednesday 28 January 2009 07:05:17 Robert Haas wrote: > Easy to do with git. > > $ git log > > Use "/Allow AS" to find the commit. Oh, there it is. > > $ git log -1 -p 4f08d8492598a518f803d8c2979b39f0050b0f8d > allow-as.patch > > Attached for your convenience. :-) More compactly, git show $HASH also works. TIMTOWTDI, apparently. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [OT] there is a way to extract a previously applied patch?
Robert Haas wrote: Easy to do with git. $ git log Use "/Allow AS" to find the commit. Oh, there it is. $ git log -1 -p 4f08d8492598a518f803d8c2979b39f0050b0f8d > allow-as.patch Attached for your convenience. :-) Please don't post patches for private use on the list. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 21:07:48 Tom Lane wrote: > Robert Treat writes: > > The more I think about it, the more I feel that where we failed for 8.3 > > was not having a short 8.4 cycle lined up, which would give more freedom > > to bump patches to the next release. > > Heh. The reason we wanted a short 8.3 cycle was so we could push out > patches that had been held over from 8.2. And the reason that didn't work was because when we got to feature freeze, we once again had several large, complex patches which people didn't want to push for the long 8.4 cycle. (But note: people are willing to push patches when they believe the wait time won't be excessive for eventual inclusion) > We are going to have exactly > no credibility if we tell Simon et al "we're pushing these patches to > 8.5, but don't worry, it'll be a short release cycle". > The other options being we stall 8.4 indefinatly waiting for HS (which, honestly I am comfortable with), or his patches get pushed and he doesnt get them for another 14 months. Seems to me our credibility isn't really even a factor here. Right now I'm really trying to figure out how to solve this problem for the long term. If we say up front now that the next 2 cycles are short cycles, then I think people will be more willing to push patches come end-of-8.5 (and let's not pretend we're not going to have this same argument over streaming replication or synchronous replay or merge command or whatever hot feature is almost ready at that time) > I think the best thing we could do overall is to set release dates and > stick to them. If your patch is not ready, well, at least it will get > out in a defined amount of time. Right now, the *real* problem with it > being pushed to the next release is you don't know how successful some > other guy will be at persuading us to delay the next release. > I wont argue that setting release dates and sticking to them is a bad idea, but that extra month at the end that that occures due to feature lobbying doesn't strike me as the straw breaking the camels back, it's the 12-14 months in front of that people don't want to wait through. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Andrew Dunstan wrote: Magnus Hagander wrote: Andrew Dunstan wrote: I've installed drmingw to handle exceptions instead, so we'll see if that gives us useful info. If not, I'll see what I can do with gdb. Hadn't heard of drwmingw, I see how that can be useful :-) report from DrMingw is below Further data point: The suspect patch is quite definitely the source of the problem. I undid the configure changes and surrounded the additions to port/win32.h with #ifdef WIN32_ONLY_COMPILER ... #endif. Result: the problem disappeared, and "make check" completed perfectly. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, 27 Jan 2009, Robert Haas wrote:
That implies a fairly robust and configurable system for adding to and
rewriting system catalogs, which today we haven't got.
Right, and designing such a system requires a fairly deep analysis of the
catalog changes that have historically shown up, to make sure you've got
something that will handle them. The first step here that bears some
immediate fruit would be to sit down and analyze everything that changed
in the catalog from 8.3 to 8.4. Then you can do two things. You can
confirm whether the path used by pg_upgrade really can be expected to
work. You could also then talk intelligently about what a design for a
catalog rewriting system would look like, using "something that could have
handled all of these" as your spec. Then it's onto worrying about page
format stuff, which you can already get a handle on by looking at 8.2->8.3
under the same lens.
Tom even generated a helpful starting list of things to chase down a few
months ago:
http://archives.postgresql.org/pgsql-hackers/2008-09/msg00451.php
As someone who has been investigating this particular path, the part that
troubles me is that you'd have to pushing a new and completely
non-glamorous overhead burden ("create an in-place implementation for your
catalog change") on people who don't necessarily care about that
particular goal, which would create considerable new friction for patch
submission that required a catversion bump. Not the sort of boring work
volunteers on a project are traditionally good at, and you can expect any
developers who aren't managing databases too large to dump/reload to
scream about how it will slow the advance of the project.
--
* Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [OT] there is a way to extract a previously applied patch?
On Wed, Jan 28, 2009 at 12:05 AM, Robert Haas wrote: > Easy to do with git. > > $ git log > > Use "/Allow AS" to find the commit. Oh, there it is. > > $ git log -1 -p 4f08d8492598a518f803d8c2979b39f0050b0f8d > allow-as.patch > > Attached for your convenience. :-) > thanks, i will have to start to learn GIT -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [OT] there is a way to extract a previously applied patch?
Easy to do with git.
$ git log
Use "/Allow AS" to find the commit. Oh, there it is.
$ git log -1 -p 4f08d8492598a518f803d8c2979b39f0050b0f8d > allow-as.patch
Attached for your convenience. :-)
...Robert
commit 4f08d8492598a518f803d8c2979b39f0050b0f8d
Author: Tom Lane
Date: Fri Feb 15 22:17:06 2008 +
Allow AS to be omitted when specifying an output column name in SELECT
(or RETURNING), but only when the output name is not any SQL keyword.
This seems as close as we can get to the standard's syntax without a
great deal of thrashing. Original patch by Hiroshi Saito, amended by me.
diff --git a/doc/src/sgml/queries.sgml b/doc/src/sgml/queries.sgml
index 0f347be..0d237ab 100644
--- a/doc/src/sgml/queries.sgml
+++ b/doc/src/sgml/queries.sgml
@@ -491,7 +491,7 @@ FROM table_reference AS alias
FROM table_reference alias
- The AS key word is noise.
+ The AS key word is optional noise.
alias can be any identifier.
@@ -1040,13 +1040,32 @@ SELECT a AS value, b + c AS sum FROM ...
-If no output column name is specified using AS, the system assigns a
-default name. For simple column references, this is the name of the
-referenced column. For function
+If no output column name is specified using AS,
+the system assigns a default column name. For simple column references,
+this is the name of the referenced column. For function
calls, this is the name of the function. For complex expressions,
the system will generate a generic name.
+
+The AS keyword is optional, but only if the new column
+name does not match any
+PostgreSQL keyword (see ). To avoid an accidental match to
+a keyword, you can double-quote the column name. For example,
+VALUE is a keyword, so this does not work:
+
+SELECT a value, b + c AS sum FROM ...
+
+but this does:
+
+SELECT a "value", b + c AS sum FROM ...
+
+For protection against possible
+future keyword additions, it is recommended that you always either
+write AS or double-quote the output column name.
+
+
The naming of output columns here is different from that done in
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
index ccc959e..f838814 100644
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -23,7 +23,7 @@ PostgreSQL documentation
DELETE FROM [ ONLY ] table [ [ AS ] alias ]
[ USING usinglist ]
[ WHERE condition | WHERE CURRENT OF cursor_name ]
-[ RETURNING * | output_expression [ AS output_name ] [, ...] ]
+[ RETURNING * | output_expression [ [ AS ] output_name ] [, ...] ]
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
index 154ea98..aa3c26c 100644
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -22,7 +22,7 @@ PostgreSQL documentation
INSERT INTO table [ ( column [, ...] ) ]
{ DEFAULT VALUES | VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query }
-[ RETURNING * | output_expression [ AS output_name ] [, ...] ]
+[ RETURNING * | output_expression [ [ AS ] output_name ] [, ...] ]
diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml
index 33fc71e..e463d15 100644
--- a/doc/src/sgml/ref/select.sgml
+++ b/doc/src/sgml/ref/select.sgml
@@ -21,7 +21,7 @@ PostgreSQL documentation
SELECT [ ALL | DISTINCT [ ON ( expression [, ...] ) ] ]
-* | expression [ AS output_name ] [, ...]
+* | expression [ [ AS ] output_name ] [, ...]
[ FROM from_item [, ...] ]
[ WHERE condition ]
[ GROUP BY expression [, ...] ]
@@ -477,23 +477,45 @@ HAVING condition
SELECT and FROM) specifies expressions
that form the output rows of the SELECT
statement. The expressions can (and usually do) refer to columns
-computed in the FROM clause. Using the clause
-AS output_name, another
-name can be specified for an output column. This name is
-primarily used to label the column for display. It can also be
-used to refer to the column's value in ORDER BY and
-GROUP BY clauses, but not in the WHERE or
-HAVING clauses; there you must write out the
-expression instead.
+computed in the FROM clause.
+
+
+
+Just as in a table, every output column of a SELECT
+has a name. In a simple SELECT this name is just
+used to label the column for display, but when the SELECT
+is a sub-query of a larger query, the name is seen by the larger query
+as the column name of the virtual table produced by the sub-query.
+To specify the name to use for an output column, write
+AS output_name
+after the column's expression. (You can omit AS,
+but only if the desired output name does not match any
+PostgreSQL keyword (see ). For protection against possible
+future keyword additions, it is recommended that you always either
+write AS or double-quote the output name.)
+If you do
Re: [HACKERS] 8.4 release planning
Greg Smith wrote: Where I suspect this is all is going to settle down into is that if 1) the SE GUC is on and 2) one of the tables in a join has rows filtered, then you can expect that a) it's possible that the result will leak information, which certainly need to be documented, and b) the optimizations Tom mentioned that "assume foreign key constraints hold" will not be possible to implement, so performance will suck compared to a similar situation in an unsecured environment. c) security feature gives the optimizer a hint "don't optimize out this table, please!" via a security hook. I think it is a quite reasonable approach, as I noted in another message. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Jaime Casanova wrote: On Tue, Jan 27, 2009 at 2:18 PM, Tom Lane wrote: This seems to me to be exactly parallel to deciding that SELinux should control only table/column permissions within SQL; an approach that would be enormously less controversial, less expensive, and more reliable than what SEPostgres tries to do. seems that the controversial part of sepgsql is row level permissions, can we try to commit (obviously with good revision and test) the table/column privileges part of that patch? Actually, it is already done. http://code.google.com/p/sepgsql/source/browse/trunk/sepgsql/src/backend/utils/misc/guc.c#1242 http://code.google.com/p/sepgsql/source/browse/trunk/sepgsql/src/backend/security/sepgsql/permissions.c#518 Its original purpose is different, to reduce storage consumption. But it can be a point of compromise. See, http://archives.postgresql.org/message-id/492691a8.8030...@ak.jp.nec.com Is it a reasonable option to allow users to turn on/off? I can add a documentation about its background and tradeoffs, for user's correct decision. Thanks, that is still a step on the direction of full centralized security management on the system... let the row level privileges part for 8.5, that way the patch will be smaller now and then... remember, postponed is not rejected is just a way to give more time to think (WITH patch comes from the prior release cycle and was committed in this release), not to think about one scenario but about all possible scenarios in a more wide audience -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: > The flaw in that argument is that as you are doing it, the > de-optimization only happens on queries that actually need the behavior. > As the SEPostgres patch is constructed, the planner could *never* trust > an FK for optimization since it would have no way to know whether row > level permissions might be present (perhaps only for some rows) at > execution time. Is the "never" is really correct? In the following case, it is necessary not to apply optimization: - SE-PostgreSQL is working, and its row-level access controls are available (sepostgresql_row_level=on). - Row-level ACL is available on the target relation. It is controlable via table option. So, it is necessary to add a new security hook to give a hint the optimizer. Sorry, I overlooked this optimization. But is is not a fundamental design issue. PGACE already has a hook to give a hint to optimizer. It will be a similar one. See, pgaceAllowFunctionInlined(...); http://code.google.com/p/sepgsql/source/browse/trunk/sepgsql/src/backend/security/pgaceHooks.c#948 Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Robert Haas writes: > That implies a fairly robust and configurable system for adding to and > rewriting system catalogs, which today we haven't got. And we won't ever have, because it's unnecessary and would be impossibly complex. We know how to do the catalog update: basically, dump, initdb, reload, then move the data in. There are some corner case issues like how to preserve toast table OIDs, but the idea that we are going to invent a special process for each catalog change is just not reasonable. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] [OT] there is a way to extract a previously applied patch?
Hi, I want to put the patch for "Allow AS to be omitted when specifying an output column name" [1] in 8.3.5 because i'm doing a migration and need that (for avoiding large changes in application) before May 1st, because is only 2 .y files i tried to just copy 'n paste but the links doesn't work. Other idea is: is there anyway to extract patches from the repository? The reason i want that is that the commit message let me thing that the patch is not exactly the same that Hiroshi Saito sent [1] http://archives.postgresql.org/pgsql-committers/2008-02/msg00172.php -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Wed, 28 Jan 2009, KaiGai Kohei wrote: It shows a fact that not negligible number of users accept row-level granularity, even if it has covert channels. From my read of the examples that Chad provided, keeping the existence of things secret from complete outsiders doesn't look like the prime concern. There's plenty of these applications floating around where everyone involved is cleared, but to different levels and projects. The person working on a "secret" project knows perfectly well that there are also "top secret" projects floating around they aren't cleared for, and that's OK. They'll probably detect their existence by the doors they're not allowed through long before they notice that database rows are missing. If you're able to sit in front of a computer that's capable of even reaching this information but aren't supposed to be anywhere near it, that means there's been a physical security breach. Where I suspect this is all is going to settle down into is that if 1) the SE GUC is on and 2) one of the tables in a join has rows filtered, then you can expect that a) it's possible that the result will leak information, which certainly need to be documented, and b) the optimizations Tom mentioned that "assume foreign key constraints hold" will not be possible to implement, so performance will suck compared to a similar situation in an unsecured environment. And all that may very well be just fine as far as the people wanting to build applications with SEPostgreSQL are concerned. It will just hint toward using a schema design with table-level controls instead if you care about high performance on that style of join. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
> for getting 8.4->8.5 upgrade into place. You should first work on an > update process that supports catalog changes, and get that committed. > Once that's in place you'll have enough political capital to prevent > changes in user data layout, and then we'd start to think about schemes > like space reservation that could relax that ground rule. I agree that catalog changes are the best way to start. And I also think it's important that this logic get written in C, primarily so that it gets deeply integrated into core. When a patch submitter adds a function like array_length and puts in the markup for the BKI, they should also be responsible for adding the function to the list of changes that the UIP code needs to make when upgrading from the catversion of the previous release to the current catversion. If upgrade in place consists of Zdenek running around during every beta cycle trying to figure out everything that happened to get done during that dev cycle, it's not going to work (especially if Zdenek gets hit by a bus). That implies a fairly robust and configurable system for adding to and rewriting system catalogs, which today we haven't got. Once we do have it, we can think about how to handle things like page layout bumps and toast chunk changes. Frankly, with things as they are today, there's no way I'm using pg_upgrade on my production system. Even if the whole thing gets rewritten in perl, it's an ex post facto attempt to catch up with a gazillion changes that got made without any regard to whether they'd work with pg_upgrade. It seems very difficult to be confident that this will be 100% correct... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 2:18 PM, Tom Lane wrote: > > This seems to me to be exactly parallel to deciding that SELinux should > control only table/column permissions within SQL; an approach that would > be enormously less controversial, less expensive, and more reliable than > what SEPostgres tries to do. > seems that the controversial part of sepgsql is row level permissions, can we try to commit (obviously with good revision and test) the table/column privileges part of that patch? that is still a step on the direction of full centralized security management on the system... let the row level privileges part for 8.5, that way the patch will be smaller now and then... remember, postponed is not rejected is just a way to give more time to think (WITH patch comes from the prior release cycle and was committed in this release), not to think about one scenario but about all possible scenarios in a more wide audience -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
> I think the best thing we could do overall is to set release dates and > stick to them. If your patch is not ready, well, at least it will get > out in a defined amount of time. Right now, the *real* problem with it > being pushed to the next release is you don't know how successful some > other guy will be at persuading us to delay the next release. +1, LOL. Let's not forget that we've already got CTE, window functions, partial vacuums, and column-level permissions, all of which are major features that should benefit a lot of people. I hope Hot Standby gets committed but even if it doesn't, I'm still going to get a lot of benefit out of this release, so I'd like it to happen on some sort of reasonable time scale. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
>> I thought we pretty much had agreement that space reservation was not >> a good solution to anything, although I admit I'm not quite clear on >> what alternative was being proposed. > > Maybe I miss something, but space reservation was selected as a best > way. Please, Could you point me related mailing thread? Hmm, maybe I misrepresented it slightly. But I did find these: http://archives.postgresql.org/message-id/49425d07.6030...@enterprisedb.com http://archives.postgresql.org/message-id/4948d567.4060...@enterprisedb.com ...and just today: http://archives.postgresql.org/message-id/497f5b7d.90...@enterprisedb.com ...which I think are at least an argument against this patch, if not a total refutation of the concept of space reservation in general. Personally, I think you're both barking up the wrong tree. If the amount of space that we need to reserve is more complicated than X bytes per page or per tuple, then your approach will fail. But it doesn't have to be a lot more complicated than that before Heikki's approach of back-patching the space reservation code will fail too. I foresee vehement arguments about whether the space reservation code for feature Y is too complex to back-patch, with the argument going something like this: UIP Dude: If we don't back-patch this code, we won't be able to upgrade-in-place for this release. That's a huge problem, man! Conservative Guy: Yeah, but if we do, and we fry somebody's database, that's a lot worse than not being able to upgrade-in-place. UIP Dude: I don't think that's going to happen. This code is pretty solid. Conservative Guy: You can't prove it doesn't have bugs. Conservative Guy has a point, but the real issue is that UIP Dude is making extra work for himself by splitting the work of upgrading the system between release N and release N+1. Sure, with enough work, it will be possible to test the back branch enough to convince yourself that there are no horrible bugs, and it will also be possible to test that CVS HEAD can complete the upgrade, but now you have to exhaustively test TWO releases both independently and for compatibility with each other. It's not my project, but I'd rather walk through fire ants. I just can't imagine that it will be easier to change the behavior of the old release to be compatible with what the new release needs than visca versa. Even if the new release has to reshuffle data between pages or stand on its head and wrestle an epiletic porcupine, I still think it will not be as bad as putting the space reservation code in the old release and the new page format in the new release. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Joshua Brindle wrote:
Stephen Frost wrote:
* Joshua Brindle (met...@manicmethod.com) wrote:
They are separate. If you look at the patches you'll see a pgace
part, this is where the core interfaces to the security backends, and
you'll see a rowacl backend and an sepgsql backend.
Right, guess it wasn't clear to me that the PGACE bits for row-level
access control could be used independently of SELinux (and maybe even on
systems that don't have SELinux..?).
Sure, if you look at pgaceHooks.c you'll see:
It is basically possible to implement something like "PostgreSQL
Label Security" on PGACE framework.
But I don't want to discuss it now, because it surely burst
SE-PostgreSQL until v8.4 beta.
If desired, I'll queue it my todo list next to SE-PostgreSQL...
bool
pgaceExecScan(Scan *scan, Relation rel, TupleTableSlot *slot)
{
/* Hardwired DAC checks */
if (!rowaclExecScan(scan, rel, slot))
return false;
switch (pgace_feature)
{
#ifdef HAVE_SELINUX
case PGACE_FEATURE_SELINUX:
if (sepgsqlIsEnabled())
return sepgsqlExecScan(scan, rel, slot);
break;
#endif
default:
break;
}
return true;
}
Notice the rowacl call outside of the HAVE_SELINUX ifdefs
FYI:
In the earlier version, these are mutually exclusive, so we could
not apply SE-PostgreSQL, when a binary is built with RowAcl feature.
However, Bruce Momjian suggested it is not proper manner in
PostgreSQL, because it intend to wrap all available features
into a single binary due to packaging benefit, and all the
available options should be configured by runtime.
In addition, IIRC, Peter E suggested it is not symmetrical
that we cannot apply both of DAC and MAC on tuples simultaneously,
although SE-PostgreSQL applies MAC on tables/columns which
PostgreSQL has DAC features on.
So, I add a support simultaneous DAC&MAC.
Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer writes: > Tom Lane wrote: >> I think the best thing we could do overall is to set release dates and >> stick to them. > On the other hand, we might be better throwing out release dates > and releasing at the end of any Commit Fest where there is enough > demand/interest. > Then we could release 8.4 now, with few objections since people would > know that if Hot Standby has enough demand it could be released within one > commitfest of it being in good shape. Huh? The people who'd been working with the expectation of completing their patch in the *next* fest would scream loud and long at having the release schedule pulled out from under them. Any development plan with more than a two-month time horizon would turn into a game of schedule russian roulette. The only way this works without having a lot of pissed-off developers is if we actually release a lot more often, which is not going to fly. We don't have the manpower to manage that, and we don't have users that want it, either. (They might like it better if we had a better upgrade-in-place story, but ...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: > Joshua Brindle writes: >> Tom Lane wrote: >>> Right, which is why it's bad for something like a foreign key constraint >>> to expose the fact that the row does exist after all. > >> Once again, this is not an issue for us. > > Yes it is an issue; claiming it isn't doesn't make it so. You just > stated, in effect, that you don't implement data hiding in the > filesystem because it would break standard Unix filesystem semantics. > How is that consistent with your opinion that it's okay to break > standard SQL semantics in order to implement data hiding in a database? > > The question of whether there is a covert channel is only a small part > of my complaint here. If it's the judgement of security experts that > that's an acceptable thing, that's fine, it's their turf. But SQL > behavior is my turf, and I'm not happy with discarding fundamental > semantic properties. Do you forget a GUC option: sepostgresql_row_leve = on/off ? It was a requirement from Simon Riggs at Oct 2008, IIRC. Its original purpose is to reduce storage consumption due to the security label of each tuples, however, it can allow users to choose the granularity of access controls in finally. I can understand you have a complaint about covert channels via PK/FK due to the row-level granularity. However, in other hand, we can already see a commercial producet which provides row-level access controls without any cares about covert channels. It shows a fact that not negligible number of users accept row-level granularity, even if it has covert channels. What is needed is an explicit documentation about covert channels and providing a few options to users. It is not a reasonable stance to deny anything due to a single item which is acceptable some of people, at least. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] log_duration_sample config option patch
Timo Savola escreveu: > Hello. The attached patch has made it more feasible for us to gather > profiling data on a production system for analysis with pgFouine. It > has been written against PostgreSQL 8.3.5 and tested on Linux. Comments > welcome. > IIRC pgFouine shows exact percentage of query by type; this new GUC implies we can not rely on it anymore. Also, you could skip us from logging long (and bad) running queries. Another statistics data in pgFouine will suffer from the same problem. I see your point in reducing amount of log data but I think it will limit the tool usability (you can always start/stop statement collection at runtime). -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: > Heh. The reason we wanted a short 8.3 cycle was so we could push out > patches that had been held over from 8.2. We are going to have exactly > no credibility if we tell Simon et al "we're pushing these patches to > 8.5, but don't worry, it'll be a short release cycle". > > I think the best thing we could do overall is to set release dates and > stick to them. On the other hand, we might be better throwing out release dates and releasing at the end of any Commit Fest where there is enough demand/interest. Then we could release 8.4 now, with few objections since people would know that if Hot Standby has enough demand it could be released within one commitfest of it being in good shape. Likewise the SE* guys would be aware that if they want that patch to drive a release, they'd need to round up more visible demand. Heck, 8.4 could have been released a whole commitfest ago if enough people think FSM is the killer feature in that one. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Andrew Sullivan wrote: On Tue, Jan 27, 2009 at 12:41:36PM -0500, Stephen Frost wrote: * Gregory Stark (st...@enterprisedb.com) wrote: It does seem weird to simply omit records rather than throw an error and require the user to use a where clause, even if it's something like WHERE pg_accessible(tab). […] do row-level security and security labels. Requiring a where clause or you throw an error would certainly make porting applications that depend on that mechanism somewhat difficult, and doesn't really seem like it'd gain you all that much... Throwing an error would entail a side-channel leak that would not be acceptable to the security community, I bet. That said, I have reservations, along the lines of Peter E's, that the early design-level objections to the approach were never answered. I certainly never got any real answer to questions I asked, for what it's worth. I will note that I tried to have a look at the literature on this topic. As I started to read, it became obvious that it was copious, but pretty well-determined. What bothered me most about the answers I got was that there never seemed to be an answer to "please outline the design principles" except for "it's what SE-Linux does". The OS-level control rules seemed to me to be totally foreign to the database world, precisely because ACID is a problem in databases in a way it isn't for filesystems under the traditional UNIX model. I formed the impression -- only an impression, mind, that there was a poor fit between SE-Linux and database systems, and that the proponents had decided that enough caulk (in the form of "don't do that") would seal the gap. As I noted before, there is a symmetrical structure between OS and DBMS. In operating system, a process accesses objects (like file, socket, ...) managed by operating system via system calls. Its security system (filesystem permission, SELinux, ...) acquires the request and applies its access control rules. In DBMS, a client accesses database objects managed by DBMS via SQL queries. Its security system (Database ACL, SE-PostgreSQL, ...) aquires the request and applies its access control rules. We have similar structures everywhere, not only DBMS and OS. What we should pay attention is a subject entity accesses via a method provided by object managers (DBMS, OS, ...), and object managers apply its rules to decide either "allowed" or "denied" on acquired request. I haven't (obviously) been paying much attention to this topic since, but I detect something of the same sort of response in the recent discussion. Maybe the goal isn't explicit enough. If the goal is compliance with some set of well-defined tests, what are they? If the goal is buzzword compliance, what are the tests of that (to the extent there ever are some)? If the goal is just "security enhancement", I confess that I am still unable to understand the definitions of "security" and "enhancement" such that I think we have some operationalization of what the patch is supposed to provide. The most significant feature is centralized access control policy between OS and DBMS. Did you attend PGcon2008? I talked here we should consider the value of information asset is independent from the way to store them. Needless to say, the value of information asset is decided by its contents. If your credit card number is recorded on a paper, do you think it has lesser value than recorded on database? Thus, from the viewpoint of security, we need to consider the way to apply unified centralized access controls. SELinux works as "security server". It enables to provide a centralized access control decision any other object managers (like, Linux kernel, X-window, PostgreSQL, ...). It is quite consistent because of common security policy and common clearance of entities. It finally enables to apply centralized access control policy on whole of application stack. Please note that 95% of attacks in 2008 targeted to web system, so it gives a nightmare for security folks. Thanks, I know there are people who think this is cool. I guess, if I were running the circus, I'd want to know what's cool about it, and why. Then maybe the project would be in a position to understand whether that kind of cool is the way it wants to be. But without a clear problem statement, and a roadmap of how the patches solve the problem, I'm at a loss. And last I checked (which was, admittedly, not today), the project pages didn't have that information. A -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 21:07 -0500, Tom Lane wrote: > Robert Treat writes: > > The more I think about it, the more I feel that where we failed for 8.3 was > > not having a short 8.4 cycle lined up, which would give more freedom to > > bump > > patches to the next release. > > Heh. The reason we wanted a short 8.3 cycle was so we could push out > patches that had been held over from 8.2. We are going to have exactly > no credibility if we tell Simon et al "we're pushing these patches to > 8.5, but don't worry, it'll be a short release cycle". > > I think the best thing we could do overall is to set release dates and > stick to them. If your patch is not ready, well, at least it will get > out in a defined amount of time. Right now, the *real* problem with it > being pushed to the next release is you don't know how successful some > other guy will be at persuading us to delay the next release. +1 Joshua D. Drake > > regards, tom lane > -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat writes: > The more I think about it, the more I feel that where we failed for 8.3 was > not having a short 8.4 cycle lined up, which would give more freedom to bump > patches to the next release. Heh. The reason we wanted a short 8.3 cycle was so we could push out patches that had been held over from 8.2. We are going to have exactly no credibility if we tell Simon et al "we're pushing these patches to 8.5, but don't worry, it'll be a short release cycle". I think the best thing we could do overall is to set release dates and stick to them. If your patch is not ready, well, at least it will get out in a defined amount of time. Right now, the *real* problem with it being pushed to the next release is you don't know how successful some other guy will be at persuading us to delay the next release. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/27/09 6:57 PM, "Greg Smith" wrote: > On Tue, 27 Jan 2009, Chad Sellers wrote: > >> I'll speak to this a bit (Josh is also a Tresys employee). I can't say who >> my customers are, but I can speak to their needs. They really need row-level >> mandatory access controls (so both). > > From the perspective of what this would buy as far as this feature being a > PostgreSQL advocacy point, one of the questions Tom asked about a bit > upthread is still a bit hazy here. There are commercial database > offerings selling into the "trusted" space already. While the use-cases > you describe make perfect sense, I don't think it's clear to everyone yet > if there's a unique draw to a PostgreSQL + selinux solution that the class > of customers you're talking about would prefer it to purchasing one of > those products. Is the cost savings the main driver here, or is there > something else about a secure LAPP stack that makes it particularly > compelling? > Cost savings may be a driver, but it's certainly not the main driver. A problem with the current systems (e.g. Oracle) is that they largely operate in their own world. Oracle Label Security labels do not extend outside the database to the rest of the system, which makes it impossible to build certain things. For instance, you can't really build a trusted LAPP stack. With SE-PostgreSQL, the label used for access control is bound by the kernel and used by it and the other components of the stack (apache, PostgreSQL). We can even extend all the way to the end system using labeled IPSec. Another major drawback to the currently available trusted databases is that they have hard-coded policy rules rather than flexible ones. While these hard-coded rules may be appropriate in some scenarios, they often end up being impractical. For instance, any sort of pipeline (like that of my example 3) is difficult if not impossible to implement on most of these hard-coded systems. Additionally, changing the label in most of these systems requires a privilege that lets you bypass the policy rules, while a flexible Type Enforcement system (like SE-PostgreSQL) allows you to specify exactly how relabels should be allowed within the policy. I'll be happy to provide more details where required. Hopefully that provided a little light. Thanks, Chad Sellers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Joshua Brindle wrote:
Tom Lane wrote:
Ron Mayer writes:
It seems to me that there are two different standards to which this
feature
might be held.
Is the goal
a) SEPostgres can provide useful rules to add security to some
specific applications so long as you're careful to avoid crafting
policies that produce bizarre behaviors (like avoiding restricing
access to foreign key data you might need). On the other hand it
gives you enough rope to hang yourself and produce weird results
that don't make sense from a SQL standard point of view if you
aren't careful matching the SEPostgres rules with your apps.
or
b) SEPostgreSQL should only give enough rope that you can not
craft rules that produce unexpected behavior from a SQL point
of view; and that it would be bad if one can produce SEPostgres
policies that produce unexpected SQL behavior.
With my other hat on (the red one) what I'm concerned about is whether
this patch will ever produce a feature that I could turn on in the
standard Red Hat/Fedora build of Postgres. Right at the moment it seems
that the potential performance hit, for users who are *not using*
SEPostgres but merely have to use a build in which it is present,
might be bad enough to guarantee that that will never happen.
According to the comments in security/sepgsql/core.c:
/*
* sepgsqlIsEnabled
*
* This function returns the state of SE-PostgreSQL when PGACE hooks
* are invoked, to prevent to call sepgsql() functions when
* SE-PostgreSQL is disabled.
*
* We can config the state of SE-PostgreSQL in $PGDATA/postgresql.conf.
* The GUC option "sepostgresql" can have the following four parameter.
*
* - default: It always follows the in-kernel SELinux state. When it
*works in Enforcing mode, SE-PostgreSQL also works in
*Enforcing mode. Changes of in-kernel state are delivered
*to userspace SE-PostgreSQL soon, and SELinux state
*monitoring process updates it rapidly.
* - enforcing : It always works in Enforcing mode. In-kernel SELinux
*has to be enabled.
* - permissive : It always works in Permissive mode. In-kernel SELinux
*has to be enabled.
* - disabled : It disables SE-PostgreSQL feature. It works as if
*original PostgreSQL
*/
and in the hooks there is a pgace_feature that turns off the checks:
void
pgaceGramAlterRelation(Relation rel, HeapTuple tuple, DefElem *defel)
{
switch (pgace_feature)
{
#ifdef HAVE_SELINUX
case PGACE_FEATURE_SELINUX:
if (sepgsqlIsEnabled())
{
sepgsqlGramAlterRelation(rel, tuple, defel);
return;
}
break;
#endif
default:
break;
}
if (defel)
ereport(ERROR,
(errcode(ERRCODE_PGACE_ERROR),
errmsg("unable to set security
attribute of table "
"via ALTER TABLE")));
}
So the pgace_feature turns off the backend call, there is an extra
function call, and a branch but that shouldn't cause the kind of
performance degradation you are talking about.
This hook is only available when an enhanced security feature is
enabled, so I injected ereport() at the tail.
(It is invoked on ALTER TABLE ... SECURITY_LABEL = '...';)
However, most of hooks do nothing or don't change existing behavior
when enhanced security is disabled.
So, I can't understand why it gives adverse affects in performances.
I can admit it needs additional steps to invoke empty functions at least.
However, using "static inline" was arguable in the previous discussion
due to the GCC dependency.
Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 19:04:49 Tom Lane wrote: > Robert Treat writes: > > On Tuesday 27 January 2009 10:34:59 Joshua D. Drake wrote: > >> We have tried the short release cycle before, it was called 8.2. It > >> fails, remarkably. > > > > I think this is a bit of revisionsit history. > > JD got the release number wrong, it was 8.3, but otherwise there's no > revisionism involved: > http://archives.postgresql.org/pgsql-hackers/2006-12/msg00786.php > The revisionism was that of "remarkable failure". That was our shortest release cycle in the modern era. And it didn't have the advantage of the commitfest process. But I think what is important here is to recognize why it didn't work. Once again we ended up with large, complex features (HOT, tsearch) that people didn't want to wait 14 months to see if they missed the 8.3 release. And yes, most of these same arguements were raised then... "full text search is killer feature", "whole applications are waiting for in-core full text search", "hot will give allow existing customers to use postgres on a whole new level", "not fair to push back patches so long when developers followed the rules", "sponsors wont want to pay for features they wont see for years", "developers dont want to wait so long to see features committed", and on and on... The more I think about it, the more I feel that where we failed for 8.3 was not having a short 8.4 cycle lined up, which would give more freedom to bump patches to the next release. > The theme that our release cycles are too long is not exactly new, > of course, eg > http://archives.postgresql.org/pgsql-hackers/2000-05/msg00574.php > http://archives.postgresql.org/pgsql-hackers/2001-06/msg00766.php > http://archives.postgresql.org/pgsql-hackers/2003-11/msg00889.php Yeah, I remember all that, and I think you'll find that I mostly was on the other side of this issue :-) But many of the arguments from back then don't apply any more. Remember when you couldn't depend on pg_dump giving you dumps in the right order? Now that was a recipe for painful upgrades. With things like Slony, it's now possible to upgrade a majority of the Postgres installations with extremely minimal downtime. (And yes, I happen to have one that wont work with slony, but hey...) > but by now I think we've learned to stop banging our heads against > that particular rock. One-year major cycles work for this project, > shorter ones are wishful thinking. > Do they? 1 year cycles certainly don't solve the problem of being left with big/complex left over patches. They don't decrease the amount of time (as a percentage) we spend in freeze/beta/release mode. And we don't even get them out in 1 year; this 1 year cycle looks like at least 15 months. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Here is morning now, so I started to follow the discussion now... Stephen Frost wrote: > * Gregory Stark (st...@enterprisedb.com) wrote: >> It does seem weird to simply omit records rather than throw an error and >> require the user to use a where clause, even if it's something like WHERE >> pg_accessible(tab). It was an implementation of very earlier version of SE-PostgreSQL. (Maybe, its revision number was still less than 500.) It rewrites WHERE clause of given queries, but Tom suggested such a query rewrite easily makes a bug and hard to maintain in the future, so I removed the code and put a hook in ExecScan(), which featch a tuple from relation and checks condition. (I think it was a good suggestion. It also enables to reduce the scale of SE-PostgreSQL patches.) Indeed, it requires additional checks and disables a few kind of optimization, when these enhanced-security features are activated. However, I made clear some times that we assume security focused users don't give their first priority on performance. I can understand performance is a significant factor for database management system, so the default of these features are *disabled* unless user explicitly activate them. > It is weird from an SQL perspective, I agree with you there. On the > other hand, it's what the security community is looking for, and is > what's implemented by other databases (Oracle, SQL Server...) that > do row-level security and security labels. Requiring a where clause > or you throw an error would certainly make porting applications that > depend on that mechanism somewhat difficult, and doesn't really seem > like it'd gain you all that much... -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 18:51:01 Tom Lane wrote: > Robert Treat writes: > > Now I am starting to think that we cannot prevent large patches from > > showing up at the end of a cycle no matter what, and the only way to > > really "solve" that problem is to lesson the pain of getting bumped from > > a release. Ie. instead of being bump meaning you must wait 12-14 months > > till next release, we move toward more of a 6 month cycle of development. > > I really can't see going in that direction. In the first place, no one > wants to spend a third or more of the time in beta mode. Yeah, I was thinking that, but the truth is we do that now. We released last Febuary right? and we're looking at releasing (optimisttically) May 1st, right? So thats 15months, of which November - May (6 months) will have been feature freeze / beta / rc phase of development. > In the > second place, if the problem is big patches that take a long time to > develop, halving the length of the development cycle is no solution. > (If it did work, our plea to break large patches into segments landing > in different commitfests would have had more results.) I think this is a mis-assesment of our problem. The problem is not that big patches take a long time; not so much that they don't, just that is not a problem we can solve... "Hey Simon, code faster!" is not going to work ;-) The problem is that the pain point is extremely high for missing a given release cycle. If you don't make a specific release, you have a 12-14 month wait for feature arrival. Given that choice, someone who deperately need (aka wants) HS/SEPostgres/Win32/HOT/IPU/etc... will likely be willing to push a release 3-6 months for that one feature. Incidentally, this is probably why people didnt worry about making a given commitfest. The pain point was low, so there was no percieved need to rework a patch for a specific commit, since there was another one just a couple months away. However, we still see a rush of patches at the final freeze because people know that "there is no tommorrow" at that point. > In the third > place, unless we get an upgrade-in-place process that works all the > time, we would be looking at maintaining twice as many back branches > in order to provide the same kind of release lifespan we have today. > We are at the limit of what we can realistically do in back-branch > maintenance already :-( > Yeah, I can't argue with that. I'm not sure it's an unsolvable problem though; if odd/even release maintenance doesn't sound good, we could do something like ubuntus LTS, where we pick 1 release every 3 years to make a long-term support commitment (I think 5 years is our current max), and keep other releases on a shorter lifespan (1 or 2 years). Certainly having IPU (or is that UIP?) would make that an easier decision. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat writes: > On Tuesday 27 January 2009 10:34:59 Joshua D. Drake wrote: >> We have tried the short release cycle before, it was called 8.2. It >> fails, remarkably. > I think this is a bit of revisionsit history. JD got the release number wrong, it was 8.3, but otherwise there's no revisionism involved: http://archives.postgresql.org/pgsql-hackers/2006-12/msg00786.php The theme that our release cycles are too long is not exactly new, of course, eg http://archives.postgresql.org/pgsql-hackers/2000-05/msg00574.php http://archives.postgresql.org/pgsql-hackers/2001-06/msg00766.php http://archives.postgresql.org/pgsql-hackers/2003-11/msg00889.php but by now I think we've learned to stop banging our heads against that particular rock. One-year major cycles work for this project, shorter ones are wishful thinking. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 27 Jan 2009, Chad Sellers wrote: I'll speak to this a bit (Josh is also a Tresys employee). I can't say who my customers are, but I can speak to their needs. They really need row-level mandatory access controls (so both). From the perspective of what this would buy as far as this feature being a PostgreSQL advocacy point, one of the questions Tom asked about a bit upthread is still a bit hazy here. There are commercial database offerings selling into the "trusted" space already. While the use-cases you describe make perfect sense, I don't think it's clear to everyone yet if there's a unique draw to a PostgreSQL + selinux solution that the class of customers you're talking about would prefer it to purchasing one of those products. Is the cost savings the main driver here, or is there something else about a secure LAPP stack that makes it particularly compelling? -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Treat writes: > Now I am starting to think that we cannot prevent large patches from showing > up at the end of a cycle no matter what, and the only way to really "solve" > that problem is to lesson the pain of getting bumped from a release. Ie. > instead of being bump meaning you must wait 12-14 months till next release, > we move toward more of a 6 month cycle of development. I really can't see going in that direction. In the first place, no one wants to spend a third or more of the time in beta mode. In the second place, if the problem is big patches that take a long time to develop, halving the length of the development cycle is no solution. (If it did work, our plea to break large patches into segments landing in different commitfests would have had more results.) In the third place, unless we get an upgrade-in-place process that works all the time, we would be looking at maintaining twice as many back branches in order to provide the same kind of release lifespan we have today. We are at the limit of what we can realistically do in back-branch maintenance already :-( regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 10:34:59 Joshua D. Drake wrote: > On Tue, 2009-01-27 at 06:40 +0100, Pavel Stehule wrote: > > > 8.4-stable > > > 8.4-experimental > > > > > > stable is everything that stable is. PostgreSQL at its best. > > > > I dislike this idea - it's same like short processed 8.5 - > > Actually it isn't because we wouldn't accept features into > 8.4-experimental. The only thing we would accept into 8.4-experimental > would be bug fixes that would automatically be ported up to 8.5 (or > perhaps the other way around). We would still continue to build 8.5 as > normal. > > > that is > > more simple. > > We have tried the short release cycle before, it was called 8.2. It > fails, remarkably. > I think this is a bit of revisionsit history. While I'd agree it didn't work, the cycle was shorter... or to put it another way, would you say that the commitfest model failed miserably? should we scrap that for next cycle? I wonder... Feb 1 - all remaining patches bump / beta starts March 15th - beta ends / rc starts (note, giving 3 months for beta/rc, since we had a longer dev round) May 1st - release 8.4, open 8.5 dev June 1st - first 8.5 commitfest (we don't worry about short May because we have built up patche queue) July - second round of dev August - second/final commitfest Sept - beta opens October - rc opens November - release 8.5, open 8.6 December - first commitfest for 8.6 Jan 2010 - second dev cycle Feb - final commitfest March - 8.6 beta April - 8.6 rc May 2010 - release 8.6 -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby (v9d)
On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote:
> I'm doing some test runs with this now. I notice an old flatfiles
> related bug has reappeared:
Bug fix patch against git repo.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
*** a/src/backend/access/transam/xact.c
--- b/src/backend/access/transam/xact.c
***
*** 1381,1387 RecordTransactionAbort(bool isSubXact)
* main xacts, the equivalent happens just after this function returns.
*/
if (isSubXact)
! XidCacheRemoveRunningXids(xid, nchildren, children, latestXid);
/* Reset XactLastRecEnd until the next transaction writes something */
if (!isSubXact)
--- 1381,1387
* main xacts, the equivalent happens just after this function returns.
*/
if (isSubXact)
! XidCacheRemoveRunningXids(MyProc, xid, nchildren, children, latestXid);
/* Reset XactLastRecEnd until the next transaction writes something */
if (!isSubXact)
***
*** 4536,4541 RecordKnownAssignedTransactionIds(XLogRecPtr lsn, TransactionId top_xid, Transac
--- 4536,4548
{
int nxids = myproc->subxids.nxids;
+ /*
+ * It's possible for us to overflow the subxid cache and then
+ * for a subtransaction abort to reduce the number of subxids
+ * in the cache below the cache threshold again. If that happens
+ * then it's still OK for us to use the subxid cache again, since
+ * once its in the cache it lives there till abort or commit.
+ */
if (nxids < PGPROC_MAX_CACHED_SUBXIDS)
{
/*
***
*** 4621,4629 RecordKnownAssignedTransactionIds(XLogRecPtr lsn, TransactionId top_xid, Transac
LWLockRelease(ProcArrayLock);
elog(trace_recovery(DEBUG4),
! "record known xact top_xid %u child_xid %u %slatestObservedXid %u",
top_xid, child_xid,
(unobserved ? "unobserved " : " "),
latestObservedXid);
/*
--- 4628,4637
LWLockRelease(ProcArrayLock);
elog(trace_recovery(DEBUG4),
! "record known xact top_xid %u child_xid %u %s%slatestObservedXid %u",
top_xid, child_xid,
(unobserved ? "unobserved " : " "),
+ (mark_subtrans ? "mark subtrans " : " "),
latestObservedXid);
/*
***
*** 4690,4707 xact_redo_commit(xl_xact_commit *xlrec, TransactionId xid, bool preparedXact)
PGPROC *proc;
int i;
- /* Make sure nextXid is beyond any XID mentioned in the record */
- max_xid = xid;
sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]);
! /*
! * Find the highest xid and remove unobserved xids if required.
! */
! for (i = 0; i < xlrec->nsubxacts; i++)
! {
! if (TransactionIdPrecedes(max_xid, sub_xids[i]))
! max_xid = sub_xids[i];
! }
/* Mark the transaction committed in pg_clog */
TransactionIdCommitTree(xid, xlrec->nsubxacts, sub_xids);
--- 4698,4706
PGPROC *proc;
int i;
sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]);
! max_xid = TransactionIdLatest(xid, xlrec->nsubxacts, sub_xids);
/* Mark the transaction committed in pg_clog */
TransactionIdCommitTree(xid, xlrec->nsubxacts, sub_xids);
***
*** 4720,4726 xact_redo_commit(xl_xact_commit *xlrec, TransactionId xid, bool preparedXact)
*/
if (IsRunningXactDataValid() && !preparedXact)
{
! ProcArrayRemove(proc, InvalidTransactionId, xlrec->nsubxacts, sub_xids);
FreeRecoveryProcess(proc);
}
--- 4719,4725
*/
if (IsRunningXactDataValid() && !preparedXact)
{
! ProcArrayRemove(proc, max_xid, xlrec->nsubxacts, sub_xids);
FreeRecoveryProcess(proc);
}
***
*** 4790,4821 xact_redo_commit(xl_xact_commit *xlrec, TransactionId xid, bool preparedXact)
/*
* Be careful with the order of execution, as with xact_redo_commit().
* The two functions are similar but differ in key places.
*/
static void
! xact_redo_abort(xl_xact_abort *xlrec, TransactionId xid, bool preparedXact)
{
PGPROC *proc = NULL;
TransactionId *sub_xids;
TransactionId max_xid;
int i;
- /* Make sure nextXid is beyond any XID mentioned in the record */
- max_xid = xid;
sub_xids = (TransactionId *) &(xlrec->xnodes[xlrec->nrels]);
!
! /*
! * Find the highest xid and remove unobserved xids if required.
! */
! for (i = 0; i < xlrec->nsubxacts; i++)
! {
! if (TransactionIdPrecedes(max_xid, sub_xids[i]))
! max_xid = sub_xids[i];
! }
/* Mark the transaction aborted in pg_clog */
TransactionIdAbortTree(xid, xlrec->nsubxacts, sub_xids);
! if (InArchiveRecovery && (proc = BackendXidGetProc(xid)) != NULL)
{
/*
* We must mark clog before we update the ProcArray. Only update
--- 4789,4815
/*
* Be careful with the order of execution, as with xact_redo_commit().
* The two functions are similar but differ in key places.
+ *
+ * Note also that an abort can be for a subtransac
Re: [HACKERS] Table Inheritance and dropped columns
Caleb Welton writes: > Two child tables, one created with inheritance, one altered to have > inheritance: > When you drop a column from the parent table only the tables CREATED with > inheritance drop the column: Yeah, this is intentional. The other table is presumed to still own its columns even though it is (perhaps transiently) a child of the parent. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On 1/27/09 4:40 PM, "Ron Mayer" wrote: > Joshua Brindle wrote: >>> FWIW, as you know, sepostgresql is already included in Fedora. You can >>> continue shipping it as a seperate RPM set. >> >> That is non-ideal. Getting the capability in to the standard database >> shipped with RHEL is very important to me and my customers. > > Could you speak - even in general terms - about who your customers are > and what kinds of needs (is row-level acls the most important to them? > mandantory access control at the table level? both?) they have? > I'll speak to this a bit (Josh is also a Tresys employee). I can't say who my customers are, but I can speak to their needs. They really need row-level mandatory access controls (so both). I'll give a few examples that will hopefully help here. 1) One customer had several networks, each with a different classification. When a user needed to find anything, it was very painful as they would have to log into each network one at a time to search for anything. What they wanted to do was have a trusted database with a combined index of all the networks. They wanted to be able to write a flexible policy that could grant access to individual entries in the database based on what network the request came from in addition to the security clearance of the user requesting data. 2) Another example that we've been asked for repeatedly but had to turn away as the capability did not yet exist is a trusted LAMP/LAPP stack (Note that KaiGai is also working on the apache portion of this to provide a complete trusted LAPP stack). Under this model, individual web scripts can be run with a specific type. Combined with an SE-PostgreSQL policy, this gives the ability to restrict what a specific script can access. Additionally, as SE-PostgreSQL ties back in to the operating system mandatory access control mechanism, we can tie the type of the script back to the type of the actual user making the request. Coupled with labeled IPSec, this means we can control access to data in the database based on the clearance or role (or anything else you want to represent in their type) of the user on their end system. 3) A customer wanted to implement an approval process that required several steps. Without SE-PostgreSQL, we were forced to implement this by making lots of copies. Stage 1 would approve the package and copy it into Stage 2's inbox. We would grant each stage write access to the next stage's inbox in order to enforce information flow. This was very expensive, and didn't scale well. With SE-PostgreSQL, we could leave the data where it was and simply relabel the row(s). Each stage would be granted the ability to relabel from its type to the type of the next stage. No copies are necessary. I hope those help. I realize that many of you may not be used to dealing with customers who have such stringent security requirements, but if SE-PostgreSQL gets merged, that could change. Thanks, Chad Sellers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Table Inheritance and dropped columns
Hello, I'm trying to figure out if this is a bug or a feature. Two child tables, one created with inheritance, one altered to have inheritance: create table A(a text, b text); create table A1() inherits (A); create table A2(a text, b text); alter table A2 INHERIT A; When you drop a column from the parent table only the tables CREATED with inheritance drop the column: alter table A drop column b; \d A1 Column | Type | Modifiers +--+--- a | text | Inherits: a \d A2 Column | Type | Modifiers +--+--- a | text | b | text | Inherits: a Is this intended behavior? I have postgres 8.2.9. -Caleb -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
That's modest. I've talked to several oracle and db2 shops that want a standby for reporting that has relatively easy setup/maintenance (handling ddl is a big part of this) and the HS feature your working on will give them something as good as what they are getting now. So yeah, HS appeals to future users as well. I've talked to some of my clients, and while they *want* synch or near-synch HS, even slow HS is useful to them *now*. One client is planning on deploying a rather complex FS cloning infrastructure just to have a bunch of reporting, testing and read-only search databases they need. They'd be thrilled with an HS feature which produced DBs which were an hour out of date (or even 6 hours out of date), but ran read-only queries. --Josh. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 11:56:51 Simon Riggs wrote: > On Tue, 2009-01-27 at 11:36 -0500, Tom Lane wrote: > > David Fetter writes: > > > On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote: > > >> I don't think this is correct. > > > > > > I do. > > > > > > People literally grab my shoulder and ask when we'll have it. > > > > Do these people understand the difference between HS and a complete > > replication solution? Are they still as excited after you explain > > the difference? > > Yes, I think they do. > > http://www.postgresql.org/community/survey.55 > These people seem to understand also. > > Sync rep *is* important, but it opens up new classes of applications for > us. As does SEP. Both of those are more speculative and harder to > measure, but we've seen big impact before from this type of new feature. > > HS appeals to current users. Current users aren't so worried about new > applications, they look forward to being able to run queries on their > currently idle standby servers. > That's modest. I've talked to several oracle and db2 shops that want a standby for reporting that has relatively easy setup/maintenance (handling ddl is a big part of this) and the HS feature your working on will give them something as good as what they are getting now. So yeah, HS appeals to future users as well. -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Marko Kreen wrote: > On 1/27/09, Peter Eisentraut wrote: >> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: >> > Such app already exists: >> > >> > http://ozlabs.org/~jk/projects/patchwork/ >> > >> > So it's a matter of just setting it up. >> >> >> I was in fact in the process of setting that up just now. :-) > > Nice to know. :) I feel that even if we decide to do our own > solution it would be good to try existing solution first. IIRC, we already installed and tried this a while ago. I don't remember exactly what it failed on, but there was something pretty clear. But maybe it's been fixed by now. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Monday 26 January 2009 15:13:56 dp...@pgadmin.org wrote: > On 1/26/09, Josh Berkus wrote: > > All, > > > >>> 1) having the last CF on Nov. 1 was a mistake. That put us square in > >>> the path of the US & Christian holidays during the critical integration > >>> phase .. > >>> which means we haven't really had 3 months of integration, we've had > >>> *two*. > > > > Actually, I'm thinking about this again, and made a mistake about the > > mistake. The *original plan* was that we were not going to accept any > > new patches for Nov-CF. Just revised patches from eariler Fests. We > > didn't stick to that, which is mostly why we are still reviewing now. > > I don't recall us discussing that, but it sounds like it might help next > cycle. > What would be the significance of opening up the tree to future development between the last commitfest and last commitfest -1, if no new patches could be introduced? Essentially this is where we are at now... November commit fest finished, December we "re-opened" for development, and we're in the January commitfest where no new features have been accepted. The problem is what to do when we get to the end of the commit fests, and we have a few reamining (invariably large/complex) patches that people don't want to push. I had been leaning toward the idea of pushing the 8.4 release back six months, but reopening development for 2-3 more development/commitfest cycles, but I am starting to think this is moving things in the wrong direction. Now I am starting to think that we cannot prevent large patches from showing up at the end of a cycle no matter what, and the only way to really "solve" that problem is to lesson the pain of getting bumped from a release. Ie. instead of being bump meaning you must wait 12-14 months till next release, we move toward more of a 6 month cycle of development. I'm not sure it's feasible to boil down beta/rc phase to two months, tough it seems possible if we were strict about bumping features that aren't ready, and if our development cycles only went 4 months. For end users who are concerned about continuous upgrading, we might think about putting restrictions on every other release (ie. require binary or data file level compatability with 8.4 for the 8.5 release, and remove that restriction for 8.6) to lesson the upgrade path. (alternativly, a working IPU plan could make that less of an issue) -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Joshua Brindle wrote: >> FWIW, as you know, sepostgresql is already included in Fedora. You can >> continue shipping it as a seperate RPM set. > > That is non-ideal. Getting the capability in to the standard database > shipped with RHEL is very important to me and my customers. Could you speak - even in general terms - about who your customers are and what kinds of needs (is row-level acls the most important to them? mandantory access control at the table level? both?) they have? I'm guessing a better understanding of how real-world users would use this feature would be enlightening. > Since you can turn this off with GUC I don't see why it makes sense to > ship 2 databases (nevermind the maintenance issues) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Mon, Jan 26, 2009 at 08:28:32PM -0500, Tom Lane wrote: > Hmm, you think selinux people read pgsql-announce? Maybe not, but it got it onto LWN, which is a lot more people. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines. signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 15:46 -0500, Tom Lane wrote: > Peter Eisentraut writes: > > Not to pick on you personally, but this is the kind of review that should > > have > > happened six months ago, not during a "why is our development process > > inadequate" discussion on the eve of beta. > > Right now, today, in this thread, is the first time that we've had any > opportunity to debate the design of SEPostgres with knowledgeable people > other than KaiGai-san. It would likely be better if we started a new > thread with a more appropriate title, but I see nothing wrong with > asking pretty fundamental questions. Except that Bruce and I already checked detailed documentation references on this very topic months ago. Check with Bruce; he was careful to point those things out to me, so I'm sure he'll do the same for you. I'm satisfied, on this point. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Devrim GÜNDÜZ (dev...@gunduz.org) wrote: > On Tue, 2009-01-27 at 15:03 -0500, Tom Lane wrote: > > With my other hat on (the red one) what I'm concerned about is whether > > this patch will ever produce a feature that I could turn on in the > > standard Red Hat/Fedora build of Postgres. > > FWIW, as you know, sepostgresql is already included in Fedora. You can > continue shipping it as a seperate RPM set. hrmpf, I wonder if that's why RH hasn't been as interested in pushing for inclusion upstream... Maybe the feel that's a workable solution? I wouldn't agree... but I don't run RH. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Devrim GÜNDÜZ wrote: On Tue, 2009-01-27 at 15:03 -0500, Tom Lane wrote: With my other hat on (the red one) what I'm concerned about is whether this patch will ever produce a feature that I could turn on in the standard Red Hat/Fedora build of Postgres. FWIW, as you know, sepostgresql is already included in Fedora. You can continue shipping it as a seperate RPM set. That is non-ideal. Getting the capability in to the standard database shipped with RHEL is very important to me and my customers. Since you can turn this off with GUC I don't see why it makes sense to ship 2 databases (nevermind the maintenance issues) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 15:03 -0500, Tom Lane wrote: > > With my other hat on (the red one) what I'm concerned about is whether > this patch will ever produce a feature that I could turn on in the > standard Red Hat/Fedora build of Postgres. FWIW, as you know, sepostgresql is already included in Fedora. You can continue shipping it as a seperate RPM set. -- Devrim GÜNDÜZ, RHCE devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org signature.asc Description: This is a digitally signed message part
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
On Tue, Jan 27, 2009 at 3:36 PM, Tom Lane wrote: > Peter Eisentraut writes: >> On Tuesday 27 January 2009 17:19:28 Tom Lane wrote: >>> It's a potential security hole, since GRANT ALL on a view used to >>> be de facto the same as GRANT SELECT, if you hadn't bothered to >>> create any rules. > >> That is a good point. But the only clean solution would be to make views >> never updatable by default, and invent a nonstandard syntax to make them so, >> which seems very unattractive to me. A GUC variable as a transition measure >> could work, though. > > Yeah, I tend to prefer the GUC approach over nonstandard syntax too. > We'd need a GUC anyway to determine the default behavior if no > nonstandard clause appeared; so we might as well just do that and not > bother with the syntax options. > +1 -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Personally, I think it'd be terrible to implement the suggestion that > > started this sub-thread since it breaks with what is currently done > > elsewhere and what the users of this feature would expect. > > Upthread we were being told that this patch breaks new ground and will > offer capability available nowhere else. Now I'm hearing that it's just > a "me too" patch to catch up with capability already available from N > commercial vendors. Which is it? argh, it's a combination, in the end. Oracle and SQL Server offer row level security, that's something we don't have today and is provided through PGACE and is a big piece of the security labels/context part of the high security RDBMS world. Neither of them (far as I know..) interoperate with a OS-level policy system to provide that additional integration with the rest of the system as a whole (the SE-Linux bits). I wasn't sure how easy they were to seperate and to use seperately. It looks like they can be used independently, which is great, and means you could implement row level security on a BSD platform, but you wouldn't get the integration with the OS policy unless you hooked in with the Trusted BSD system (which I think actually can be done through an SE-Linux userland port.. but I've never played with it). Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Stephen Frost writes: Personally, I think it'd be terrible to implement the suggestion that started this sub-thread since it breaks with what is currently done elsewhere and what the users of this feature would expect. Upthread we were being told that this patch breaks new ground and will offer capability available nowhere else. Now I'm hearing that it's just a "me too" patch to catch up with capability already available from N commercial vendors. Which is it? It is like the difference between Trusted Solaris (really all the old trusted OS's) and SELinux. They both implement mandatory access control and both implement Bell and LaPadula as needed by the government/military but SELinux, via type enforcement, goes further to provide a completely flexible mandatory access control system. SELinux is useful to meet all sorts of security goals, from system and application integrity to data pipelining and confidentiality. The SELinux community believes this sort of access control is important to not only the military but commercial and even small scale systems. Further, because sepostgresql integrates well with SELinux the same system wide access controls flow seamlessly into the database. Are you able to access secret data on the filesystem? If so you'll be able to access secret data in the database. Are you able to update accounting information in the filesystem? Then you'll be able to update accounting information in the database. This also integrates with KaiGai's other work to SELinux-ize apache so that an apache server can run a user script from a users home directory and a type transition occurs to run the script in the appropriate domain for that user, then when that script accesses the database they'll have only the access that users script should have. This kind of end-to-end integration with mandatory access control is certainly ground breaking and isn't just the same ol' same ol' that other database vendors are doing (and have been doing for quite some time). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Peter Eisentraut writes: > On Tuesday 27 January 2009 16:52:15 Tom Lane wrote: >> Indeed. We might put up with a perl script for awhile for the sake of >> development expediency, but the long-term expectation would have to be >> that someone would rewrite it in C. Given that, I wonder whether >> there's much point in a rewrite into Perl if we already have a working >> shell script. I suppose someone will say "but you'll get no testing >> from Windows users then..." > The existing ksh script needs about two weeks of work to make it work outside > of Solaris and to make it more robust. Then you might as well rewrite it in > a more portable and robust language. Agreed, if it has to be gone over in that much detail, conversion to perl might not be a bad idea. I still say it'd have to be C eventually, but it'd be good to use something more concise until all the design issues are shaken out. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Tom Lane (t...@sss.pgh.pa.us) wrote: > BTW, whilst we are being beat about the head and shoulders with how > Oracle et al already have features like this, it is entirely appropriate > to wonder how come it's not in the standard. Those companies surely > pretty much control the standards committee, and they have managed to > push a ton of rather dubious things into the last couple of SQL updates. > If row-level security is such a mess that they couldn't standardize it, > that tells me something. For my 2c, for whatever it's worth, it's a combination of a specialized user base and the fact that this kind of security used to only be in a seperate product (eg: Trusted Oracle). Perhaps it will be in the standard some day, I don't think it's unreasonable to think that. Certainly it'd be nice if it was already there. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] pg_upgrade project status
On Tuesday 27 January 2009 23:02:30 Dave Page wrote: > On Tue, Jan 27, 2009 at 8:54 PM, Peter Eisentraut wrote: > > On Tuesday 27 January 2009 21:36:01 Dave Page wrote: > >> If it's been around for a year, why hasn't it been > >> submitted long ago so we could have rewritten and reviewed etc. in > >> plenty of time? > > > > It's in the commit fest listing, and the discussion about which > > programming language to port it to happened weeks ago. > > It was added to the commitfest page on December 5th, and links to an > email from the day before. The only discussion on the language I can > see in the linked thread is David Fetter offering to rewrite the code > in perl. Well, the is ultimately about the same discussion we are having now, except that people are a lot more excited when you threaten to cancel the project. ;-) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
On Tue, 2009-01-27 at 15:09 -0500, Tom Lane wrote: > Robert Haas writes: > > On Tue, Jan 27, 2009 at 1:50 PM, Jeff Davis wrote: > >> I think both of these deserve at least a glance by a committer before > >> bouncing them. > > > While we're at it, I think the Ramon Lawrence/Bryce Cutt patch to > > "Improve Performance of Multi-Batch Hash Join for Skewed Data Sets" > > also deserves a look. > > There is no proposal or intention to bounce any of the remaining > commitfest items without consideration. SEPostgres and Hot Standby > are the ones under debate. > I must have misunderstood Peter's comment. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On 1/27/09, Peter Eisentraut wrote: > On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: > > Such app already exists: > > > > http://ozlabs.org/~jk/projects/patchwork/ > > > > So it's a matter of just setting it up. > > > I was in fact in the process of setting that up just now. :-) Nice to know. :) I feel that even if we decide to do our own solution it would be good to try existing solution first. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Stephen Frost writes: > Personally, I think it'd be terrible to implement the suggestion that > started this sub-thread since it breaks with what is currently done > elsewhere and what the users of this feature would expect. Upthread we were being told that this patch breaks new ground and will offer capability available nowhere else. Now I'm hearing that it's just a "me too" patch to catch up with capability already available from N commercial vendors. Which is it? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, 27 Jan 2009, Peter Eisentraut wrote: On Tuesday 27 January 2009 21:36:01 Dave Page wrote: If it's been around for a year, why hasn't it been submitted long ago so we could have rewritten and reviewed etc. in plenty of time? It's in the commit fest listing, and the discussion about which programming language to port it to happened weeks ago. But as Robert Haas pointed out upthread that didn't show up there until after the start of the main commitfest, and I will admit being the culprit who snuck it onto there. The script was brought up repeatedly going back to April, but Zdenek didn't explicitly submit it as part of his final set for the commitfest. It was obvious to me you needed it all along--Zdenek's other CF submissions implicitly depended on it if your goal was to upgrade from 8.3 to 8.4 in-place--which is why I nudged him to do that and then added the result to the page. I'm not sure what arguing over the exact sequence here accomplishes anyway. It's come up a bunch of times before, nobody ran with it, and Zdenek was more concerned about space reservation. Anyway, I see multiple people who seem excited by playing with a Perl port. I found a shell->Perl conversion tool recently: http://search.cpan.org/~clive/App-sh2p-0.04/ If you comment out all the calls to create_mapping which it chokes on for some reason, it produces a non-functional but helpful conversion into Perl. But since the primary dirty job here is to review whether the original script works as expected on the database problems it tries to solve rather than play with programming languages, I didn't think reporting progress just on that front was particularly exciting. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tuesday 27 January 2009 17:27:32 Kaare Rasmussen wrote: > Hi > > > I have had a very brief look at this. Translation to perl doesn't look > > difficult. I'll see what I can do during the next week or so. > > Perhaps I can lend you a hand if you need help with this. Not to stop you are anything, but in the lifetime of this script, there have been about half a dozen people offering to port or already porting it to Perl. What it needs is a design and functional review to see if it is worth porting or doing anything with at all. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, Jan 27, 2009 at 8:54 PM, Peter Eisentraut wrote: > On Tuesday 27 January 2009 21:36:01 Dave Page wrote: >> If it's been around for a year, why hasn't it been >> submitted long ago so we could have rewritten and reviewed etc. in >> plenty of time? > > It's in the commit fest listing, and the discussion about which programming > language to port it to happened weeks ago. It was added to the commitfest page on December 5th, and links to an email from the day before. The only discussion on the language I can see in the linked thread is David Fetter offering to rewrite the code in perl. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tuesday 27 January 2009 16:54:02 Magnus Hagander wrote: > Either way, there's no point to discuss that in detail until there > actually is a working implementation out there... perl will do fine > until then There is a working implementation out there. It is -- you guessed it -- sitting in the commit fest queue waiting for review. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tuesday 27 January 2009 16:52:15 Tom Lane wrote: > Indeed. We might put up with a perl script for awhile for the sake of > development expediency, but the long-term expectation would have to be > that someone would rewrite it in C. Given that, I wonder whether > there's much point in a rewrite into Perl if we already have a working > shell script. I suppose someone will say "but you'll get no testing > from Windows users then..." The existing ksh script needs about two weeks of work to make it work outside of Solaris and to make it more robust. Then you might as well rewrite it in a more portable and robust language. But yes, you have to think weeks here in terms of getting this polished. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, Jan 27, 2009 at 8:52 PM, Zdenek Kotala wrote: > > http://archives.postgresql.org/pgsql-hackers/2008-04/msg01051.php > > But you can read there that it is korn shell script and there is link to > it. OK, yes that does mention it. It's not exactly clear that it's something you wanted to include though. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tuesday 27 January 2009 21:36:01 Dave Page wrote: > If it's been around for a year, why hasn't it been > submitted long ago so we could have rewritten and reviewed etc. in > plenty of time? It's in the commit fest listing, and the discussion about which programming language to port it to happened weeks ago. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Magnus Hagander wrote:
Andrew Dunstan wrote:
I've installed drmingw to handle exceptions instead, so we'll see if
that gives us useful info. If not, I'll see what I can do with gdb.
Hadn't heard of drwmingw, I see how that can be useful :-)
report from DrMingw is below
cheers
andrew
initdb.exe caused an Access Violation at location 7c91b1fa in module
ntdll.dll Writing to location 20202030.
Registers:
eax=20202020 ebx= ecx= edx=003eab70 esi=003eab70
edi=
eip=7c91b1fa esp=0022b820 ebp=0022b894 iopl=0 nv up ei pl nz na
po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=
efl=0206
Call stack:
7C91B1FA ntdll.dll:7C91B1FA RtlpWaitForCriticalSection
7C901046 ntdll.dll:7C901046 RtlEnterCriticalSection
77C3F34F msvcrt.dll:77C3F34F _popen
00401493 initdb.exe:00401493 popen_check initdb.c:477
static FILE * popen_check(
const char * command = ,
const char * mode =
)
...
errno = 0;
cmdfd = popen(command, mode);
> if (cmdfd == NULL)
fprintf(stderr, _("%s: could not execute command \"%s\": %s\n"),
progname, command, strerror(errno));
...
00404DA0 initdb.exe:00404DA0 main initdb.c:1650
int main(
int argc = 7,
char * * argv = &0x003e3d21
)
...
DEVNULL);
> PG_CMD_OPEN;
for (line = sysviews_setup; *line != NULL; line++)
...
004011E7 initdb.exe:004011E7
00401238 initdb.exe:00401238
7C817067 kernel32.dll:7C817067 RegisterWaitForInputIdle
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Peter Eisentraut writes: > On Tuesday 27 January 2009 16:36:50 Stephen Frost wrote: >> The SQL spec doesn't define row-level security, and coming >> up with something willy-nilly on our own doesn't really strike me as the >> best approach. > Exactly. But there is plenty of discussion on that elsewhere. BTW, whilst we are being beat about the head and shoulders with how Oracle et al already have features like this, it is entirely appropriate to wonder how come it's not in the standard. Those companies surely pretty much control the standards committee, and they have managed to push a ton of rather dubious things into the last couple of SQL updates. If row-level security is such a mess that they couldn't standardize it, that tells me something. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Dave Page píše v út 27. 01. 2009 v 20:36 +: > On Tue, Jan 27, 2009 at 8:31 PM, Zdenek Kotala wrote: > >> > >> Neither of which are the correct place to submit work for inclusion in > >> PostgreSQL. > > > > http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php > > > > I sent it in April. > > Thats a patch to the server, not a korn script. Wrong URL perhaps? Sorry, I copied wrong email URL from thread http://archives.postgresql.org/pgsql-hackers/2008-04/msg01051.php But you can read there that it is korn shell script and there is link to it. Thats all Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Peter Eisentraut writes: > > Not to pick on you personally, but this is the kind of review that should > > have > > happened six months ago, not during a "why is our development process > > inadequate" discussion on the eve of beta. > > Right now, today, in this thread, is the first time that we've had any > opportunity to debate the design of SEPostgres with knowledgeable people > other than KaiGai-san. It would likely be better if we started a new > thread with a more appropriate title, but I see nothing wrong with > asking pretty fundamental questions. I agree with asking the questions, but I don't like the immediate assumption that we're going to have to kick the patch because someone asked a question or suggested an alternative design unless we actively decide that's the approach we want to go and it requires a serious rework of the patch. Personally, I think it'd be terrible to implement the suggestion that started this sub-thread since it breaks with what is currently done elsewhere and what the users of this feature would expect. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
* Peter Eisentraut (pete...@gmx.net) wrote: > > The SQL spec doesn't define row-level security, and coming > > up with something willy-nilly on our own doesn't really strike me as the > > best approach. > > Exactly. But there is plenty of discussion on that elsewhere. That's the nice thing about the SE-PostgreSQL patch.. It's at least following established row-level security setups in other enterprise RDBMSs... The folks coming out now and saying we should require using a WHERE clause or similar would cause a serious deviation from what's being done today, without any real change or advantage that I see.. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Peter Eisentraut writes: > Not to pick on you personally, but this is the kind of review that should > have > happened six months ago, not during a "why is our development process > inadequate" discussion on the eve of beta. Right now, today, in this thread, is the first time that we've had any opportunity to debate the design of SEPostgres with knowledgeable people other than KaiGai-san. It would likely be better if we started a new thread with a more appropriate title, but I see nothing wrong with asking pretty fundamental questions. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, Jan 27, 2009 at 8:31 PM, Zdenek Kotala wrote: > > Dave Page píše v út 27. 01. 2009 v 19:36 +: >> On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala wrote: >> > >> > Dave Page píše v út 27. 01. 2009 v 14:56 +: >> >> I'd rather it was written in an appropriate language before feature >> >> freeze, not in a language that makes it easier for the author but a >> >> shade harder for thousands of users three months into feature freeze. >> > >> > The script is one year old. It is integrated in the OpenSolaris and >> > Solaris 10U6. I presented the script on PGCon2008. Everybody could >> > complain about it long time ago. >> >> Neither of which are the correct place to submit work for inclusion in >> PostgreSQL. > > http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php > > I sent it in April. Thats a patch to the server, not a korn script. Wrong URL perhaps? -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 19:10:40 Gregory Stark wrote: > It does seem weird to simply omit records rather than throw an error and > require the user to use a where clause, even if it's something like WHERE > pg_accessible(tab). Not to pick on you personally, but this is the kind of review that should have happened six months ago, not during a "why is our development process inadequate" discussion on the eve of beta. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
Peter Eisentraut writes: > On Tuesday 27 January 2009 17:19:28 Tom Lane wrote: >> It's a potential security hole, since GRANT ALL on a view used to >> be de facto the same as GRANT SELECT, if you hadn't bothered to >> create any rules. > That is a good point. But the only clean solution would be to make views > never updatable by default, and invent a nonstandard syntax to make them so, > which seems very unattractive to me. A GUC variable as a transition measure > could work, though. Yeah, I tend to prefer the GUC approach over nonstandard syntax too. We'd need a GUC anyway to determine the default behavior if no nonstandard clause appeared; so we might as well just do that and not bother with the syntax options. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote:
Ron Mayer writes:
It seems to me that there are two different standards to which this feature
might be held.
Is the goal
a) SEPostgres can provide useful rules to add security to some
specific applications so long as you're careful to avoid crafting
policies that produce bizarre behaviors (like avoiding restricing
access to foreign key data you might need). On the other hand it
gives you enough rope to hang yourself and produce weird results
that don't make sense from a SQL standard point of view if you
aren't careful matching the SEPostgres rules with your apps.
or
b) SEPostgreSQL should only give enough rope that you can not
craft rules that produce unexpected behavior from a SQL point
of view; and that it would be bad if one can produce SEPostgres
policies that produce unexpected SQL behavior.
With my other hat on (the red one) what I'm concerned about is whether
this patch will ever produce a feature that I could turn on in the
standard Red Hat/Fedora build of Postgres. Right at the moment it seems
that the potential performance hit, for users who are *not using*
SEPostgres but merely have to use a build in which it is present,
might be bad enough to guarantee that that will never happen.
According to the comments in security/sepgsql/core.c:
/*
* sepgsqlIsEnabled
*
* This function returns the state of SE-PostgreSQL when PGACE hooks
* are invoked, to prevent to call sepgsql() functions when
* SE-PostgreSQL is disabled.
*
* We can config the state of SE-PostgreSQL in $PGDATA/postgresql.conf.
* The GUC option "sepostgresql" can have the following four parameter.
*
* - default: It always follows the in-kernel SELinux state. When it
*works in Enforcing mode, SE-PostgreSQL also works in
*Enforcing mode. Changes of in-kernel state are delivered
*to userspace SE-PostgreSQL soon, and SELinux state
*monitoring process updates it rapidly.
* - enforcing : It always works in Enforcing mode. In-kernel SELinux
*has to be enabled.
* - permissive : It always works in Permissive mode. In-kernel SELinux
*has to be enabled.
* - disabled : It disables SE-PostgreSQL feature. It works as if
*original PostgreSQL
*/
and in the hooks there is a pgace_feature that turns off the checks:
void
pgaceGramAlterRelation(Relation rel, HeapTuple tuple, DefElem *defel)
{
switch (pgace_feature)
{
#ifdef HAVE_SELINUX
case PGACE_FEATURE_SELINUX:
if (sepgsqlIsEnabled())
{
sepgsqlGramAlterRelation(rel, tuple, defel);
return;
}
break;
#endif
default:
break;
}
if (defel)
ereport(ERROR,
(errcode(ERRCODE_PGACE_ERROR),
errmsg("unable to set security attribute of
table "
"via ALTER TABLE")));
}
So the pgace_feature turns off the backend call, there is an extra function
call, and a branch but that shouldn't cause the kind of performance degradation
you are talking about.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer writes: > Tom Lane wrote: >> This seems to me to be exactly parallel to deciding that SELinux should >> control only table/column permissions within SQL; an approach that would >> be enormously less controversial, less expensive, and more reliable than >> what SEPostgres tries to do. > With the table/column approach, could users who needed some row-level > capabilities work around this easily by setting table-level access > control on partitions? Yeah, the same thing had just occurred to me. We currently throw an error if a user doesn't have permissions on every partition (child table), but perhaps that behavior could be adjusted. Ignoring unreadable children would provide behavior pretty similar to that proposed by SEPostgres. To some extent that just postpones the semantic pain until the day when we try to do unique and FK constraints that span partitions. However, I think (after only minimal thought) that that will only re-introduce the covert-channel issue, which Joshua has already stated to be acceptable. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 16:36:50 Stephen Frost wrote: > * Peter Eisentraut (pete...@gmx.net) wrote: > > As one of the earlier reviewers, I think the design is OK, but the way > > the implementation is presented was not acceptable, and very little has > > been accomplished in terms of reacting to our comments. For example, > > where is the SQL row security feature, which should have been designed, > > implemented, and committed separately, in the opinion of most > > commentaries. > > Eh? Are you thinking of column-level privileges, which was committed > last week? No. > The SQL spec doesn't define row-level security, and coming > up with something willy-nilly on our own doesn't really strike me as the > best approach. Exactly. But there is plenty of discussion on that elsewhere. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Dave Page píše v út 27. 01. 2009 v 19:36 +: > On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala wrote: > > > > Dave Page píše v út 27. 01. 2009 v 14:56 +: > >> I'd rather it was written in an appropriate language before feature > >> freeze, not in a language that makes it easier for the author but a > >> shade harder for thousands of users three months into feature freeze. > > > > The script is one year old. It is integrated in the OpenSolaris and > > Solaris 10U6. I presented the script on PGCon2008. Everybody could > > complain about it long time ago. > > Neither of which are the correct place to submit work for inclusion in > PostgreSQL. http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php I sent it in April. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tuesday 27 January 2009 16:48:21 Sam Mason wrote: > > though at EAL1 they're quite far from the EAL4+ that DB2, > > Oracle, etc get. > > As far as I understand, the different levels are about assuring a > set of code/features to some assurance level. The Wikipedia page[1] > gives a reasonable overview of the levels, but basically EAL1 says > that a limited amount of effort (in practical terms, several person > months/years of time for something like PG) was put in, EAL1 pretty much means, a test suite exists and passes. So that was easy. :-) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote: > Such app already exists: > > http://ozlabs.org/~jk/projects/patchwork/ > > So it's a matter of just setting it up. I was in fact in the process of setting that up just now. :-) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle
On Tuesday 27 January 2009 17:19:28 Tom Lane wrote: > If we'd had the SQL-spec behavior from day one, it wouldn't be a > problem, but you can't just blow off the old behavior like that. > It's a potential security hole, since GRANT ALL on a view used to > be de facto the same as GRANT SELECT, if you hadn't bothered to > create any rules. That is a good point. But the only clean solution would be to make views never updatable by default, and invent a nonstandard syntax to make them so, which seems very unattractive to me. A GUC variable as a transition measure could work, though. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, 2009-01-27 at 22:12 +0200, Heikki Linnakangas wrote: > Joshua D. Drake wrote: > > On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: > > > >> And last I think license is not BSD, I don't know how big problem it is. > > > > Uhh.. that kills it. We only accept BSD. > > Actually, it has been relicensed to BSD license, after a short > discussion started by Bruce within EnterpriseDB (all the code is written > by EDB employees). This happened yesterday, so I can't blame you for not > noticing ;-). Phew! :) Joshua D. Drake > > -- >Heikki Linnakangas >EnterpriseDB http://www.enterprisedb.com > -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: Simon Riggs writes: On Tue, 2009-01-27 at 13:57 -0500, Joshua Brindle wrote: Josh Berkus wrote: Hmmm. Why try to hide individual rows in tables then? That would seem not in keeping with the filesystem policies. Because rows have data in them. It is analogous to not allowing the contents of the file to be visible. However, the primary key is still known to exist through various means, which is more analogous to the filename. Since most keys are likely to be non-meaningful IDs, its not going to help you much. Even more to the point: if the expectation is that you can hide a row's data payload but not its primary key, you can accomplish that with column-level permissions, without having to get into any non-standard or even faintly surprising SQL behavior, We aren't saying we want to hide the payload of the data in an entire column, just the data in some of the rows. For example, if you have top secret and secret data in the same table a secret user would be able to see the entire row for secret rows but maybe only some of the data on the top secret rows (or maybe not see the rows at all). Further, the top secret rows may have some fields that are inaccessible but are accessible through a trusted stored procedure that does fuzzing on the data (back to the coordinates example I used earlier) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Joshua D. Drake wrote: On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: And last I think license is not BSD, I don't know how big problem it is. Uhh.. that kills it. We only accept BSD. Actually, it has been relicensed to BSD license, after a short discussion started by Bruce within EnterpriseDB (all the code is written by EDB employees). This happened yesterday, so I can't blame you for not noticing ;-). -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Joshua D. Drake wrote: On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: And last I think license is not BSD, I don't know how big problem it is. Uhh.. that kills it. We only accept BSD. Actually, that was just fixed. Once somebody raised the issue, EDB changed the license (there have been on other contributors). I'm waiting for some documentation from Denis on that, but I expect to have it in a few days. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: 8.4 release planning (was Re: [HACKERS] [COMMITTERS] pgsql: Automatic view update rules)
Robert Haas writes: > On Tue, Jan 27, 2009 at 1:50 PM, Jeff Davis wrote: >> I think both of these deserve at least a glance by a committer before >> bouncing them. > While we're at it, I think the Ramon Lawrence/Bryce Cutt patch to > "Improve Performance of Multi-Batch Hash Join for Skewed Data Sets" > also deserves a look. There is no proposal or intention to bounce any of the remaining commitfest items without consideration. SEPostgres and Hot Standby are the ones under debate. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Simon Riggs writes: > On Tue, 2009-01-27 at 13:57 -0500, Joshua Brindle wrote: >> Josh Berkus wrote: >>> Hmmm. Why try to hide individual rows in tables then? That would seem >>> not in keeping with the filesystem policies. >> >> Because rows have data in them. It is analogous to not allowing the contents >> of >> the file to be visible. However, the primary key is still known to exist >> through >> various means, which is more analogous to the filename. > Since most keys are likely to be non-meaningful IDs, its not going to > help you much. Even more to the point: if the expectation is that you can hide a row's data payload but not its primary key, you can accomplish that with column-level permissions, without having to get into any non-standard or even faintly surprising SQL behavior, regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
> The flaw in that argument is that as you are doing it, the > de-optimization only happens on queries that actually need the behavior. > As the SEPostgres patch is constructed, the planner could *never* trust > an FK for optimization since it would have no way to know whether row > level permissions might be present (perhaps only for some rows) at > execution time. You could only get back the optimization in builds with > SEPostgres compiled out. That's pretty nasty, especially for packagers > who have to decide which build setting will displease fewer users. OK, I think I am starting to understand your concern now. My understanding of how the world works is SE-PostgreSQL would always be compiled in but could be turned off at run-time with a GUC. I know that the original design called for a compile-time switch, but everyone hated it and I am pretty sure KaiGai changed it. If he hasn't, he will. :-) There was also talk of having a table-level option to include/exclude the security ID (I'm not sure if it's currently implemented that way). Obviously that wouldn't be relevant for row-level MAC (because presumably you would need/want that turned on for all tables) but it would be very relevant for row-level DAC (because it's easy, at least for me, to imagine that you would only turn this on for a subset, possibly quite a small subset, of your tables where you knew that it was really needed). If, by default, we make sepostgresql disabled, MAC security IDs on newly created tables off, and DAC security IDs on newly created tables off, then the pain will be confined to people who explicitly request sepostgresql or row-level DAC. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer writes: > It seems to me that there are two different standards to which this feature > might be held. > Is the goal > a) SEPostgres can provide useful rules to add security to some > specific applications so long as you're careful to avoid crafting > policies that produce bizarre behaviors (like avoiding restricing > access to foreign key data you might need). On the other hand it > gives you enough rope to hang yourself and produce weird results > that don't make sense from a SQL standard point of view if you > aren't careful matching the SEPostgres rules with your apps. > or > b) SEPostgreSQL should only give enough rope that you can not > craft rules that produce unexpected behavior from a SQL point > of view; and that it would be bad if one can produce SEPostgres > policies that produce unexpected SQL behavior. With my other hat on (the red one) what I'm concerned about is whether this patch will ever produce a feature that I could turn on in the standard Red Hat/Fedora build of Postgres. Right at the moment it seems that the potential performance hit, for users who are *not using* SEPostgres but merely have to use a build in which it is present, might be bad enough to guarantee that that will never happen. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote: > And last I think license is not BSD, I don't know how big problem it is. Uhh.. that kills it. We only accept BSD. Joshua D. Drake > > Zdenek > > > > -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Heikki Linnakangas píše v út 27. 01. 2009 v 21:37 +0200: > Zdenek Kotala wrote: > > Set general value for heap or btree is possible but not optimal. For > > example If you have 5TB table and page layout changes do not affected > > this table but some small table needs perform a cleanup then 5TB will be > > processed in general approach, but in optimal variant it will be mark as > > prepared on upgrade automatically. > > Only the space reservation of *new* inserts/updates need to be > integrated in the backend. In addition to that, we'll need a pre-upgrade > script that decides which tables need processing, and process them. This > is no different whether we accept some generic space reservation patch > now, or backpatch a more precise one later. If it is possible then OK, but I little bit afraid about it. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Stephen Frost wrote:
* Joshua Brindle (met...@manicmethod.com) wrote:
They are separate. If you look at the patches you'll see a pgace part,
this is where the core interfaces to the security backends, and you'll
see a rowacl backend and an sepgsql backend.
Right, guess it wasn't clear to me that the PGACE bits for row-level
access control could be used independently of SELinux (and maybe even on
systems that don't have SELinux..?).
Sure, if you look at pgaceHooks.c you'll see:
bool
pgaceExecScan(Scan *scan, Relation rel, TupleTableSlot *slot)
{
/* Hardwired DAC checks */
if (!rowaclExecScan(scan, rel, slot))
return false;
switch (pgace_feature)
{
#ifdef HAVE_SELINUX
case PGACE_FEATURE_SELINUX:
if (sepgsqlIsEnabled())
return sepgsqlExecScan(scan, rel, slot);
break;
#endif
default:
break;
}
return true;
}
Notice the rowacl call outside of the HAVE_SELINUX ifdefs
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade project status
Tom Lane píše v út 27. 01. 2009 v 14:31 -0500: > Zdenek Kotala writes: > > Merlin Moncure píše v út 27. 01. 2009 v 09:47 -0500: > >> Just to clarify, does that mean that your patch has to be in for there > >> to be any chance of in-place upgrade 8.4->8.5? > > > Ok, There two patch in the queue for 8.5 which will bump page layout > > version. Then we will need it. > > I see nothing on the 2009-First list that requires any such thing. There are page CRC and item alignment optimization. And If IIRC that somebody what to modify compression? > In any case we've been over this ground before: we have agreed in the > past that we'd be willing to reject/postpone patches that change on-disk > data layout if in-place upgrade would otherwise be available. I think > that space reservation is extremely far down the list of "must haves" > for getting 8.4->8.5 upgrade into place. If it means that we will not change Page Layout Version in 8.5 then I'm happy. > You should first work on an > update process that supports catalog changes, and get that committed. > Once that's in place you'll have enough political capital to prevent > changes in user data layout, and then we'd start to think about schemes > like space reservation that could relax that ground rule. OK. There is pg_upgrade.sh for 8.3->8.4 while I or someone develop something better. But new solution will be at first for 8.4->8.5. thanks Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
* Joshua Brindle (met...@manicmethod.com) wrote: > They are separate. If you look at the patches you'll see a pgace part, > this is where the core interfaces to the security backends, and you'll > see a rowacl backend and an sepgsql backend. Right, guess it wasn't clear to me that the PGACE bits for row-level access control could be used independently of SELinux (and maybe even on systems that don't have SELinux..?). > Personally I'd like to see all of the access control moved out to use > pgace, including the standard DAC permissions but I doubt that would > never happen. Yeah... That's a whole 'nother discussion. Stephen signature.asc Description: Digital signature
Re: [HACKERS] 8.4 release planning
Tom Lane wrote: >> We do not consider that a short coming, anyone who needs to hide >> existence of files needs to set up their directory structure to >> disallow read/search/create on the directories they aren't allowed to >> discover filenames in. > > This seems to me to be exactly parallel to deciding that SELinux should > control only table/column permissions within SQL; an approach that would > be enormously less controversial, less expensive, and more reliable than > what SEPostgres tries to do. With the table/column approach, could users who needed some row-level capabilities work around this easily by setting table-level access control on partitions? In some ways that seems like it'd be easier to manage as well. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Robert Haas writes: > On Tue, Jan 27, 2009 at 12:52 PM, Tom Lane wrote: >> Right, but you expect that to be a small and predictable cost, say in >> the single-digits-percentage range. Plan optimizations that >> suddenly stop happening can cost you multiple orders of magnitude. > Well, look at it another way. If we don't accept row-level security > into PostgreSQL, then people will have to implement it themselves. In > fact, I currently have a real application that does exactly this. The > row-filtering is done, in essence, by having the web application add > certain conditions to the WHERE clause of certain queries depending on > which user is making the request. And if those WHERE clauses happen > to mention columns from table X, then table X won't be a candidate for > join removal. The only difference is that the logic is in my app > rather than in the database itself. > To put that another way, row-level permissions are just another > attribute of a table that could potentially affect the query result, > and the impact of referring to that attribute will be exactly the same > as the impact of referring to any other attribute in that table. The flaw in that argument is that as you are doing it, the de-optimization only happens on queries that actually need the behavior. As the SEPostgres patch is constructed, the planner could *never* trust an FK for optimization since it would have no way to know whether row level permissions might be present (perhaps only for some rows) at execution time. You could only get back the optimization in builds with SEPostgres compiled out. That's pretty nasty, especially for packagers who have to decide which build setting will displease fewer users. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, 2009-01-27 at 14:18 -0500, Tom Lane wrote: > Joshua Brindle writes: > > Tom Lane wrote: > >> Right, which is why it's bad for something like a foreign key constraint > >> to expose the fact that the row does exist after all. > > > Once again, this is not an issue for us. > > Yes it is an issue > The question of whether there is a covert channel is only a small part > of my complaint here. If it's the judgement of security experts that > that's an acceptable thing, that's fine, it's their turf. But SQL > behavior is my turf, and I'm not happy with discarding fundamental > semantic properties. Why did we bother to invite Joshua here if we aren't going to listen to him? Thanks for coming to help Joshua, much appreciated. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
On Tue, Jan 27, 2009 at 2:18 PM, Tom Lane wrote: > Joshua Brindle writes: >> Tom Lane wrote: >>> Right, which is why it's bad for something like a foreign key constraint >>> to expose the fact that the row does exist after all. > >> Once again, this is not an issue for us. > > Yes it is an issue; claiming it isn't doesn't make it so. You just > stated, in effect, that you don't implement data hiding in the > filesystem because it would break standard Unix filesystem semantics. > How is that consistent with your opinion that it's okay to break > standard SQL semantics in order to implement data hiding in a database? I think you're being pedantic. There are different levels of breakage and it is a matter of finding one that produces an acceptable cost-benefit trade-off. ISTM that we have plenty of evidence on these threads that other databases do things in a way that is similar to what SE-PostgreSQL is proposing to do. If people are using the feature in Oracle and getting value out of it, why should it suddenly become useless when ported to PostgreSQL? BTW, Oracle also has join removal, which proves that it isn't impossible for a high-quality database product to support both features. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 release planning
Ron Mayer wrote: Stephen Frost wrote: * Gregory Stark (st...@enterprisedb.com) wrote: It does seem weird to simply omit records rather than throw an error and require the user to use a where clause, even if it's something like WHERE pg_accessible(tab). It is weird from an SQL perspective, I agree with you there. On the other hand, it's what the security community is looking for, and is what's implemented by other databases (Oracle, SQL Server...) that do row-level security and security labels. Requiring a where clause or you throw an error would certainly make porting applications that depend on that mechanism somewhat difficult, and doesn't really seem like it'd gain you all that much... It seems to me that there are two different standards to which this feature might be held. Is the goal a) SEPostgres can provide useful rules to add security to some specific applications so long as you're careful to avoid crafting policies that produce bizarre behaviors (like avoiding restricing access to foreign key data you might need). On the other hand it gives you enough rope to hang yourself and produce weird results that don't make sense from a SQL standard point of view if you aren't careful matching the SEPostgres rules with your apps. This is what we like to call a flexible mandatory access control system, such as type enforcement. Give everything a type, write rules around the types. If you mess up your system may not boot (or your database queries might be strange). In talking to KaiGai the rules should not allow you to create inconsistencies, however. If you delete a row with a foreign key constraint and you cannot delete the associated row in another table the whole operation should fail. We want to fail on the side of maintaining data integrity and consistency while still allowing you enough rope to have a fine grained, flexible access control system. or b) SEPostgreSQL should only give enough rope that you can not craft rules that produce unexpected behavior from a SQL point of view; and that it would be bad if one can produce SEPostgres policies that produce unexpected SQL behavior. SELinux (and associated security aware applications) tend to go the way of very fine grained and let the policy author decide how to use it. I've seen SELinux policies with as few as 3 types, to as many as 3000. It is up to the policy author to choose the granularity appropriate for their purpose. I'd hate to see arbitrary limits put on the policy because of applications that might now operate correctly if they get an unexpected error (the applications should be fixed, not the security policy crippled) It seems to me many of the security-enhanced products seem to do the former; while it seems some of the objections to this patch are more of the latter. Historically trusted systems were fairly course grained, with only Bell and LaPadula policies[1]. SELinux is probably the first fine grained mandatory access control system in the mainstream. 1 http://en.wikipedia.org/wiki/Bell-LaPadula_model -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
