date:20091211

Stephen Frost wrote:
 * Greg Smith (g...@2ndquadrant.com) wrote:
 I personally feel that Steven  
 Frost's recent comments here about how the PostgreSQL code makes this  
 harder than it should be really cuts to the core of a next step here.   
 The problem facing us isn't is SEPostgreSQL the right solution for  
 providing external security checks?; it's how can the PostgreSQL code  
 be improved so that integrating external security is easier?  Looking  
 
 Thanks for that support, Greg.  This was what I was principally trying
 to do with KaiGai and the commitfest patch I reviewed of his last round.
 Unfortunately, the committer comments I received on that patch didn't
 help us outline a path forward, just declared that the approach in the
 current patch wasn't workable.  My, now much more optimistic thanks to
 our meeting, view is that the concept of abstracting the access controls
 is solid and a necessary first step; but we need to find a better way to
 implement it.

I agree. SELinux should be one of the user selectable options.
An abstraction layer for enhanced access controls are already adopted in
a few open source projects (Linux kernel, X-window), and gets successed.

 Also thanks to our discussion, I've got a much better handle on how
 SELinux and the general secuirty community views PostgreSQL (and the
 Linux kernel for that matter)- it's an application which consists of a
 set of object managers.  That then leads into an approach to address at
 least some of Tom's comments:
 
 Let's start by taking the patch I reviewed and splitting up
 security/access_control.c along object lines.  Of course, the individual
 security/object_ac.c files would only include the .h's that are
 necessary.  This would be a set of much smaller and much more
 managable files which only know about what they should know about- the
 object type they're responsible for.

Basically, agreed. This file had more than 4KL in the last commit fest.

IMO, however, we should consider these issues before splitting up old patch.

(1) Whether the framework should host the default PG model, not only
enhanced security features, or not?
  This patch tried to host both of the default PG model and SELinux.
  But, the default PG model does not have same origin with label-based
  mandatory access controls, such as SELinux, from a historical angle.
  So, this patch needed many of unobvious changes to the core routines.

(2) Whether the framework should be comprehensive, or not?
  This patch tried to replace all the default PG checks in the core
  routines from the begining. It required to review many of unobvious
  changes in the core routine. Because I've been repeatedly pointed out
  to keep changeset smaller, I'm frightened for the direction.
  The coverage of the latest SE-PgSQL/Lite patch is only databases,
  schemas, tables and columns. I think it is a good start to deploy
  security hooks on the routines related to these objects, rather than
  comprehensive support from the begining.

(3) In the future, whether we should allow multiple enhanced securities
at the same time, or not?
  It was not clear in the previous commit fest. But, in fact, Linux
  kernel does not support multiple security features in same time,
  because it just makes security label management complex. (It also
  have another reasons, but just now unlear.)
  I don't think we have any good reason to activate multiple enhanced
  securities at same time.

I think most of folks can agree with (2) and (3). But I expect opinion
is divided at (1).

For example, if we host the default PG checks to create a new database,
all the existing checks shall be moved as follows:

  
http://code.google.com/p/sepgsql/source/browse/trunk/pgsec/src/backend/security/access_control.c?r=2273#1430

In addition, it has to return the security context to be assigned on
the new database, when it hosts SELinux.
Even if this rework is limited to four kind of object classes, I'm
worry about it overs an acceptable complexity.

If the framework hosts only enhanced securities, this hook shall be
implemented as:

  /*
   * ac_database_create
   * It checks permission to create a new database, and returns a security
   * label to be assigned on, or NULL if unnecessary.
   *
   * datName   : name of the new database
   * srcDatOid : OID of the template database
   * datLabel  : user given security label, or NULL
   */
  Value *
  ac_database_create(const char *datName, Oid srcDatOid, Value *datLabel)
  {
  #ifdef HAVE_SELINUX
  if (sepgsql_is_enabled())
  return sepgsql_database_create(datName, srcDatOid, datLabel);
  #endif
  return NULL;
  }

In the caller (createdb()), this invocation shall be placed as follows:

  :
+  datLabel = ac_database_create(dbname, src_dboid, userDatLabel);
  :
+  if (!datLabel)
+  new_record_nulls[Anum_pg_database_datselabel - 1] = true;
+  else
+  new_record[Anum_pg_database_datselabel - 1]
+  =

Re: [HACKERS] Adding support for SE-Linux security

2009-12-11 Thread Magnus Hagander

On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Thu, Dec 10, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 My guess is that a credible SEPostgres offering will require a long-term
 amount of work at least equal to, and very possibly a good deal more
 than, what it took to make a native Windows port.

 The SEPostgres community is surely a lot smaller than the Windows
 community, but I'm not sure whether the effort estimate is accurate or
 not.  If credible includes row-level security, then I think I
 might agree, but right now we're just trying to get off the ground.

 It's been perfectly clear since day one, and was reiterated as recently
 as today
 http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com
 that what the security community wants is row-level security.  The

If that is true, then shouldn't we have an implementation of row level
security *first*, and then an implementation of selinux hooks that
work with this row level security feature? Rather than first doing
selinux hooks, then row level security, which will likely need new
and/or changed hooks...

I'm not convinced that row level security is actually that necessary
(though it's a nice feature, with or without selinux), but if it is,
it seems we are approaching the problem from the wrong direction.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Tom,

* Tom Lane (t...@sss.pgh.pa.us) wrote:
It's been perfectly clear since day one, and was reiterated as recently
as today
http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com
that what the security community wants is row-level security.

Yes, they do want row-level security. That being said, KaiGai, and
others, have pointed out time and time over again that SEPG without
row-level security is still useful. Additionally, I see absolutely no
way that PG would accept a full SEPG+PGACE+row-level security, etc,
patch in as one whole patch, ever. I have extreme doubt it would even
be something done over one *release*.

That all aside, for the moment, I feel that we should begin a
'two-prong' attack here. First, continue down the path that I've
started to lay out for SEPG. Second, let's hash out a design for
row-level security using the existing PG security model; ideally
using the best features and design decisions of the numerous row-level
security systems already implemented by the major SQL vendors today.

I'll start a new thread on this specific topic to hopefully pull out
anyone who's focus is more on that than on SEPG.

The
proposals to make SEPostgres drive regular SQL permissions never came
out of anyone from that side, they were proposed by PG people looking
for a manageable first step.

I do not believe this to be accurate. Josh, were you able to find any
public documentation on Trusted Rubix (is that the right name?)? The
RDBMS security policy hashed out on the SELinux list during the
discussion of Rubix and SEPG certainly included support for table-level
objects, did it not? I expect that the SELinux list contributors would
have pointed out if they didn't feel that was at all valuable.

Perhaps what is at issue is the terminology being used here though, or
the approach to enforment being considered. Part of the discussion at
the BWPUG meeting involved the option for SEPG to be a more-restrictive
only model in it's implementation. Essentially, this means that all
permissions handling is done the same as it is today, except that once
the PG model has decided an action is allowed, SEPG kicks in and does
any additional checking of the action being requested it wants and may
deny it.

At the end of the day, I don't feel that it really changes the
architecture of the code though. Perhaps users of SELinux will always
want that, but the argument we've heard time and time again here is that
this should be a generalized approach that other security managers could
hook into and use. To do that, I feel we first have to start with the
PG model, which *does* care about all the SQL permissions. Let's
extract the various complaints and concerns about SELinux that have been
thrown around this list and instead consider our first client of the
PG modular security framework to be the existing PG SQL permissions
system. If we can agree to that, then it's clear that we can't just
hand-wave the requirement that it be capable of driving the regular SQL
permissions.

Whatever you might believe about the
potential market for SEPostgres, you should divide by about a hundred
as long as it's only an alternate interface to SQL permissions. See
particularly here:
http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG#Revisiting_row-level_security
Without it, it's questionable whether committing the existing
stripped-down patch really accomplishes anything --- how much
clearer can they be?

Again, let's please address row-level security first at the PG level.
That was recommended previously by many on this list and is clearly a
useful feature which can stand alone in any case.

If you're not prepared to assume that we're going to do row level
security, it's not apparent why we should be embarking on this course
at all. And if you do assume that, I strongly believe that my effort
estimate above is on the optimistic side.

I do assume we're going to do row level security, but I do not feel that
we need to particularly put one in front of the other. I also feel that
SEPG will be valuable even without row-level security. One of the
realms that we discussed at BWPUG for this is PCI compliance. I'm
hopeful Josh will have an opportunity to review the PCI compliance
cheat-sheet that I recall Robert Treat offering and comes to agreement
that SEPG w/o row-level security would greatly improve our ability to
have a PCI compliant system backed with PG.

Thanks,

Stephen

signature.asc
Description: Digital signature

Re: [HACKERS] navigation menu for documents

2009-12-11 Thread Andrew Dunstan




Richard Huxton wrote:

Reworked jquery-based document menu is attached.

Untar will produce bin/ and html/
cd .../html
cp /path/to/htmldocs/* .
../bin/add_js.pl *html

The contents should be on almost all relevant pages, and:
- tested on FF3.5, IE7, Opera 9.x
- expanded if screen width  800px, closed otherwise
- contain all page-headings and sub-page headings
- highlight the current page in the tree
- provide tooltips for entries that are too wide for the panel

Issues:
- slow to reload on each page (can be fixed)
- based on hacked-together perl (this is temporary)
- layout is perhaps a bit cramped?

This last one is tricky. I can space it better if I can assume as 1280px
width screen. I can space it better if I have a menu that slides out and
  covers the whole page. Having the menu items wrap onto a second or
third line just didn't work. At least I couldn't find a way that made it
clear where the separating line between items was without cluttering the
whole menu horribly. Settled on not indenting too much, no word-wrap and
tooltips.

Comments/suggestions/rotten tomatoes?
  


Richard,

First, many thanks for doing this.

I don't see any comments having been made. I guess either everyone has 
been busy or they all think it's just fine (or both) :-)


I think you are definitely on the right track. I will use this, 
committed or not. But I'd like to see it finished and committed. Can you 
fix the 'slow to reload' problem - that is a bit of a pain. I'm not so 
worried about the use of perl to add the script calls - you need perl to 
build the docs anyway.


It wasn't terribly cramped for me.

I'm not sure everyone will want this. Maybe we need to provide 
with-js-menu and without-js-menu makefile target variants for html.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 4:31 AM, Magnus Hagander mag...@hagander.net wrote:
 On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Thu, Dec 10, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 My guess is that a credible SEPostgres offering will require a long-term
 amount of work at least equal to, and very possibly a good deal more
 than, what it took to make a native Windows port.

 The SEPostgres community is surely a lot smaller than the Windows
 community, but I'm not sure whether the effort estimate is accurate or
 not.  If credible includes row-level security, then I think I
 might agree, but right now we're just trying to get off the ground.

 It's been perfectly clear since day one, and was reiterated as recently
 as today
 http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com
 that what the security community wants is row-level security.  The

 If that is true, then shouldn't we have an implementation of row level
 security *first*, and then an implementation of selinux hooks that
 work with this row level security feature? Rather than first doing
 selinux hooks, then row level security, which will likely need new
 and/or changed hooks...

 I'm not convinced that row level security is actually that necessary
 (though it's a nice feature, with or without selinux), but if it is,
 it seems we are approaching the problem from the wrong direction.

I don't think there's a correct ordering to SE-PostgreSQL and
row-level security.  They're better together, but I don't think either
has to be done first.  If we were going to pick one to do first, I'd
pick SE-PostgreSQL.  Row-level security is going to be a @$#! of a
project if we want it done right (and we do).

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

KaiGai,

* KaiGai Kohei (kai...@ak.jp.nec.com) wrote:
(1) Whether the framework should host the default PG model, not only
enhanced security features, or not?
This patch tried to host both of the default PG model and SELinux.
But, the default PG model does not have same origin with label-based
mandatory access controls, such as SELinux, from a historical angle.
So, this patch needed many of unobvious changes to the core routines.

I certainly understand what you're saying here. However, I feel that
these changes to the core of PG are good, and are taking PG in the right
direction to have a much clearer and better implemented security
infrastructure. The fact that you've found a number of odd cases
(duplicate checking, odd failure cases due to checks being done at
various parts of the process, etc) is a testement that this is bringing
the PG code forward even without the addition of SELinux support.

I realize that makes it more work for you and I don't envy you that.

(2) Whether the framework should be comprehensive, or not?
This patch tried to replace all the default PG checks in the core
routines from the begining. It required to review many of unobvious
changes in the core routine. Because I've been repeatedly pointed out
to keep changeset smaller, I'm frightened for the direction.
The coverage of the latest SE-PgSQL/Lite patch is only databases,
schemas, tables and columns. I think it is a good start to deploy
security hooks on the routines related to these objects, rather than
comprehensive support from the begining.

I don't believe we will get support to commit a framework which is
intended to eventually support the PG model (following from my answer to
#1) in a partial form. Perhaps it would be good to split the patch up
on a per-object-type basis, to make it simpler/easier to review (as
in, patch #1: database_ac.c + changes to core for it; patch #2:
table_ac.c + changes to core for it, etc; earlier patches assumed to be
already done in later patches), but the assumption will almost certainly
be that they will all be committed to wonderful CVS at the same time.

(3) In the future, whether we should allow multiple enhanced securities
at the same time, or not?
It was not clear in the previous commit fest. But, in fact, Linux
kernel does not support multiple security features in same time,
because it just makes security label management complex. (It also
have another reasons, but just now unlear.)
I don't think we have any good reason to activate multiple enhanced
securities at same time.

To be honest, I feel that this will just fall out once we get through
doing #1. I've just heard from Casey (author of the SMACK security
manager, an alternative to SELinux) that the PGACE framework (which is
what we suggested he look at) would be easily modified to support SMACK.
I feel the same would be true of what we're talking about in #1. If you
disagree with that, please let me know.

I think most of folks can agree with (2) and (3). But I expect opinion
is divided at (1).

For example, if we host the default PG checks to create a new database,
all the existing checks shall be moved as follows:

http://code.google.com/p/sepgsql/source/browse/trunk/pgsec/src/backend/security/access_control.c?r=2273#1430

In addition, it has to return the security context to be assigned on
the new database, when it hosts SELinux.
Even if this rework is limited to four kind of object classes, I'm
worry about it overs an acceptable complexity.

I feel that what was at issue is that the complexity of
access_control.c was far too great- because *everything* was in that
one file. Splitting access_control.c into smaller files would make them
more managable, and if we implement them all in a similar manner using a
similar policy for function names, arguments, etc, then once the first
object type is reviewed, the others will go more easily for the reviwer.

If the security framework does *not* support the SQL model, I think
we'll get push back from the community that it clearly couldn't be used
by any other security manager generically without alot of potential
changes and additional hooks that could end up bringing us all the way
to being capable of supporting the SQL model anyway. I'm certainly
willing to hear core indicate otherwise though.

Thanks again, KaiGai.

Stephen

signature.asc
Description: Digital signature

Re: [HACKERS] [PATCH] dtrace probes for memory manager

2009-12-11 Thread Bernd Helmle




--On 10. Dezember 2009 23:55:49 -0500 Robert Haas robertmh...@gmail.com 
wrote:



If there's some real-world test where this probe costs 0.3%-0.4%, I
think that is sufficient grounds for rejecting this patch.  I
understand the desire of people to be able to use dtrace, but our
performance is too hard-won for me to want to give any measurable of
it up for tracing and instrumentation hooks that will only be used by
a small number of users in a small number of situations.


I repeated the pgbench runs per Greg's advice (see upthread) and it seems 
there is actually a small slowdown which supports this argument, 
unfortunately. After repeating the pgbench runs with and without the new 
probes (note: i've used the new version of the patch, too), the numbers are 
going to stabilize as follows:


without compiled probes: AVG(2531.68)
with compiled probes: AVG(2511.97)

I can repeat that tests over and over, but this doesn't seem to change the 
whole picture (so there seems some real argument for a 0.4 - 0.6% cost, at 
least on *my* machine here with pgbench).


--
Thanks

Bernd

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

2009-12-11 Thread Alvaro Herrera

Bernd Helmle escribió:

 I repeated the pgbench runs per Greg's advice (see upthread) and it
 seems there is actually a small slowdown which supports this
 argument, unfortunately. After repeating the pgbench runs with and
 without the new probes (note: i've used the new version of the
 patch, too), the numbers are going to stabilize as follows:
 
 without compiled probes: AVG(2531.68)
 with compiled probes: AVG(2511.97)

Were the probes enabled?

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

2009/12/11 KaiGai Kohei kai...@ak.jp.nec.com:
 It tried to provide a set of comprehensive entry points to replace existing
 PG checks at once.
 However, the SE-PgSQL/Lite patch covers accesses on only database, schema,
 tables and columns. Is it necessary to be comprehensive from the beginning?
 It might be too aggressive changes at once.

 Frankly, I hesitate to salvage the patch once rejected, as is.

 If we implement a set of security hooks, It seems to me the following approach
 is reasonable:

 * It does not touch the existing PG default checks.
  The purpose of security hooks are to host enhanced security features.

 * It does not deploy hooks on which no security provider is now proposed.
  It is important to reduce unnecessary changeset.

I think that we should try to move the PG default checks inside the
hook functions.  If we can't do that cleanly, it's a good sign that
the hook functions are not correctly placed to enforce arbitrary
security policy.  Furthermore, it defeats what I think would be a good
side goal here, which is to better modularize the existing code.

What I would suggest is that you develop a version of that patch that
is stripped down to apply to only a single object type (databases?
tables and columns - these might have to be together??) and which
addresses Tom's criticisms from the last time around, and post that
(on a new thread) for discussion.  That will be much easier to review
(and I will personally commit to reviewing it) but should allow us to
flush out some of the design issues.  If we can get agreement on that
as a concept patch, we can move on to talking about which object types
should be included in a committable version of that patch.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients

2009-12-11 Thread Alvaro Herrera

Jaime Casanova wrote:
 On Thu, Dec 10, 2009 at 11:33 PM, Jaime Casanova
 jcasa...@systemguards.com.ec wrote:
  On Thu, Dec 10, 2009 at 11:22 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 
  My bet is that the real problem was a build inconsistency in
  the backend.  Does make distclean and rebuild make it go away?
 
 
  actually it was a clean build just after a cvs co (not an updated
  tree), i build the binaries and installed it in just created
  directory...
  i will try again now with the patch Bruce just committed
 
 the problem has gone

Yes, but what if you test with the broken pgbench?  As Tom says, it
should not be able to crash the backend no matter what it does.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Stephen Frost wrote:

Tom,

snip

The
proposals to make SEPostgres drive regular SQL permissions never came
out of anyone from that side, they were proposed by PG people looking
for a manageable first step.

Trusted RUBIX does use the same SELinux object classes and permissions
that were originally added to the policy to support SEPostgreSQL. You
can look at
http://rubix.com/downloads/documentation/RX_SELinux_Guide_6_0_R4.pdf
and see how they use SELinux in their RDBMS. Pay particular attention to
page 15 where they are saying which object classes and permissions they
are using. They even implement row level security (the db_tuple object
class)

Again, let's please address row-level security first at the PG level.
That was recommended previously by many on this list and is clearly a
useful feature which can stand alone in any case.

Thanks,

Stephen

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

2009-12-11 Thread Stephen Smalley

On Fri, 2009-12-11 at 09:20 -0500, Robert Haas wrote:
 On Fri, Dec 11, 2009 at 4:31 AM, Magnus Hagander mag...@hagander.net wrote:
  On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote:
  Robert Haas robertmh...@gmail.com writes:
  On Thu, Dec 10, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  My guess is that a credible SEPostgres offering will require a long-term
  amount of work at least equal to, and very possibly a good deal more
  than, what it took to make a native Windows port.
 
  The SEPostgres community is surely a lot smaller than the Windows
  community, but I'm not sure whether the effort estimate is accurate or
  not.  If credible includes row-level security, then I think I
  might agree, but right now we're just trying to get off the ground.
 
  It's been perfectly clear since day one, and was reiterated as recently
  as today
  http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com
  that what the security community wants is row-level security.  The
 
  If that is true, then shouldn't we have an implementation of row level
  security *first*, and then an implementation of selinux hooks that
  work with this row level security feature? Rather than first doing
  selinux hooks, then row level security, which will likely need new
  and/or changed hooks...
 
  I'm not convinced that row level security is actually that necessary
  (though it's a nice feature, with or without selinux), but if it is,
  it seems we are approaching the problem from the wrong direction.
 
 I don't think there's a correct ordering to SE-PostgreSQL and
 row-level security.  They're better together, but I don't think either
 has to be done first.  If we were going to pick one to do first, I'd
 pick SE-PostgreSQL.  Row-level security is going to be a @$#! of a
 project if we want it done right (and we do).

I'm not sure it is a strong analogy, but as an example, in the case of
Linux, we started by integrating support for a base set of access
controls over directories and files, and only later introduced support
for multi-level/polyinstantiated directories by building upon Linux's
per-process filesystem namespace construct.  The base set of access
controls for directories and files were certainly useful on their own
even before we had the support for polyinstantiated directories.

In any event, I would agree that support for applying MAC over the
database objects and operations is useful even without row-level
security, although ultimately we would like both.

-- 
Stephen Smalley
National Security Agency


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review


Robert Haas wrote:

One comment I have in general about this process is that I think it
would enormously reduce the level of pain associated with making these
kinds of changes if we could get patches that were not full of minor
issues that need to be cleaned up (like comments not properly
adjusted, spelling/grammar errors, formatting inconsistent with the
rest of the code base, poor English).
Sure, it's not fair to ask the committers to do that.  But it's not 
really something we can expect to get in all situations from our English 
as a second language contributors either--it's hard enough to get these 
details right even on stuff done in this country by native speakers.  At 
least we've got a big stack of comments that usually make sense if you 
read them to work with on the SE-PostgreSQL patches, which is much 
better than the alternative.


I think we need a two pronged attack on this issue.  Eventually I think 
someone who wants this feature in there will need to sponsor someone 
(and not even necessarily a coder) to do a sizable round of plain old 
wording cleanup on the comment text of the patch.  Robert is right to 
point out the committers are too expensive, in terms of what work for 
the community they could be doing besides that, to waste on that job.  
English editing help good enough to help a lot for this purpose isn't 
hard to find though.  As far as formatting, it drives me a little crazy 
that it's not more common practice for contributors to do their own 
targeted pgindent runs to at least nail some of the low-hanging fruit in 
that area, due the difficulty of working with the tool.  That's 
something else I think it would be nice to invest a little time in 
improving.  For example, there's no reason we should still have to kick 
back patches regularly just because of tab/space issues, and I know I 
run into one of those nearly every CommitFest.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients

Alvaro Herrera alvhe...@commandprompt.com writes:
 Yes, but what if you test with the broken pgbench?  As Tom says, it
 should not be able to crash the backend no matter what it does.

The crash is real --- I've replicated it here.  Still trying to figure
out what is the real cause.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

Stephen Frost wrote:
 KaiGai,
 
 * KaiGai Kohei (kai...@ak.jp.nec.com) wrote:
 (1) Whether the framework should host the default PG model, not only
 enhanced security features, or not?
   This patch tried to host both of the default PG model and SELinux.
   But, the default PG model does not have same origin with label-based
   mandatory access controls, such as SELinux, from a historical angle.
   So, this patch needed many of unobvious changes to the core routines.
 
 I certainly understand what you're saying here.  However, I feel that
 these changes to the core of PG are good, and are taking PG in the right
 direction to have a much clearer and better implemented security
 infrastructure.  The fact that you've found a number of odd cases
 (duplicate checking, odd failure cases due to checks being done at
 various parts of the process, etc) is a testement that this is bringing
 the PG code forward even without the addition of SELinux support.
 
 I realize that makes it more work for you and I don't envy you that.

The reason why I prefer MAC only framework is not a technical reason.
As long as we can review it and acceptable, I have no reason to avoid it.

 (2) Whether the framework should be comprehensive, or not?
   This patch tried to replace all the default PG checks in the core
   routines from the begining. It required to review many of unobvious
   changes in the core routine. Because I've been repeatedly pointed out
   to keep changeset smaller, I'm frightened for the direction.
   The coverage of the latest SE-PgSQL/Lite patch is only databases,
   schemas, tables and columns. I think it is a good start to deploy
   security hooks on the routines related to these objects, rather than
   comprehensive support from the begining.
 
 I don't believe we will get support to commit a framework which is
 intended to eventually support the PG model (following from my answer to
 #1) in a partial form.  Perhaps it would be good to split the patch up
 on a per-object-type basis, to make it simpler/easier to review (as
 in, patch #1: database_ac.c + changes to core for it; patch #2:
 table_ac.c + changes to core for it, etc; earlier patches assumed to be
 already done in later patches), but the assumption will almost certainly
 be that they will all be committed to wonderful CVS at the same time.

As Rober Haas already suggested in another message, my patch in the last
commit fest is too large. It tried to rework anything in a single patch.
The per-object-type basis make sense for me.

In my cosmetic preference, ace_ is better than ac_. The 'e' means
extendable, and ace feels like something cool. :-)
In X-window system, it calls its framework XACE.

And I'd like to store these files in backend/security/ace/*.c, because
backend/security will also store other security modules!

src/
 + backend/
+ security/
   + ace/ ... access control framework
   + sepgsql/ ... selinux specific code
   + smack/   ... (upcoming?) smack specific code
   + solaristx/   ... (upcoming?) solaris-tx specific code
  :


 (3) In the future, whether we should allow multiple enhanced securities
 at the same time, or not?
   It was not clear in the previous commit fest. But, in fact, Linux
   kernel does not support multiple security features in same time,
   because it just makes security label management complex. (It also
   have another reasons, but just now unlear.)
   I don't think we have any good reason to activate multiple enhanced
   securities at same time.
 
 To be honest, I feel that this will just fall out once we get through
 doing #1.  I've just heard from Casey (author of the SMACK security
 manager, an alternative to SELinux) that the PGACE framework (which is
 what we suggested he look at) would be easily modified to support SMACK.
 I feel the same would be true of what we're talking about in #1.  If you
 disagree with that, please let me know.

Ahh, what I want to say is, whether we should allow enhanced securities.
If we host the default PG model on the framework, it is an exception of
the count.

The reason why I prefer the default PG check + one enhanced security at
most is simplification of the security label management.
If two label-based enhanced securities are enabled at same time,
we need two field on pg_class and others to store security context.
In addition, OS allows to choose one enhanced security at most eventually.

In my image, the hook should be as:

  Value *
  ac_database_create([arguments ...])
  {
  /*
   * The default PG checks here.
   * It is never disabled
   */

  /* enhanced security as follows */
  #if defined(HAVE_SELINUX)
  if (sepgsql_is_enabled())
  return sepgsql_database_create(...);
  #elif defined(HAVE_SMACK)
  if (smack_is_enabled())
  return smack_database_create(...);
  #elif defined(HAVE_SOLARISTX)
  if (soltx_is_enabled())
  return soltx_database_create(...);
  #endif

Re: [HACKERS] Adding support for SE-Linux security

On Fri, 2009-12-11 at 09:32 -0500, Robert Haas wrote:
 2009/12/11 KaiGai Kohei kai...@ak.jp.nec.com:
  It tried to provide a set of comprehensive entry points to replace existing
  PG checks at once.
  However, the SE-PgSQL/Lite patch covers accesses on only database, schema,
  tables and columns. Is it necessary to be comprehensive from the beginning?
  It might be too aggressive changes at once.
 
  Frankly, I hesitate to salvage the patch once rejected, as is.
 
  If we implement a set of security hooks, It seems to me the following 
  approach
  is reasonable:
 
  * It does not touch the existing PG default checks.
   The purpose of security hooks are to host enhanced security features.
 
  * It does not deploy hooks on which no security provider is now proposed.
   It is important to reduce unnecessary changeset.
 
 I think that we should try to move the PG default checks inside the
 hook functions.  If we can't do that cleanly, it's a good sign that
 the hook functions are not correctly placed to enforce arbitrary
 security policy.  Furthermore, it defeats what I think would be a good
 side goal here, which is to better modularize the existing code.

So from the meeting on Wednesday I got the impression that Steve already
did this. However it was rejected because too much information was need
to be passed around. I gathered and I could be wrong but the reason for
this is that instead of developing proper abstractions for the security
code people made use of whatever local stuff was available to them at
that location. With the X-ACE work that Eamon Walsh did he did exactly
what you're saying. He figured out the object model for the X-Server and
created the hook framework. In the merge of the hook framework he also
moved the existing X server security model into the framework. 

 
 What I would suggest is that you develop a version of that patch that
 is stripped down to apply to only a single object type (databases?
 tables and columns - these might have to be together??) and which
 addresses Tom's criticisms from the last time around, and post that
 (on a new thread) for discussion.  That will be much easier to review
 (and I will personally commit to reviewing it) but should allow us to
 flush out some of the design issues.  If we can get agreement on that
 as a concept patch, we can move on to talking about which object types
 should be included in a committable version of that patch.

They may have been said before but what exactly are the design issues?
The main concern I hear is that people are worried that this is an
SELinux specific design. I heard at the meeting on Wednesday that the
Trusted Extensions people looked at the framework and said it meets
their needs as well. If thats the case where does the concept that the
design is SELinux specific stem from? We've asked Casey Schaufler the
developer of another label based MAC system for Linux to look at the
hooks as well and make a statement about their usability.

Dave


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients

2009-12-11 Thread Marko Kreen

On 12/11/09, Tom Lane t...@sss.pgh.pa.us wrote:
 Alvaro Herrera alvhe...@commandprompt.com writes:
   Yes, but what if you test with the broken pgbench?  As Tom says, it
   should not be able to crash the backend no matter what it does.


 The crash is real --- I've replicated it here.  Still trying to figure
  out what is the real cause.

Several threads writing to single connection?

-- 
marko

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review


Greg Smith wrote:

It's funny; we started out this CommitFest with me scrambling to find
someone, anyone, willing to review the latest SE-PostgreSQL patch,
knowing it was a big job and few were likely to volunteer. Then
schedules lined up just right, and last night I managed to get a great
group of people all together to do perhaps the biggest single patch
review ever, to work on just that. I gathered up a list of the biggest
concerns about this feature and its associated implementation, we got a
number of regular PostgreSQL hackers and two of the security guys you've
seen on this list all in the same room, and we talked about little but
SEPostgreSQL for hours. Minutes are at
http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG and I'd
suggest anyone interested in this feature (or in rejecting this feature)
to take a look at what we covered.



I just wanted to add some talking notes here.

User base for the feature:

While my goals for this feature line up with military/government users 
this is in no way the extent of the potential user base. The fact is 
most people won't know they want this feature until it is available. Why 
is that? Well, how many of you have written webapps and implemented 
policy logic in your application rather than the database level? Why do 
people currently feel the need to do this? Is it even possible to 
implement some complex policies (eg., PCI compliance) at the database 
level? If PostgreSQL version whatever suddenly had the ability to 
implement the policy logic in the database, would you move it there? I 
know I would..


Audit:

In past conversations it sounded like some of the Postgres community was 
skeptical even about the design of the security model. For an even 
earlier look (September 2006) of KaiGai and the SELinux community 
talking about the object model and even high level design of the 
solution see http://marc.info/?l=selinuxm=115762285013528w=2


I've read through some of the prior patches, but haven't done an 
extensive audit, not only because of the size but because it became 
apparent relatively quickly that it was a waste of time and the 
community here wasn't going to accept it anyway. If this situation 
changes I think you'll find a few of us willing to donate time to the cause.



Policy:

The policy is easy once you have an object model that covers your use 
cases. You can see in the above discussion how we came to the object 
model we have now and why I've been comfortable with it since then.


An interesting aside, we must have hit the object model pretty well 
since another RDBMS (Trusted Rubix) uses the same one as SEPostgreSQL. 
See http://rubix.com/downloads/documentation/RX_SELinux_Guide_6_0_R4.pdf



Works outside of SELinux:

As Stephen already pointed out, Casey Schaufler (CC'd) who is the author 
of SMACK http://schaufler-ca.com/ believes that the abstractions 
provided by PGACE will allow integration of his security system as well. 
SMACK is already in the Linux kernel as an alternative LSM to SELinux.


Further, not that I've seen the code or know how they did it, Trusted 
Rubix has support for multiple access control systems 
http://rubix.com/cms/features including Trusted Solaris, Solaris 
Trusted Extensions, SELinux and their own RBAC system.


--

With all that said, I've very interested in seeing this work move along. 
In its current shape it has limited utility in my world (although I know 
of at least 2 solutions I've seen that run 20 Postgres servers on a 
single system just to have database separation). The main thing that 
prevents it from being used in that situation today is superuser access 
(eg., we can't have superusers that can go around and muck in data he's 
not cleared for). But I recognize that this is a first step to a 
potentially great system and I definitely want to see it moving forward.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Magnus,

* Magnus Hagander (mag...@hagander.net) wrote:
 On Fri, Dec 11, 2009 at 05:45, Tom Lane t...@sss.pgh.pa.us wrote:
  It's been perfectly clear since day one, and was reiterated as recently
  as today
  http://archives.postgresql.org/message-id/4b21757e.7090...@2ndquadrant.com
  that what the security community wants is row-level security.  The
 
 If that is true, then shouldn't we have an implementation of row level
 security *first*, and then an implementation of selinux hooks that
 work with this row level security feature? Rather than first doing
 selinux hooks, then row level security, which will likely need new
 and/or changed hooks...

The proposal we're currently grappling with is to pull all the various
checks which are sprinkled through our code into a single area.
Clearly, if that work is done before we implement row-level security,
then the patch for row-level security will just add it's checks in the
security/ area and it'd be then easily picked up by SELinux, etc.

 I'm not convinced that row level security is actually that necessary
 (though it's a nice feature, with or without selinux), but if it is,
 it seems we are approaching the problem from the wrong direction.

It has to be implemented independent of the security/SELinux/etc changes
in any case, based on what was said previously..  So I don't
particularly understand why it matters a great deal which one happens
first.  They're independently useful features, though both are not
nearly as good on their own as when they are combined.  Sorry, I just
don't see this as a cart-before-the-horse case.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

On Fri, 2009-12-11 at 08:56 -0500, Stephen Frost wrote:
[snip...]

 I do assume we're going to do row level security, but I do not feel that
 we need to particularly put one in front of the other.  I also feel that
 SEPG will be valuable even without row-level security.  One of the
 realms that we discussed at BWPUG for this is PCI compliance.  I'm
 hopeful Josh will have an opportunity to review the PCI compliance
 cheat-sheet that I recall Robert Treat offering and comes to agreement
 that SEPG w/o row-level security would greatly improve our ability to
 have a PCI compliant system backed with PG.
 

So I downloaded and read through the PCI DSS document (74 pages is
pretty light compared to NFSv4.1 hehe...) and There are several areas
there where I think strong access controls in the database will not only
fulfill the requirement but provide much stronger guarantees than can be
provided from the application server alone.

The requirements in section 7 can definitely benefit from SEPG. If you
implement these requirements in the application server and in PG access
controls alone there is still an attack vector where a malicious user
manages to steal the credentials for a particular role. With PG-ACE you
can write a security module (although SEPG already allows for this) to
restrict access to the data using the existing role-based access
controls in PG and then apply additional restrictions such as, only this
program may act as this role or access this database. This provides
better guarantees than exist in current PCI compliant implementations
using PG today.

Dave


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Largeobject Access Controls (r2460)

KaiGai Kohei wrote:
 Takahiro Itagaki wrote:
  KaiGai Kohei kai...@ak.jp.nec.com wrote:
  
  Tom Lane wrote:
  Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes:
 structnamepg_largeobject/structname should not be readable by the
 public, since the catalog contains data in large objects of all users.
  This is going to be a problem, because it will break applications that
  expect to be able to read pg_largeobject.  Like, say, pg_dump.
  Is it a right behavior, even if we have permission checks on large objects?
  
  Can we use column-level access control here?
  
 REVOKE ALL ON pg_largeobject FROM PUBLIC;
  = GRANT SELECT (loid) ON pg_largeobject TO PUBLIC;
 
 Indeed, it seems to me reasonable.
 
  We use SELECT loid FROM pg_largeobject LIMIT 1 in pg_dump. We could
  replace pg_largeobject_metadata instead if we try to fix only pg_dump,
  but it's no wonder that any other user applications use such queries.
  I think to allow reading loid is a balanced solution.
 
 Right, I also remind this query has to be fixed up by other reason right now.
 If all the large objects are empty, this query can return nothing, even if
 large object entries are in pg_largeobject_metadata.

metadata seems very vague.  Can't we come up with a more descriptive
name?

Also, how will this affect pg_migrator?  pg_migrator copies
pg_largeobject and its index from the old to the new server.  Is the
format inside pg_largeobject changed by this patch?  What happens when
there is no entry in pg_largeobject_metadata for a specific row?

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

2009-12-11 Thread Bernd Helmle




--On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera 
alvhe...@commandprompt.com wrote:




without compiled probes: AVG(2531.68)
with compiled probes: AVG(2511.97)


Were the probes enabled?


Hmm, they were just compiled in, i didn't anything to instrument them with 
dtrace.


I've just started a pgbench/dtrace run with instrumented probes aset_alloc, 
aset_free and aset_realloc which just counts the calls to them during 
pgbench, the first run gives me


tps = 1035.045523 (excluding connections establishing)

Ideas?

--
Thanks

Bernd

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

David,

* David P. Quigley (dpqu...@tycho.nsa.gov) wrote:
 So I downloaded and read through the PCI DSS document (74 pages is
 pretty light compared to NFSv4.1 hehe...) and There are several areas
 there where I think strong access controls in the database will not only
 fulfill the requirement but provide much stronger guarantees than can be
 provided from the application server alone.

Thanks for taking a look!  That sounds like excellent news.  My
apologies for attributing the action item to the wrong individual. :)

 The requirements in section 7 can definitely benefit from SEPG. 

I don't mean to be a pain, and we're all busy, but perhaps you could
include a short description of what 'requirements in section 7' are..
It would help keep the mailing list archive coherent, and be simpler for
folks who aren't familiar with PCI to play along.  A link to the
specific PCI DSS document you looked at would be an alternative, tho not
as good as a 'dumbed-down' description. ;)  That would at least avoid
confusion over which document, since I presume there's more than one out
there.

Thanks again for looking over this!  

Treat, you've dealt alot with PCI in your commercial work; could you
comment on this for the benefit of the list?  I don't doubt David in
the least, but it never hurts to have someone as lucky as yourself in
frequent dealings with PCI compliance to provide any additional
insight.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

* David P. Quigley (dpqu...@tycho.nsa.gov) wrote:
 On Fri, 2009-12-11 at 09:32 -0500, Robert Haas wrote:
  I think that we should try to move the PG default checks inside the
  hook functions.  If we can't do that cleanly, it's a good sign that
  the hook functions are not correctly placed to enforce arbitrary
  security policy.  Furthermore, it defeats what I think would be a good
  side goal here, which is to better modularize the existing code.
 
 So from the meeting on Wednesday I got the impression that Steve already
 did this. However it was rejected because too much information was need
 to be passed around.

KaiGai did all the work, but it was my suggestion to go down this
route and I reviewed KaiGai's patch to do it.  The specific
'review/rejection' email is here:
http://archives.postgresql.org/message-id/10495.1255627...@sss.pgh.pa.us

 I gathered and I could be wrong but the reason for
 this is that instead of developing proper abstractions for the security
 code people made use of whatever local stuff was available to them at
 that location.

That's certainly one concern I continue to have, but as I've been
rereading the threads I'm less confident it's a huge problem- the issue
seemed to be more about the single access_control.c knowing about the
entire PG world.

 With the X-ACE work that Eamon Walsh did he did exactly
 what you're saying. He figured out the object model for the X-Server and
 created the hook framework. In the merge of the hook framework he also
 moved the existing X server security model into the framework. 

It's great to hear specific examples of other projects which have gone
through this headache and come out the other side better for it.

  What I would suggest is that you develop a version of that patch that
  is stripped down to apply to only a single object type (databases?
  tables and columns - these might have to be together??) and which
  addresses Tom's criticisms from the last time around, and post that
  (on a new thread) for discussion.  That will be much easier to review
  (and I will personally commit to reviewing it) but should allow us to
  flush out some of the design issues.  If we can get agreement on that
  as a concept patch, we can move on to talking about which object types
  should be included in a committable version of that patch.
 
 They may have been said before but what exactly are the design issues?

Unfortunately, design isn't nearly as well defined a term as one would
hope.  I believe KaiGai's latest suggestion (which is probably what his
original PGACE implementation was, but I'm going to avoid looking, the
community has enough egg on it's face already wrt this :/) is a good
approach, along with splitting the huge access_control.c file into
per-object pieces.  That's a design change, tho perhaps not the kind of
one others who have commented on the design were thinking about when
they made those statements.

Basically, there's the design of the code layout and how each piece
knows about the other pieces (through header files, etc), and then
there's the design of the function calls/ABI and actual code paths
which are taken when the code is executed (which doesn't particularly
care how the code is laid out in the source tree).  I feel like the
design issues raised have been more about the former and less about
the latter.

 The main concern I hear is that people are worried that this is an
 SELinux specific design. I heard at the meeting on Wednesday that the
 Trusted Extensions people looked at the framework and said it meets
 their needs as well. If thats the case where does the concept that the
 design is SELinux specific stem from? We've asked Casey Schaufler the
 developer of another label based MAC system for Linux to look at the
 hooks as well and make a statement about their usability.

Hope I didn't steal your thunder wrt Casey!  Thanks again.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 10:07 AM, David P. Quigley
dpqu...@tycho.nsa.gov wrote:
 On Fri, 2009-12-11 at 09:32 -0500, Robert Haas wrote:
 2009/12/11 KaiGai Kohei kai...@ak.jp.nec.com:
  It tried to provide a set of comprehensive entry points to replace existing
  PG checks at once.
  However, the SE-PgSQL/Lite patch covers accesses on only database, schema,
  tables and columns. Is it necessary to be comprehensive from the beginning?
  It might be too aggressive changes at once.
 
  Frankly, I hesitate to salvage the patch once rejected, as is.
 
  If we implement a set of security hooks, It seems to me the following 
  approach
  is reasonable:
 
  * It does not touch the existing PG default checks.
   The purpose of security hooks are to host enhanced security features.
 
  * It does not deploy hooks on which no security provider is now proposed.
   It is important to reduce unnecessary changeset.

 I think that we should try to move the PG default checks inside the
 hook functions.  If we can't do that cleanly, it's a good sign that
 the hook functions are not correctly placed to enforce arbitrary
 security policy.  Furthermore, it defeats what I think would be a good
 side goal here, which is to better modularize the existing code.

 So from the meeting on Wednesday I got the impression that Steve already
 did this. However it was rejected because too much information was need
 to be passed around.

I am not sure who Steve is or which patch you're talking about, but
suffice it to say that I think the problem you are articulating here
is exactly the one we need to get out from under.  I don't know how to
do that yet and...

 They may have been said before but what exactly are the design issues?

...that's the design issue I think we need to surmount.  I think it
will be easier to talk through that with a mini-patch that only
affects one object type.

I'll stop here because I see that Stephen Frost has just sent an
insightful email on this topic as well.  Hmm, maybe that's the Steve
you were referring to.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review


Joshua Brindle wrote:

Greg Smith wrote:

It's funny; we started out this CommitFest with me scrambling to find
someone, anyone, willing to review the latest SE-PostgreSQL patch,
knowing it was a big job and few were likely to volunteer. Then
schedules lined up just right, and last night I managed to get a great
group of people all together to do perhaps the biggest single patch
review ever, to work on just that. I gathered up a list of the biggest
concerns about this feature and its associated implementation, we got a
number of regular PostgreSQL hackers and two of the security guys you've
seen on this list all in the same room, and we talked about little but
SEPostgreSQL for hours. Minutes are at
http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG and I'd
suggest anyone interested in this feature (or in rejecting this feature)
to take a look at what we covered.



I just wanted to add some talking notes here.

User base for the feature:

While my goals for this feature line up with military/government users
this is in no way the extent of the potential user base. The fact is
most people won't know they want this feature until it is available. Why
is that? Well, how many of you have written webapps and implemented
policy logic in your application rather than the database level? Why do
people currently feel the need to do this? Is it even possible to
implement some complex policies (eg., PCI compliance) at the database
level? If PostgreSQL version whatever suddenly had the ability to
implement the policy logic in the database, would you move it there? I
know I would..

Audit:

In past conversations it sounded like some of the Postgres community was
skeptical even about the design of the security model. For an even
earlier look (September 2006) of KaiGai and the SELinux community
talking about the object model and even high level design of the
solution see http://marc.info/?l=selinuxm=115762285013528w=2



I highly suggest a quick read of the above thread, it shows how we 
established an object model in fairly short order. The conversation also 
continues here: http://marc.info/?l=selinuxm=115786457722767w=2


and also here:
http://marc.info/?l=selinuxm=117160445604805w=2
http://marc.info/?l=selinuxm=117160445611588w=2
http://marc.info/?l=selinuxm=117160445608517w=2

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

KaiGai,

* KaiGai Kohei (kai...@kaigai.gr.jp) wrote:
 As Rober Haas already suggested in another message, my patch in the last
 commit fest is too large. It tried to rework anything in a single patch.
 The per-object-type basis make sense for me.

Agreed.

 In my cosmetic preference, ace_ is better than ac_. The 'e' means
 extendable, and ace feels like something cool. :-)

No complaints here..  I just hope this doesn't end up being *exactly*
the same as your original PGACE patches..  I'd feel terrible if we
weren't able to at least improve something with this extremely long and
drawn our process!

 And I'd like to store these files in backend/security/ace/*.c, because
 backend/security will also store other security modules!

This is perfectly reasonable in my view.  No complaints here.

 src/
  + backend/
 + security/
+ ace/ ... access control framework
+ sepgsql/ ... selinux specific code
+ smack/   ... (upcoming?) smack specific code
+ solaristx/   ... (upcoming?) solaris-tx specific code

Looks good to me.  Perhaps we'll have a smack/ patch showing up very
shortly as well..

 The reason why I prefer the default PG check + one enhanced security at
 most is simplification of the security label management.
 If two label-based enhanced securities are enabled at same time,
 we need two field on pg_class and others to store security context.

Ah, yes, I see your point must more clearly now.  This sounds reasonable
to me.

 In addition, OS allows to choose one enhanced security at most eventually.
 
 In my image, the hook should be as:
 
   Value *
   ac_database_create([arguments ...])
   {
   /*
* The default PG checks here.
* It is never disabled
*/
 
   /* enhanced security as follows */
   #if defined(HAVE_SELINUX)
   if (sepgsql_is_enabled())
   return sepgsql_database_create(...);
   #elif defined(HAVE_SMACK)
   if (smack_is_enabled())
   return smack_database_create(...);
   #elif defined(HAVE_SOLARISTX)
   if (soltx_is_enabled())
   return soltx_database_create(...);
   #endif
   return NULL;
   }
 
 We can share a common image image?

If all of these security modules make sense as only-more-restrictive,
then I have no problem with this approach.  Honestly, I'm fine with the
initial hooks looking as above in any case, since it provides a clear
way to deal with switching out the 'default PG checks' if someone
desires it- you could #if around that block as well.  As it's the
principle/primary, and I could see only-more-restrictive being more
popular, I don't mind having that code in-line in the hook.  The check
itself is still being brought out and into the security/ framework.

I do think that, technically, there's no reason we couldn't allow for
multiple only-more-restrictive models to be enabled and built in a
single binary for systems which support it.  As such, I would make those
just #if defined() rather than #elif.  Let it be decided at runtime
which are actually used, otherwise it becomes a much bigger problem for
packagers too.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] SE-PostgreSQL/Lite Review

Greg,

* Greg Smith (g...@2ndquadrant.com) wrote:
 I think we need a two pronged attack on this issue.  Eventually I think  
 someone who wants this feature in there will need to sponsor someone  
 (and not even necessarily a coder) to do a sizable round of plain old  
 wording cleanup on the comment text of the patch.

For the benefit of this list (this was already discussed some at the
BWPUG meeting), I agree with Greg on this and am actively looking to
try and help (either directly in my spare time or indirectly as a
sponsor).  If anyone else is interested or can do so as well, that would
be great!  Please speak up!

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

* Robert Haas (robertmh...@gmail.com) wrote:
 I'll stop here because I see that Stephen Frost has just sent an
 insightful email on this topic as well.  Hmm, maybe that's the Steve
 you were referring to.

I have doubts- but then I don't ever see my comments as insightful for
some reason. ;)

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] EXPLAIN BUFFERS

2009-12-11 Thread Euler Taveira de Oliveira

Robert Haas escreveu:
 On Thu, Dec 10, 2009 at 9:35 PM, Takahiro Itagaki
 itagaki.takah...@oss.ntt.co.jp wrote:
 Anyway, a revised patch according to the comments is attached.
 The new text format is:
  Buffers: shared hit=675 read=968, temp read=1443 written=1443
* Zero values are omitted. (Non-text formats could have zero values.)
* Rename Blocks: to Buffers:.
* Remove parentheses and add a comma between shared, local and temp.
 
 I did a bit of copy-editing of your doc changes to make the English a
 bit more correct and idiomatic.  Slightly revised patch attached for
 your consideration.  The output format looks really nice (thanks for
 bearing with me), and the functionality is great.
 
Please, document that zero values are omitted in the text format. It seems
intuitive but could be surprise because zero values are in non-text formats.


-- 
  Euler Taveira de Oliveira
  http://www.timbira.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Largeobject Access Controls (r2460)

KaiGai Kohei wrote:
  We use SELECT loid FROM pg_largeobject LIMIT 1 in pg_dump. We could
  replace pg_largeobject_metadata instead if we try to fix only pg_dump,
  but it's no wonder that any other user applications use such queries.
  I think to allow reading loid is a balanced solution.
  Right, I also remind this query has to be fixed up by other reason right 
  now.
  If all the large objects are empty, this query can return nothing, even if
  large object entries are in pg_largeobject_metadata.
  
  metadata seems very vague.  Can't we come up with a more descriptive
  name?
 
 What about property?
 The metadata was the suggested name from Robert Haas at the last
 commit fest, because we may store any other properties of a large
 object in this catalog future.

Well, we usually try to be more specific about what something represents
and only later abstract it out if needed, but if everyone else is fine
with 'metadata', then just leave it unchanged.

  Also, how will this affect pg_migrator?  pg_migrator copies
  pg_largeobject and its index from the old to the new server.  Is the
  format inside pg_largeobject changed by this patch?
 
 The format of pg_largeobject was not touched.

Good.

  What happens when
  there is no entry in pg_largeobject_metadata for a specific row?
 
 In this case, these rows become orphan.
 So, I think we need to create an empty large object with same LOID on
 pg_migrator. It makes an entry on pg_largeobject_metadata without
 writing anything to the pg_largeobject.
 I guess rest of migrations are not difference. Correct?

Uh, yea, pg_migrator could do that pretty easily.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Largeobject Access Controls (r2460)


Bruce Momjian さんは書きました:

KaiGai Kohei wrote:

Takahiro Itagaki wrote:

KaiGai Kohei kai...@ak.jp.nec.com wrote:


Tom Lane wrote:

Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes:

   structnamepg_largeobject/structname should not be readable by the
   public, since the catalog contains data in large objects of all users.

This is going to be a problem, because it will break applications that
expect to be able to read pg_largeobject.  Like, say, pg_dump.

Is it a right behavior, even if we have permission checks on large objects?

Can we use column-level access control here?

   REVOKE ALL ON pg_largeobject FROM PUBLIC;
= GRANT SELECT (loid) ON pg_largeobject TO PUBLIC;

Indeed, it seems to me reasonable.


We use SELECT loid FROM pg_largeobject LIMIT 1 in pg_dump. We could
replace pg_largeobject_metadata instead if we try to fix only pg_dump,
but it's no wonder that any other user applications use such queries.
I think to allow reading loid is a balanced solution.

Right, I also remind this query has to be fixed up by other reason right now.
If all the large objects are empty, this query can return nothing, even if
large object entries are in pg_largeobject_metadata.


metadata seems very vague.  Can't we come up with a more descriptive
name?


What about property?
The metadata was the suggested name from Robert Haas at the last
commit fest, because we may store any other properties of a large
object in this catalog future.


Also, how will this affect pg_migrator?  pg_migrator copies
pg_largeobject and its index from the old to the new server.  Is the
format inside pg_largeobject changed by this patch?


The format of pg_largeobject was not touched.


What happens when
there is no entry in pg_largeobject_metadata for a specific row?


In this case, these rows become orphan.
So, I think we need to create an empty large object with same LOID on
pg_migrator. It makes an entry on pg_largeobject_metadata without
writing anything to the pg_largeobject.
I guess rest of migrations are not difference. Correct?

Thanks,


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 10:07 AM, David P. Quigley
dpqu...@tycho.nsa.gov wrote:
 The main concern I hear is that people are worried that this is an
 SELinux specific design. I heard at the meeting on Wednesday that the
 Trusted Extensions people looked at the framework and said it meets
 their needs as well. If thats the case where does the concept that the
 design is SELinux specific stem from? We've asked Casey Schaufler the
 developer of another label based MAC system for Linux to look at the
 hooks as well and make a statement about their usability.

OK, on second thought I want to address this a little more, since some
of these concerns came from me.  SE-Linux-specific may be the wrong
way to say it.  There's an old saying that goes something like: if a
function has 10 parameters, it has 11 parameters.  In other words, if
you're passing too much information across a supposed abstraction
boundary, it's not really an abstraction boundary at all.

If we design a security abstraction layer, the interfaces need to
really be abstraction boundaries.  Passing the table OID and then also
the tablespace OID because PG DAC needs that to make its access
control decision is crap.  As soon as some other security model comes
along you will need to add additional arguments for all of the things
that the new security model cares about, and when a third security
model comes along you will need to pass those things now too.  If
we're doing that sort of thing, we might as well leave the entire
jumble of spaghetti code in its original location so that at least you
don't have to flip back and forth between two different source files.
And if we're going to do that, we might as well quit now because our
heads will explode (Tom's first).

I actually have an idea how to solve the problem in this particular
case, but I'm reluctant to say what it is because I'm not sure if I'm
right, and at any rate *I don't want to write this patch*.  I may
review it and if it's good I may commit it (or more likely a more
extensive version of it after we have agreement on the basic
principles) but I don't to want to write it or do substantial cleanup
of it.  I have my own list of things to work on and I presently have
no reason to put this on that list.  As a community, I think that at
times we have a tendency to say well, the committer is just going to
rewrite this anyway, so it doesn't really matter what we do.  As far
as I am concerned that is flat false.  I do not want to rewrite your
patch.  I want to look at your patch, convince myself that it is
correct, and commit it.  That won't always happen, and I'm certainly
willing to do more than that especially in cases where it's a feature
that I really care about, but AFAIAC the initiative is with the patch
author to provide something committable, NOT with the committer to fix
what the patch author did wrong.  I realize that's subjective at
times, and I'm going to make my best effort not to be a jerk about it,
but given that there are only so many hours in the day (and not all of
the can be spent on PostgreSQL) I think that's the goal we need to
shoot for.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, 2009-12-11 at 11:28 -0500, Stephen Frost wrote:
[snip...]
  The main concern I hear is that people are worried that this is an
  SELinux specific design. I heard at the meeting on Wednesday that the
  Trusted Extensions people looked at the framework and said it meets
  their needs as well. If thats the case where does the concept that the
  design is SELinux specific stem from? We've asked Casey Schaufler the
  developer of another label based MAC system for Linux to look at the
  hooks as well and make a statement about their usability.
 
 Hope I didn't steal your thunder wrt Casey!  Thanks again.

So we contacted Casey about another MAC model for PG using PG-ACE and he
got back to us with these reponses.

Josh Brindle (JB): So my question is, does smack have a facility
for userspace object managers?

Casey Schaufler (cs): Yes. The smack-util package includes a
small library which supports a user space version of the kernel
smackaccess() function. You pass it the subject label, object
label, and desired access and it returns a yes/no answer based
on what it reads from /smack/load.

So this answers our questions on whether or not SMACK has the faculties
to act as the security decision engine for PG.


JB: If so, I want to make the argument that doing a smack
integration using the pgace abstraction layer would not only
work but be fairly easy.

CS: Looking at some of the documentation I think that you can
safely make that argument. The security_label column would just
be the Smack label. The rules can be enforced by the user space
smackaccess(). System rows, whatever that might be, could get
the floor (_) label.

Casey mentions the row level access control in here but its safe to say
we've broken row based access control into a followup
discussion/project.

JB: All the sepgsql docs and code are up at
http://code.google.com/p/sepgsql/ and I'd like to get your
feedback before I start making claims...

CS: I can't see how it would take more than about a day if pgace
does what it looks like it should.

This seems to be a favorable assesment of the pgace framework's ability
to be used by something other than SELinux. So Casey's Smack module plus
the Sun guys saying it is usable by their legacy TSOL or TX code would
lend credence to the idea that pgace is bringing to the table. It may be
possible that you're not happy with certain aspects of the
implementation but the objects and permissions listed in pgace are
definitely ones that are worth mediating.

Dave

Dave


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, 2009-12-11 at 11:16 -0500, Stephen Frost wrote:
 David,
 
 * David P. Quigley (dpqu...@tycho.nsa.gov) wrote:
  So I downloaded and read through the PCI DSS document (74 pages is
  pretty light compared to NFSv4.1 hehe...) and There are several areas
  there where I think strong access controls in the database will not only
  fulfill the requirement but provide much stronger guarantees than can be
  provided from the application server alone.
 
 Thanks for taking a look!  That sounds like excellent news.  My
 apologies for attributing the action item to the wrong individual. :)

Nahh you attributed it to the correct person I just got a little bored
yesterday and poked my nose into it :)

 
  The requirements in section 7 can definitely benefit from SEPG. 
 
 I don't mean to be a pain, and we're all busy, but perhaps you could
 include a short description of what 'requirements in section 7' are..
 It would help keep the mailing list archive coherent, and be simpler for
 folks who aren't familiar with PCI to play along.  A link to the
 specific PCI DSS document you looked at would be an alternative, tho not
 as good as a 'dumbed-down' description. ;)  That would at least avoid
 confusion over which document, since I presume there's more than one out
 there.

So the document I read is linked below [1]. Requirement 7 falls under
Implement Strong Access Control Measures. The two main requirements
under requirement 7 seem to be limit access to system components and
card holder data to only those individuals whose job requires such
access and Establish an access control system for systems components
with multiple users that restricts access based
on a user’s need to know, and is set to “deny all” unless specifically
allowed.

The major flaw with this system is that stolen credentials allow you to
completely bypass the logic for this in the application layer. If the
person manages to get the login and password for an account on the
database it doesn't matter what their authenticated use is because the
logic to prevent accountant bob from cutting his own payroll check is in
the application layer. 

The way that MAC controls can strengthen the protections by making sure
that only certain components can perform certain actions. If you want to
make sure that only the accounting application can mess with that data
regardless of whether you have the database credentials or not then you
can do that. You can write policy for SEPG to say only programs with
this label can perform these actions on the database. Only applications
labeled with the accounting_tool label can modify the table labeled
accounting_data. This way if the system is compromised and someone
manages to get the username and password for the database account/role
that you were protecting the table with since the request isn't coming
from a process labeled accounting_tool the database will deny those
accesses. 

This is why I mean by adding stronger protections. This way you've
minimized the amount of code that you have to accredit for compliance in
your application server.

[1]
https://www.pcisecuritystandards.org/security_standards/download.html?id=pci_dss_v1-2.pdf
 
 Thanks again for looking over this!  
 
 Treat, you've dealt alot with PCI in your commercial work; could you
 comment on this for the benefit of the list?  I don't doubt David in
 the least, but it never hurts to have someone as lucky as yourself in
 frequent dealings with PCI compliance to provide any additional
 insight.

It is definitely good to have a second opinion on this since I've just
only started reading the PCI compliance documents. I'm definitely not an
expert in PCI compliance but from what I've read there are definite
benefits for using SEPG or PG-ACE with a special security module in
making much stronger guarantees about who and what can touch the card
data.

Dave


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Bernd Helmle píše v pá 11. 12. 2009 v 17:13 +0100:
 
 --On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera 
 alvhe...@commandprompt.com wrote:
 
 
  without compiled probes: AVG(2531.68)
  with compiled probes: AVG(2511.97)
 
  Were the probes enabled?
 
 Hmm, they were just compiled in, i didn't anything to instrument them with 
 dtrace.
 
 I've just started a pgbench/dtrace run with instrumented probes aset_alloc, 
 aset_free and aset_realloc which just counts the calls to them during 
 pgbench, the first run gives me
 
 tps = 1035.045523 (excluding connections establishing)
 
 Ideas?

When probes are activated they have performance impact. It is normal.
Important is that you can use it when you need it on production system.
No recompilation, no extra binary, no downtime and it is safe.
Performance impact depends on Dscript

Zdenek


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

On Fri, 2009-12-11 at 11:36 -0500, Stephen Frost wrote:
[Snip...]
 
  In addition, OS allows to choose one enhanced security at most eventually.
  
  In my image, the hook should be as:
  
Value *
ac_database_create([arguments ...])
{
/*
 * The default PG checks here.
 * It is never disabled
 */
  
/* enhanced security as follows */
#if defined(HAVE_SELINUX)
if (sepgsql_is_enabled())
return sepgsql_database_create(...);
#elif defined(HAVE_SMACK)
if (smack_is_enabled())
return smack_database_create(...);
#elif defined(HAVE_SOLARISTX)
if (soltx_is_enabled())
return soltx_database_create(...);
#endif
return NULL;
}
  
  We can share a common image image?
 
 If all of these security modules make sense as only-more-restrictive,
 then I have no problem with this approach.  Honestly, I'm fine with the
 initial hooks looking as above in any case, since it provides a clear
 way to deal with switching out the 'default PG checks' if someone
 desires it- you could #if around that block as well.  As it's the
 principle/primary, and I could see only-more-restrictive being more
 popular, I don't mind having that code in-line in the hook.  The check
 itself is still being brought out and into the security/ framework.
 
 I do think that, technically, there's no reason we couldn't allow for
 multiple only-more-restrictive models to be enabled and built in a
 single binary for systems which support it.  As such, I would make those
 just #if defined() rather than #elif.  Let it be decided at runtime
 which are actually used, otherwise it becomes a much bigger problem for
 packagers too.
 

So when Eamon did the XACE work he allowed for stackable security
modules. We do not allow for this in the Linux kernel but we can still
figure out whether we want to support it for now. I'm not convinced that
the ifdef model is the right way to go. There should be a security
structure where pointers to the appropriate functions reside (like we
have in linux) so this way you have a register_module call which takes
the mac model and puts all the necessary function pointers into the
structure and the hook is just a call to security-create_db. If you
want to make it stackable you make security-create_db a list of
function pointers that get executed in turn. I'll ask Eamon when I see
him next how he handled deregistration of a particular model.

Dave


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

On Fri, Dec 11, 2009 at 12:55 PM, Zdenek Kotala zdenek.kot...@sun.com wrote:
 Bernd Helmle píše v pá 11. 12. 2009 v 17:13 +0100:

 --On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera
 alvhe...@commandprompt.com wrote:

 
  without compiled probes: AVG(2531.68)
  with compiled probes: AVG(2511.97)
 
  Were the probes enabled?

 Hmm, they were just compiled in, i didn't anything to instrument them with
 dtrace.

 I've just started a pgbench/dtrace run with instrumented probes aset_alloc,
 aset_free and aset_realloc which just counts the calls to them during
 pgbench, the first run gives me

 tps = 1035.045523 (excluding connections establishing)

 Ideas?

 When probes are activated they have performance impact. It is normal.
 Important is that you can use it when you need it on production system.
 No recompilation, no extra binary, no downtime and it is safe.
 Performance impact depends on Dscript

Yeah.  The problem here is the impact when the probes aren't enabled.

I thought we had an idea of using the AllocSet dispatch mechanism to
make this zero-overhead in the case where the probes are not enabled.
What happened to that notion?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Robert Haas píše v čt 10. 12. 2009 v 23:55 -0500:
 On Wed, Dec 9, 2009 at 9:04 AM, Zdenek Kotala zdenek.kot...@sun.com wrote:

 
  But in normal situation database does also other thing and palloc is
  only one part of code path. It is why I run second test and use sun
  studio profiling tools (collect/analyzer) to determine how much CPU
  ticks cost the probes during pg_bench run. And results are much better.
  AllocSet alloc function takes about 4-5% and probes assembler code takes
  0.1-0.2% on 64bit. I did not test 32bit but my expectation is that it
  should be about 0.3-0.4%.
 
 There's not really enough detail here to determine what you tested and
 what the results were, and I don't think this patch has any chance at
 all of getting committed without that.  Please clarify.
 
 If there's some real-world test where this probe costs 0.3%-0.4%, I
 think that is sufficient grounds for rejecting this patch.  I
 understand the desire of people to be able to use dtrace, but our
 performance is too hard-won for me to want to give any measurable of
 it up for tracing and instrumentation hooks that will only be used by
 a small number of users in a small number of situations.
 

As I mentioned I run pg_bench -c10 -t1000 and collect data from
backends. collect and  analyzer is similar tool to gprof. 

Zdenek


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Robert Haas píše v pá 11. 12. 2009 v 12:55 -0500:
 On Fri, Dec 11, 2009 at 12:55 PM, Zdenek Kotala zdenek.kot...@sun.com wrote:
  Bernd Helmle píše v pá 11. 12. 2009 v 17:13 +0100:
 
  --On 11. Dezember 2009 11:28:54 -0300 Alvaro Herrera
  alvhe...@commandprompt.com wrote:
 
  
   without compiled probes: AVG(2531.68)
   with compiled probes: AVG(2511.97)
  
   Were the probes enabled?
 
  Hmm, they were just compiled in, i didn't anything to instrument them with
  dtrace.
 
  I've just started a pgbench/dtrace run with instrumented probes aset_alloc,
  aset_free and aset_realloc which just counts the calls to them during
  pgbench, the first run gives me
 
  tps = 1035.045523 (excluding connections establishing)
 
  Ideas?
 
  When probes are activated they have performance impact. It is normal.
  Important is that you can use it when you need it on production system.
  No recompilation, no extra binary, no downtime and it is safe.
  Performance impact depends on Dscript
 
 Yeah.  The problem here is the impact when the probes aren't enabled.
 
 I thought we had an idea of using the AllocSet dispatch mechanism to
 make this zero-overhead in the case where the probes are not enabled.
 What happened to that notion?

We know that performance impact is less then 1% probably less then 0.6%.
The question is if it is acceptable or not. I personally think that it
is acceptable. However if not, I will start work on backup solution with
dtraced AllocSet and some switching mechanism. But it needs little
discussion about design. And first we need decision about current
performance impact.

Zdenek







-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Python 3.1 support

2009-12-11 Thread Joshua D. Drake

On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote:
 On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
  There was considerable debate earlier about whether we wanted to treat
  Python 3 as a separate PL so it could be available in parallel with
  plpython 2, because of the user-level coding incompatibilities.  It
  looks like this patch simply ignores that problem.  What is going to
  happen to plpython functions that depend on 2.x behavior?
 
 I have a proposal for how to handle this, and a prototype patch
 attached.  This follows essentially what the CPython distribution itself
 does, which will make this tolerably easy to follow for users.
 
 We install plpython as plpython2.so or plpython3.so, depending on the
 version used to build it.  Then, plpython.so is a symlink to
 plpython2.so.
 
 We then create three language definition templates:
 
 plpythonu - plpython.so
 plpython2u - plpython2.so
 plpython3u - plpython3.so
 

 
 Comments?

Well as a Python guy... makes sense to me :)

Joshua D. Drake


-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - 
Salamander


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients

I wrote:
 The crash is real --- I've replicated it here.  Still trying to figure
 out what is the real cause.

Okay, I've sussed it.  The crash itself is due to a memory management
mistake in the recently-rewritten EvalPlanQual code (pfree'ing a tuple
that we still have live Datum references to).  The reason the broken
pgbench is able to trigger it is that it generates concurrent updates
for the same row in pgbench_accounts with much higher frequency than
normal.  This forces the backends through the EvalPlanQual code, which
is broken and crashes.  QED.

I'll commit a fix for that shortly, but this reminds me once again that
the EvalPlanQual logic is desperately under-tested in our normal
regression testing.  We really need some kind of testing infrastructure
that would let us exercise concurrent-update cases a bit more than not
at all.

Also, the reason why Bruce's mistake exposed this is interesting.
Omitting the #define for ENABLE_THREAD_SAFETY does not actually break
pgbench -j at all -- it has a fallback strategy that uses multiple
subprocesses instead of multiple threads.  However, it has only one
srandom() call, which occurs in the parent process before forking.
This means that the subprocesses all start with the same random number
state, which means they generate the same sequence of random account
IDs to update, which is why the probability of an update collision
requiring EvalPlanQual resolution is so high, especially at the start
of the run (and the crash does happen pretty much immediately after
starting pgbench).

I'm inclined to think this is bad, and we should fix pgbench to
re-randomize after forking.  If we don't, we'll have synchronized
behavior on some platforms and not others, which doesn't seem like a
good idea.  On the other hand, if you did want that type of behavior,
it's hard to see how you'd get it.  Is it worth trying to provide that
as a (probably non-default) mode in pgbench?  If so, how would you
do it on threaded platforms?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

On Fri, Dec 11, 2009 at 1:04 PM, Zdenek Kotala zdenek.kot...@sun.com wrote:
 We know that performance impact is less then 1% probably less then 0.6%.
 The question is if it is acceptable or not. I personally think that it
 is acceptable. However if not, I will start work on backup solution with
 dtraced AllocSet and some switching mechanism. But it needs little
 discussion about design. And first we need decision about current
 performance impact.

As far as I am concerned that is way too much, particularly
considering that your test case isn't designed to be particularly
memory-allocation intensive, and if it is up to me I will reject this.
 Even a quarter-percent slowdown for a feature that will be used only
by a small fraction of users only a small fraction of time time seems
totally unacceptable to me.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, 2009-12-11 at 11:30 -0500, Robert Haas wrote:
[snip...]
 
 I'll stop here because I see that Stephen Frost has just sent an
 insightful email on this topic as well.  Hmm, maybe that's the Steve
 you were referring to.
 
 ...Robert
 

Yea I never asked Stephen if he goes by Stephen or Steve when I met him
on Wednesday. I guess calling him Steve is me being a bit
presumptuous :)


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review


Stephen Frost wrote:

KaiGai,


snip

I do think that, technically, there's no reason we couldn't allow for
multiple only-more-restrictive models to be enabled and built in a
single binary for systems which support it.  As such, I would make those
just #if defined() rather than #elif.  Let it be decided at runtime
which are actually used, otherwise it becomes a much bigger problem for
packagers too.



It isn't just a case of using #if and it magically working. You'd need a 
system to manage multiple labels on each object that can be addressed by 
different systems. So instead of having an object mapped only to 
system_u:object_r:mydb_t:s15 you'd also have to have it mapped to, 
eg., ^ for Smack.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Robert,

* Robert Haas (robertmh...@gmail.com) wrote:
 I actually have an idea how to solve the problem in this particular
 case, but I'm reluctant to say what it is because I'm not sure if I'm
 right, and at any rate *I don't want to write this patch*.  

As far as crap goes, I'd have to put this at the top.  If you're not
willing to share ideas, then I may have to reconsider my personal
feeling on if you should be a committer or not.  No one is asking you to
write the patch.  We all know that we can be wrong (I tend to be more
wrong than most), and we all hate to jerk people around, but I feel
it's far worse to self-censor discussion on ideas.

It's also about the worst form of rock-management that I think I could
come up with in an open source community.  If you don't share your idea,
yet you feel that it's right, and see nothing to dissuade you from
that position (after all, we can't present an argument for or against it
if we don't know what it is), then I find it likely that you're going to
constantly be comparing patches presented to the ideal one in your head
based on your idea and we'd never get there.

If you're not willing to discuss your idea, then I would ask that
you not be involved in either this discussion or review of the patch.

Rock-management, for those not familiar with the term, is essentially
asking I need a rock, and then, when presented with a rock saying
sorry, that's not the rock I wanted.  This doesn't provide any insight
into what kind of rock you're looking for.

Rest of the I don't want to write it whine omitted.  No one's asking
you to.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Robert Haas robertmh...@gmail.com writes:
 As far as I am concerned that is way too much, particularly
 considering that your test case isn't designed to be particularly
 memory-allocation intensive, and if it is up to me I will reject this.
  Even a quarter-percent slowdown for a feature that will be used only
 by a small fraction of users only a small fraction of time time seems
 totally unacceptable to me.

It seems to me that anyone who really needs this can instrument the
alloc functions anyway --- isn't one of the features of DTrace supposed
to be that you can monitor calls to a particular function without any
prearranged code support?  Or is that one of the things like zero
overhead that turns out to be more marketing-speak than reality?

Anyway I concur with Robert's opinion that the use-case is far too small
to justify incurring a measurable overhead for everybody.  There might
be some small argument for putting these in under an extra #ifdef, but
they wouldn't get into any regular production build.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

All,

* Robert Haas (robertmh...@gmail.com) wrote:
 If we design a security abstraction layer, the interfaces need to
 really be abstraction boundaries.  Passing the table OID and then also
 the tablespace OID because PG DAC needs that to make its access
 control decision is crap.  

Now, to address the small useful bit of this mail- I tend to agree with
this.  I'm not convinced there's an alternative, but I'd like to throw
out a couple of my ideas on how it could be addressed.  I'd like to
solicit for feedback on these.

First off, we have alot of the information available from the catalog.
Basically, given an OID, we should be able to find out the other
information associated with that OID (such as what kind of object it is,
what tablespace it resides in, etc).  OID isn't sufficient, however, as
we know from the dependency system.  To address this, we would need OID
and SubOID.  Now, any information which we can derive from those should
not be included in the API.  To be honest, I don't think we've actually
been all that bad about this, but I'll reserve any definitive answer
until I've gone back through the API we have specifically thinking about
this issue.  On further thought, I'm probably wrong and should have
caught this during my review.  Sorry.

Second, the information we *don't* have from above is generally
information about what the requesting action is.  For example, when
changing ownership of an object, we can't possibly use introspection to
find out the role which is on the receiving end, since at that time
we've just learned it from the parser.  Now, we might build an entire
new object, pass the result of this action OID to the security system
and ask it can we change OID X into OID Y?, but I don't particularly
like it.  We really don't want to do any *changes* to things until after
we've determined the permissions allow for it, and I'm not sure how to
get around that without building an independent introspection system for
what might be.  Perhaps we're comfortable enough saying we'll just
rollback the command if it turns out to have not been allowed but I
just don't like it.  Feels like a higher risk solution to me.

I don't see a way to get around the second piece since what information
is needed about the action is typically action-specific.  Perhaps we
could have an 'action-ID' (uh, we have an ID per command already, no?
Probably doesn't fit the mold close enough though), and then a way to
query information about what is this action trying to do?.  Doesn't
seem likely to be very clean though.

Other thoughts?

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Robert Haas robertmh...@gmail.com writes:
 I thought we had an idea of using the AllocSet dispatch mechanism to
 make this zero-overhead in the case where the probes are not enabled.
 What happened to that notion?

I must have missed that discussion, but +1 --- should be possible to get
to zero-overhead-when-off that way.  The trick is to figure out
what/where enables the alternate implementation.  The current design
assumes that the callers of FooContextCreate choose the implementation,
but we don't want that here.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients


Tom Lane wrote:

Also, the reason why Bruce's mistake exposed this is interesting.
Omitting the #define for ENABLE_THREAD_SAFETY does not actually break
pgbench -j at all -- it has a fallback strategy that uses multiple
subprocesses instead of multiple threads.  However, it has only one
srandom() call, which occurs in the parent process before forking.
This means that the subprocesses all start with the same random number
state, which means they generate the same sequence of random account
IDs to update


We just got that as a bug report the other day too, with suggested 
fixes:  http://archives.postgresql.org/pgsql-hackers/2009-12/msg00841.php



I'm inclined to think this is bad, and we should fix pgbench to
re-randomize after forking.  If we don't, we'll have synchronized
behavior on some platforms and not others, which doesn't seem like a
good idea.  On the other hand, if you did want that type of behavior,
it's hard to see how you'd get it.  Is it worth trying to provide that
as a (probably non-default) mode in pgbench?  If so, how would you
do it on threaded platforms?
  
It sounds like random pgbench run for a while would certainly expose the 
same thing you're concerned about eventually.  I doubt it's worth the 
trouble to codify a bug just because it found another bug (it's hard 
enough to maintain code that only has to work right).  If we want to 
stress this behavior, it would be easier to just test with a a bunch of 
clients going through a scale=1 database and use enough transactions to 
make update collisions likely.  I'm working on a guide to stress testing 
new alpha builds with pgbench that will be ready in time for alpha3.  I 
could easily add that as one of the suggested tests to run if you're 
concerned about getting more test coverage of that code path.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients

Greg Smith g...@2ndquadrant.com writes:
 It sounds like random pgbench run for a while would certainly expose the 
 same thing you're concerned about eventually.

Yeah.  Actually the odd thing about it is that the crash seemed to
invariably be on conflicting pgbench_accounts updates, which is a fairly
low-contention table in this test design (but the bug turned it into
high-contention).  What I would have expected is crashes on the very
similar updates to pgbench_branches, which is designed to be
high-contention.  It might be that there is some other effect going on
here that explains why that wasn't happening.  Need to go back and look
more closely.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Tom Lane píše v pá 11. 12. 2009 v 13:56 -0500:
 Robert Haas robertmh...@gmail.com writes:
  As far as I am concerned that is way too much, particularly
  considering that your test case isn't designed to be particularly
  memory-allocation intensive, and if it is up to me I will reject this.
   Even a quarter-percent slowdown for a feature that will be used only
  by a small fraction of users only a small fraction of time time seems
  totally unacceptable to me.
 
 It seems to me that anyone who really needs this can instrument the
 alloc functions anyway --- isn't one of the features of DTrace supposed
 to be that you can monitor calls to a particular function without any
 prearranged code support?  Or is that one of the things like zero
 overhead that turns out to be more marketing-speak than reality?

There are several types of probes. For example for PID provider probes
you can monitor all entry and return point from global function. And
also you can put probe on each asm instruction in the function. These
probes have zero overhead, because dtrace understand ABI and know where
args are.  Unfortunately user defined probes has small overhead which is
price for universal solution which works with all compilers and linkers.

 Anyway I concur with Robert's opinion that the use-case is far too small
 to justify incurring a measurable overhead for everybody.  

OK.

 There might
 be some small argument for putting these in under an extra #ifdef, but
 they wouldn't get into any regular production build.

unfortunately #ifdef solution kills main dtrace goal - without
recompilation :(.

Zdenek


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 1:52 PM, Stephen Frost sfr...@snowman.net wrote:
 * Robert Haas (robertmh...@gmail.com) wrote:
 I actually have an idea how to solve the problem in this particular
 case, but I'm reluctant to say what it is because I'm not sure if I'm
 right, and at any rate *I don't want to write this patch*.

 As far as crap goes, I'd have to put this at the top.  If you're not
 willing to share ideas, then I may have to reconsider my personal
 feeling on if you should be a committer or not.  No one is asking you to
 write the patch.  We all know that we can be wrong (I tend to be more
 wrong than most), and we all hate to jerk people around, but I feel
 it's far worse to self-censor discussion on ideas.

OK, it's clear that I've handled this badly.  Sorry.  My fear (however
unjustified) was that someone would go and rewrite the patch based on
an opinion that I express whether they agree with it or not.  I don't
know the right way to do this and I'm sorry to have given you the
impression that I think I do and am hiding the ball.

So with that said, the idea I had was to try to pass around
pre-existing data structures related to the objects on which access
control decisions are being made, rather than Oids.  It's pretty clear
that you're never going to be able to make an access control decision
just based on the Oid, but the Relation descriptor or pg_something
HeapTuple might be enough - or at least whatever else you need is
likely something you would have had to look up anyway, even without
the interface layer.  I don't know if that makes sense or actually
works, but you could give it a try.

 It's also about the worst form of rock-management that I think I could
 come up with in an open source community.  If you don't share your idea,
 yet you feel that it's right, and see nothing to dissuade you from
 that position (after all, we can't present an argument for or against it
 if we don't know what it is), then I find it likely that you're going to
 constantly be comparing patches presented to the ideal one in your head
 based on your idea and we'd never get there.

It's a little unfortunate that we're arguing about this because that's
exactly what I'm reacting AGAINST, and emphatically not when I intend
to do.  I think this is the first time in my adult life I've been
criticized for being too UN-willing to share my opinions, but I guess
there's a first time for everything.  Again, sorry for handling this
badly.  I just feel like the discussions that we've had so far have
been very much in the dynamic of throw some code over the wall and see
if the committer likes it... looks like no, let's go back around and
try again.  It does have a bit of a rock management feel to it and I
really want to see if we can find a way to break that cycle.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Tom Lane píše v pá 11. 12. 2009 v 14:38 -0500:
 Robert Haas robertmh...@gmail.com writes:
  I thought we had an idea of using the AllocSet dispatch mechanism to
  make this zero-overhead in the case where the probes are not enabled.
  What happened to that notion?
 
 I must have missed that discussion, but +1 --- should be possible to get
 to zero-overhead-when-off that way.  The trick is to figure out
 what/where enables the alternate implementation.  The current design
 assumes that the callers of FooContextCreate choose the implementation,
 but we don't want that here.

I thought about it. I think we can use GUC variable (e.g. dtraced_alloc)
and hook switch pointers to dtraced AsetFunctions. The problem is how to
distribute to all backend.

Zdenek



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

David,

* David P. Quigley (dpqu...@tycho.nsa.gov) wrote:
 So the document I read is linked below [1].

Great, thanks again.

[agree with all the rest]

 It is definitely good to have a second opinion on this since I've just
 only started reading the PCI compliance documents. I'm definitely not an
 expert in PCI compliance but from what I've read there are definite
 benefits for using SEPG or PG-ACE with a special security module in
 making much stronger guarantees about who and what can touch the card
 data.

Indeed.  The other nice piece about getting the opinion of Treat (or
others who have to deal with PCI) is that while the PCI documentation
says what you're supposed to do, the PCI folks also have auditing
requirments (as in, a third-party vendor has to audit your system, and
there are required scans and scanning tools, etc) which don't always
marry up to what they say they require.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

* David P. Quigley (dpqu...@tycho.nsa.gov) wrote:
 Yea I never asked Stephen if he goes by Stephen or Steve when I met him
 on Wednesday. I guess calling him Steve is me being a bit
 presumptuous :)

Oh, either is fine, tho people will probably follow a bit better if you
say Stephen.  As a reminder- KaiGai's the one who did all the actual
code, I just tried to steer him in a direction I thought made sense to
make it easier to get core to accept it, and reviewed the results of his
work.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Zdenek Kotala zdenek.kot...@sun.com writes:
 I thought about it. I think we can use GUC variable (e.g. dtraced_alloc)
 and hook switch pointers to dtraced AsetFunctions. The problem is how to
 distribute to all backend.

You set the GUC in postgresql.conf.  No big deal.

If we go this route it would be nice to think about making a facility
that has some usefulness for non-DTrace platforms too.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

Josh,

* Joshua Brindle (met...@manicmethod.com) wrote:
 Stephen Frost wrote:
 I do think that, technically, there's no reason we couldn't allow for
 multiple only-more-restrictive models to be enabled and built in a
 single binary for systems which support it.  As such, I would make those
 just #if defined() rather than #elif.  Let it be decided at runtime
 which are actually used, otherwise it becomes a much bigger problem for
 packagers too.

 It isn't just a case of using #if and it magically working. You'd need a  
 system to manage multiple labels on each object that can be addressed by  
 different systems. So instead of having an object mapped only to  
 system_u:object_r:mydb_t:s15 you'd also have to have it mapped to,  
 eg., ^ for Smack.

I'm not sure I see that being a problem..  We're going to have
references in our object managers which make sense to us (eg: table
OIDs) and then a way of mapping those to some label (or possibly a set
of labels, as you describe).  We might want to reconsider the catalog
structure a bit if we want to support more than one at a time, but I
don't see it as a huge problem to support more than one label existing
for a given object.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 2:11 PM, Stephen Frost sfr...@snowman.net wrote:
 Second, the information we *don't* have from above is generally
 information about what the requesting action is.  For example, when
 changing ownership of an object, we can't possibly use introspection to
 find out the role which is on the receiving end, since at that time
 we've just learned it from the parser.

I'm not sure that I follow what you're saying here.  I think maybe it
would help to discuss a concrete example, which is why I proposed a
concept patch.  Or perhaps you'd like just to pick out a specific case
to discuss.

 Now, we might build an entire
 new object, pass the result of this action OID to the security system
 and ask it can we change OID X into OID Y?, but I don't particularly
 like it.  We really don't want to do any *changes* to things until after
 we've determined the permissions allow for it, and I'm not sure how to
 get around that without building an independent introspection system for
 what might be.  Perhaps we're comfortable enough saying we'll just
 rollback the command if it turns out to have not been allowed but I
 just don't like it.  Feels like a higher risk solution to me.

Yeah, that seems like a non-starter.

 I don't see a way to get around the second piece since what information
 is needed about the action is typically action-specific.  Perhaps we
 could have an 'action-ID' (uh, we have an ID per command already, no?
 Probably doesn't fit the mold close enough though), and then a way to
 query information about what is this action trying to do?.  Doesn't
 seem likely to be very clean though.

Object creation seems to be one of the tougher cases here.  When
you're altering or dropping an existing object, the decision is likely
to be based (in any system) on the properties of that object.  When
you're creating an object, the decision will of course have to be
based on the properties of something else, but different access
control models might not agree on the value of something else.  It's
not immediately clear to me how to deal with that, but again it's hard
for me to discuss it in the abstract.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] thread safety on clients

I wrote:
 ... What I would have expected is crashes on the very
 similar updates to pgbench_branches, which is designed to be
 high-contention.  It might be that there is some other effect going on
 here that explains why that wasn't happening.  Need to go back and look
 more closely.

... and the answer to that is that pgbench_branches isn't subject to the
bug, because its only pass-by-reference column happens to be filled with
all NULLs by the initialization step, unlike the accounts filler column
which happens to be filled with non-null strings.  Null values mean no
dangling pointers and no chance for a memory management issue.  So you
could have run this all day and never seen a crash on pgbench_branches
updates.  (If you manually set the filler column non-null before
starting a run, the unpatched code crashes instantly, even with a
non-bollixed pgbench.)

So, nothing to see here except lack of test coverage, move along.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Tom Lane píše v pá 11. 12. 2009 v 15:11 -0500:
 Zdenek Kotala zdenek.kot...@sun.com writes:
  I thought about it. I think we can use GUC variable (e.g. dtraced_alloc)
  and hook switch pointers to dtraced AsetFunctions. The problem is how to
  distribute to all backend.
 
 You set the GUC in postgresql.conf.  No big deal.
 
 If we go this route it would be nice to think about making a facility
 that has some usefulness for non-DTrace platforms too.

Do you mean general facility for switching memory allocator?

Zdenek


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

2009-12-11 Thread Stephen Smalley

On Fri, 2009-12-11 at 14:11 -0500, Stephen Frost wrote:
 All,
 
 * Robert Haas (robertmh...@gmail.com) wrote:
  If we design a security abstraction layer, the interfaces need to
  really be abstraction boundaries.  Passing the table OID and then also
  the tablespace OID because PG DAC needs that to make its access
  control decision is crap.  
 
 Now, to address the small useful bit of this mail- I tend to agree with
 this.  I'm not convinced there's an alternative, but I'd like to throw
 out a couple of my ideas on how it could be addressed.  I'd like to
 solicit for feedback on these.
 
 First off, we have alot of the information available from the catalog.
 Basically, given an OID, we should be able to find out the other
 information associated with that OID (such as what kind of object it is,
 what tablespace it resides in, etc).  OID isn't sufficient, however, as
 we know from the dependency system.  To address this, we would need OID
 and SubOID.  Now, any information which we can derive from those should
 not be included in the API.  To be honest, I don't think we've actually
 been all that bad about this, but I'll reserve any definitive answer
 until I've gone back through the API we have specifically thinking about
 this issue.  On further thought, I'm probably wrong and should have
 caught this during my review.  Sorry.
 
 Second, the information we *don't* have from above is generally
 information about what the requesting action is.  For example, when
 changing ownership of an object, we can't possibly use introspection to
 find out the role which is on the receiving end, since at that time
 we've just learned it from the parser.  Now, we might build an entire
 new object, pass the result of this action OID to the security system
 and ask it can we change OID X into OID Y?, but I don't particularly
 like it.  We really don't want to do any *changes* to things until after
 we've determined the permissions allow for it, and I'm not sure how to
 get around that without building an independent introspection system for
 what might be.  Perhaps we're comfortable enough saying we'll just
 rollback the command if it turns out to have not been allowed but I
 just don't like it.  Feels like a higher risk solution to me.
 
 I don't see a way to get around the second piece since what information
 is needed about the action is typically action-specific.  Perhaps we
 could have an 'action-ID' (uh, we have an ID per command already, no?
 Probably doesn't fit the mold close enough though), and then a way to
 query information about what is this action trying to do?.  Doesn't
 seem likely to be very clean though.
 
 Other thoughts?

In the Linux LSM case, we define a separate hook interface for each
logical operation/action, where hook name identifies the
action/operation and the set of arguments to that hook interface are the
complete set of inputs that may be relevant to deciding whether to
permit the operation/action.   These arguments typically include
pointer(s) to one or more object data structures as well as ancillary
arguments (e.g. new owner value if chown).  Reference:
http://www.usenix.org/event/sec02/wright.html
http://lxr.linux.no/#linux+v2.6.32/include/linux/security.h

The XACE framework for the X server is described by:
http://www.x.org/releases/X11R7.5/doc/security/XACE-Spec.html
Unlike the LSM framework, it passes resource identifiers rather than
direct object pointers.

The FreeBSD MAC framework is described by:
http://www.freebsd.org/doc/en/books/arch-handbook/mac.html
It provides more abstraction than LSM does, at a cost in the overhead of
the framework itself.

-- 
Stephen Smalley
National Security Agency


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] random() in multi-threaded pgbench

Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes:
 While testing the pgbench setshell command patch with -j option,
 I found all threads use the same sequence of random value.

Were they actually threads, or were you testing the code while it had
the broken configure script that didn't set ENABLE_THREAD_SAFETY?
I think you might have hit the same thing I just ran into, that in
a *non-threaded* build each subprocess will generate the same random
sequence.

 At first, I think we need to call srandom() in each thread,

Each sub-job I think.

 but the manual says we should use random_r() instead of random()
 on multi-threaded programs.
 http://www.kernel.org/doc/man-pages/online/pages/man3/random_r.3.html

It only says that you need those if you want an *independent* random
sequence for each thread.  pgbench never had that before and I doubt
we need it now.  In any case, the same page also says these are a
glibc-ism not a standard API, so we can't really rely on them.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

* Robert Haas (robertmh...@gmail.com) wrote:
 OK, it's clear that I've handled this badly.  Sorry.  My fear (however
 unjustified) was that someone would go and rewrite the patch based on
 an opinion that I express whether they agree with it or not.

That's always going to be a risk in an open-discussion environment.
Additionally, that's exactly what happened to me last go round- KaiGai
rewrote the patch based on my ideas and suggestions, and the result was
summarairly tossed out by Tom.  Did it suck?  Yes, heavily, and it
frustrated me to the point that I specifically asked to not be the
reviewer for SEPG during the next commitfest.  At the same time,
what KaiGai or others spend time on is up to them (and/or their
employers).

I sincerely hope that even if you suggest an approach down the road
unrelated to this on some other patch you're reviewing, and then you see
the results and say whoah, that's horrible, and should never be
committed, that you understand none of us would want you to commit it.
Sharing your ideas or putting out suggestions isn't a commitment on your
part that you'll commit the results when someone else rights it.  Heck,
I bet you've been down that road on your own projects and come to the
realization at the end of err, bad idea and not committed it.

Allow me to say, my apologies, I feel like I may have over-reacted a bit
for my part as well.

 So with that said, the idea I had was to try to pass around
 pre-existing data structures related to the objects on which access
 control decisions are being made, rather than Oids.  

That thought had crossed my mind as well, but I wasn't convinced that
would actually be seen as a signifigantly different API to just having
the arguments passed inline...  Then again, using structures does
allow you to add to them without having to modify the function
definitions, and would allow David's suggestion of using function
pointers to work, which we do in some other specific cases.  I guess I'm
curious if we (PG) have any particular feeling one way or the other
about function pointers; I just don't recall seeing them used terribly
often and would worry about that they might be passively discouraged?

 It does have a bit of a rock management feel to it and I
 really want to see if we can find a way to break that cycle.

Agreed.  It's been a point of frustration for me, but I've been trying
to work with it so long as we at least get some constructive critisim
back (Tom's review of the patch I reviewed fell into the questionable
category for me on that call, which is what really frustrated me the
most about it).  A cyclic approach is typical in all software
development, it's when information stops flowing about why something
doesn't meet expectations or requirments that progress breaks down.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Zdenek Kotala zdenek.kot...@sun.com writes:
 Tom Lane pÃÅ¡e v pÃ¡ 11. 12. 2009 v 15:11 -0500:
 If we go this route it would be nice to think about making a facility
 that has some usefulness for non-DTrace platforms too.

 Do you mean general facility for switching memory allocator?

No, I was thinking of some sort of memory allocation stats collection
that doesn't depend on DTrace.  It's amazing to me that we've never
gone back and improved on the original quick-and-dirty
MemoryContextStats mechanism.  I certainly find myself using that a
lot for issues like tracking down memory leaks.  While palloc has a lot
of advantages, the fact that you can't easily plug in a debug-friendly
substitute malloc package is not one of them :-(

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] random() in multi-threaded pgbench

I wrote:
 Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes:
 http://www.kernel.org/doc/man-pages/online/pages/man3/random_r.3.html

 It only says that you need those if you want an *independent* random
 sequence for each thread.  pgbench never had that before and I doubt
 we need it now.  In any case, the same page also says these are a
 glibc-ism not a standard API, so we can't really rely on them.

... although if anyone was sufficiently excited about this, we could
make use of erand48() without adding any new platform dependency,
since GEQO is already relying on that.  I think there's not much
point though.  We'd just be moving the nearest point of failure to
the seeding algorithm --- careless seeding could still result in
duplicate sequences for different threads.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

* Robert Haas (robertmh...@gmail.com) wrote:
 On Fri, Dec 11, 2009 at 2:11 PM, Stephen Frost sfr...@snowman.net wrote:
  Second, the information we *don't* have from above is generally
  information about what the requesting action is.  For example, when
  changing ownership of an object, we can't possibly use introspection to
  find out the role which is on the receiving end, since at that time
  we've just learned it from the parser.
 
 I'm not sure that I follow what you're saying here.  I think maybe it
 would help to discuss a concrete example, which is why I proposed a
 concept patch.  Or perhaps you'd like just to pick out a specific case
 to discuss.

Hrm, I thought I had given a specific example.  Didn't do a good job of
it, apparently.  Let me try to be a bit more clear:

ALTER TABLE x OWNER TO y;

If given the table OID, there's a ton of information we can then pull
about the table- the tablespace, the owner, the schema, the columns, the
privileges, etc, etc.

What we can't possibly figure out from the OID is the value of y.  Yet,
in the PG security model, the value of y matters!  You have to know what
y is to check if y has 'create' rights on the schema.  If it doesn't
(and the user executing the command isn't the superuser) then the
request (under the PG model) is denied.

Does that help clarify my example case?

 Object creation seems to be one of the tougher cases here.  When
 you're altering or dropping an existing object, the decision is likely
 to be based (in any system) on the properties of that object.  When
 you're creating an object, the decision will of course have to be
 based on the properties of something else, but different access
 control models might not agree on the value of something else.  It's
 not immediately clear to me how to deal with that, but again it's hard
 for me to discuss it in the abstract.

That sounds like a pretty good example to me too, to be honest.  It goes
back to the same issue though- that's information you get from the
command that's trying to be run and isn't available from existing
objects.  Using structures to track this information, allowing the
function arguments to remain identical, is certainly something which can
be done, and probably done with a minimal amount of fuss.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] SE-PostgreSQL/Lite Review

2009-12-11 Thread Robert Treat

On Thursday 10 December 2009 21:47:18 KaiGai Kohei wrote:
 Greg Smith wrote:
  It's funny; we started out this CommitFest with me scrambling to find
  someone, anyone, willing to review the latest SE-PostgreSQL patch,
  knowing it was a big job and few were likely to volunteer.  Then
  schedules lined up just right, and last night I managed to get a great
  group of people all together to do perhaps the biggest single patch
  review ever, to work on just that.I gathered up a list of the
  biggest concerns about this feature and its associated implementation,
  we got a number of regular PostgreSQL hackers and two of the security
  guys you've seen on this list all in the same room, and we talked about
  little but SEPostgreSQL for hours.  Minutes are at
  http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG and I'd
  suggest anyone interested in this feature (or in rejecting this feature)
  to take a look at what we covered.

 I repent that I cannot attend here.
 The wikipage is a good summarize. Thanks for your efforts.

  There's comments there, with references for you [citation needed] types,
  to help answer four of the most common complaints I've seen on this list
  about this feature:
 
  -Is there really a user base for it?
  -Has it been audited by security professionals?
  -Is there a useful SELinux policty to support it?
  -Will this work with other security frameworks?
 
  I feel pretty good now that these are not really our community's
  problems--these have had at least modest, and in some cases extensive,
  due diligence applied to them.  And we've confirmed there's access to
  expertise from the security community to help out with remaining
  concerns there, in person even if we plan it out right.  I personally
  suspect they've been discouraged from getting involved more by the slow
  pace this has been proceeding within our community and the general
  disarray around it, which would be understandable.

 IMO, the slow pace is not a primary reason. In fact, SELinux was released
 at 2000 Dec, but it gets integtated into the mainline kernel at 2003 Aug
 with various kind of improvements. It takes about 3 years from the first
 relase. IIRC, now we take 2.5 years from the first announce of SE-PgSQL
 in this list, and various kind of improvements and cleanups had been done.
 It is a bit long term, but not too long term.

 The reason of this gap is that people have individual consciousness about
 their security. I often represent it as security is a subjective term.
 Needless to say, we don't have magic-bullets for any threats. Any
 technology has its metirs and limitations. However, people tend to say it
 is nonsense, if the feature does not match with their recognizable demands
 or threats.

 Security-folks know MAC is not magic-bullets, while it is a significant
 piece of system security. But some of complaints might be pointless for
 security experts, so had been dismotivated.
 From the perspective of security folks, we have to introduce it doggedly.
 And, I'd like to understand database folks there are various kind of
 security models and viewpoints here. SELinux is one of them.

  The parts I do believe we have reason to be concerned are with the code
  integration into the PostgreSQL core, and no one has any easy answers to
  things like isn't this going to increase CERT advisories? and who's
  going to maintain this besides KaiGai?  I personally feel that Steven
  Frost's recent comments here about how the PostgreSQL code makes this
  harder than it should be really cuts to the core of a next step here.
  The problem facing us isn't is SEPostgreSQL the right solution for
  providing external security checks?; it's how can the PostgreSQL code
  be improved so that integrating external security is easier?  Looking
  at SEPostgreSQL is great because it really highlights where the existing
  set of places are.  This general idea matches where thinking on things
  like row-level security was already going too--implement this for the
  database in general, then link SEPostgres in as just one provider of a
  security restriction.

 Right, it seems to me the security provider is a good phrase to represent
 this feature. It just provides additional access control decisions based on
 the different perspective of security model.

 Please note that the access control granularity is not an essential issue.
 We can also assume table-level mandatory access controls for instance.

 The latest patch provides table/column level controls without row-level,
 because the current PgSQL has facilities to know what tables and columns
 are referenced reasonably, so SE-PgSQL also can know what tables/columns
 are referenced without special tricks.

 Please remind the earlier SE-PgSQL in v8.2.x. It walked on the Query tree
 to pick up what columns were accessed.

  I hope the review from the BWPUG helps everyone out, and that the
  suggestions on the wiki for the Follow-up plan are helpful.  As CF

Re: [HACKERS] Adding support for SE-Linux security

Stephen (great name!),

* Stephen Smalley (s...@tycho.nsa.gov) wrote:
 Reference:
 http://www.usenix.org/event/sec02/wright.html
 http://lxr.linux.no/#linux+v2.6.32/include/linux/security.h
 
 The XACE framework for the X server is described by:
 http://www.x.org/releases/X11R7.5/doc/security/XACE-Spec.html
 
 The FreeBSD MAC framework is described by:
 http://www.freebsd.org/doc/en/books/arch-handbook/mac.html

Thanks for these pointers!  I'm sure KaiGai has probably already done a
review of these, but I'll take a look and would ask others who are
interested to please do the same.  Seeing the different options that
people have gone with (as opposed to just the SELinux/kernel version)
will undoubtably be helpful in deciding the best approach forward.

Thanks again!

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 3:28 PM, Stephen Frost sfr...@snowman.net wrote:
 I sincerely hope that even if you suggest an approach down the road
 unrelated to this on some other patch you're reviewing, and then you see
 the results and say whoah, that's horrible, and should never be
 committed, that you understand none of us would want you to commit it.

I have to thank you for saying this - unfortunately, I don't think
everyone takes this approach.  As you can probably figure out, my
alleged rock management upthread was actually a poorly executed
attempt to avoid being accused of bait-and-switch.  If I don't tell
you how to write the patch, you can't accuse me of moving the
goalposts (of course I've now discovered the pitfalls of that approach
as well...).

 Sharing your ideas or putting out suggestions isn't a commitment on your
 part that you'll commit the results when someone else rights it.  Heck,
 I bet you've been down that road on your own projects and come to the
 realization at the end of err, bad idea and not committed it.

Well, I haven't been a committer long enough to have gone through that
precise process, but sure, I've tossed out ideas when they don't turn
out to be good.

 So with that said, the idea I had was to try to pass around
 pre-existing data structures related to the objects on which access
 control decisions are being made, rather than Oids.

 That thought had crossed my mind as well, but I wasn't convinced that
 would actually be seen as a signifigantly different API to just having
 the arguments passed inline...

If you'll forgive me for saying so, this is exactly the sort of
thinking that I think is killing us.  Who cares how it will be seen?
Seen by whom?  We shouldn't be writing this code to be seen - we
should be writing it to be good.  If doing this makes a clean, tight
abstraction layer, then it's a good design.  If it doesn't, then it
sucks.  I realize that opinions enter into this at some level, but
let's try to proceed as though there's a technically right answer out
there and bend our best efforts to finding it.

 Then again, using structures does
 allow you to add to them without having to modify the function
 definitions,

Personal opinion time, but I think that without having to modify the
function definitions is a CRITICAL design component.  Furthermore,
adding to them [the structures] shouldn't be something that only
happens when we decide to support a new security model, or we're
little better off than if we just passed a kajillion arguments.
Again, I don't want to overstate my confidence in this point, but it
feels to me like we need to pass PRE-EXISTING data structures that are
already being used for other things and happen to be the right stuff
to enable access control decisions, and to which fields that are
likely to be needed are likely to get added anyway.  But it's
difficult to know whether this approach can be made to work without
trying it, and there are bound to be problem cases that need to be
thought about, and that thinking will be more likely to lead to a good
result if it happens in the community, rather than by KaiGai or any
other single person in isolation.

 and would allow David's suggestion of using function
 pointers to work, which we do in some other specific cases.  I guess I'm
 curious if we (PG) have any particular feeling one way or the other
 about function pointers; I just don't recall seeing them used terribly
 often and would worry about that they might be passively discouraged?

I'm going to vote fairly strongly against inserting function pointers
at the outset.  I think that we should look at the first phase of this
project as an attempt to restructure the code so that the access
control decisions are isolated from the rest of the code.  *If* we can
do that successfully, I think it will represent good progress all on
its own.  Inserting function pointers is something that can be done
later if it turns out to be useful with a very small, self-contained
patch.

 It does have a bit of a rock management feel to it and I
 really want to see if we can find a way to break that cycle.

 Agreed.  It's been a point of frustration for me, but I've been trying
 to work with it so long as we at least get some constructive critisim
 back (Tom's review of the patch I reviewed fell into the questionable
 category for me on that call, which is what really frustrated me the
 most about it).  A cyclic approach is typical in all software
 development, it's when information stops flowing about why something
 doesn't meet expectations or requirments that progress breaks down.

Part of my frustration here is that I don't see a lot of evidence that
anyone is willing to put really tangible resources into this other
than KaiGai and his employer.  I don't want to keep reviewing the same
patches over again just because they keep getting resubmitted with
various modifications.  I am willing to try to be helpful with this
project if that is something that you

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote:
 Hrm, I thought I had given a specific example.  Didn't do a good job of
 it, apparently.  Let me try to be a bit more clear:

 ALTER TABLE x OWNER TO y;

 If given the table OID, there's a ton of information we can then pull
 about the table- the tablespace, the owner, the schema, the columns, the
 privileges, etc, etc.

 What we can't possibly figure out from the OID is the value of y.  Yet,
 in the PG security model, the value of y matters!  You have to know what
 y is to check if y has 'create' rights on the schema.  If it doesn't
 (and the user executing the command isn't the superuser) then the
 request (under the PG model) is denied.

 Does that help clarify my example case?

That case doesn't seem terribly problematic to me.  It seems clear
that we'll want to pass some information about both x and y.  What is
less clear is exactly what the argument types will be, and the right
answer probably depends somewhat on the structure of the existing
code, which I have not looked at.  What I'm more concerned about is if
the access control decision in this case were based on u for PG DAC, v
for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System.
If that's the case, and our function signature looks like (x,y,u,v,w),
the we should worry.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

* Robert Haas (robertmh...@gmail.com) wrote:
 If I don't tell
 you how to write the patch, you can't accuse me of moving the
 goalposts (of course I've now discovered the pitfalls of that approach
 as well...).

Indeed, we also yell and scream when we don't know which direction the
goalposts are in. ;)

  That thought had crossed my mind as well, but I wasn't convinced that
  would actually be seen as a signifigantly different API to just having
  the arguments passed inline...
 
 If you'll forgive me for saying so, this is exactly the sort of
 thinking that I think is killing us.  Who cares how it will be seen?

erp. mea culpa.  I meant that I didn't think of it as being enough of a
novelty for you to be suggesting-it-but-not-telling-us..  It just kind
of came off as too-obvious, however, the reality is that I didn't
understand your suggestion, see below.

  Then again, using structures does
  allow you to add to them without having to modify the function
  definitions,
 
 Personal opinion time, but I think that without having to modify the
 function definitions is a CRITICAL design component.  Furthermore,
 adding to them [the structures] shouldn't be something that only
 happens when we decide to support a new security model, or we're
 little better off than if we just passed a kajillion arguments.
 Again, I don't want to overstate my confidence in this point, but it
 feels to me like we need to pass PRE-EXISTING data structures that are
 already being used for other things and happen to be the right stuff
 to enable access control decisions, and to which fields that are
 likely to be needed are likely to get added anyway.

Ah, now that makes alot of sense..  Unfortunately, having been involved
in at least some of this code in the past, sadly I don't think we have
such pre-existing data structures for the information that's primairly
at issue here.  Specifically, the data structures we tend to have
pre-existing are the ones that come from the catalog and would fall out
from an OID+SubOID based API.  Sure, we could pass those structures in
instead of using SysCache to pull the data out, but that's not really
how we 'typically' do things in the code base from my experience, and I
don't really see it having alot of advantage.  Using OID+SubOID and
SysCache *will* pick up new things as they're added to the catalogs- for
the information *in* the catalogs.

There are very few data structures outside of the parse tree which
include the information from the parse tree..  And to be honest, the
ones that *are* out there don't typically have the world's best
structure for use by this kind of a framework (thinking back to
specifically what's passed around for column-level privs..).

 But it's
 difficult to know whether this approach can be made to work without
 trying it, and there are bound to be problem cases that need to be
 thought about, and that thinking will be more likely to lead to a good
 result if it happens in the community, rather than by KaiGai or any
 other single person in isolation.

I agree with this- one issue is, unfortunately, an overabundance from
KaiGai of code-writing man-power.  This is an odd situation for this
community, in general, so we're having a hard time coming to grasp with
it.  KaiGai, can you hold off on implementing any of these approaches
until we can hammer down something a bit better for you to work from as
a baseline?  I'll start with Robert's suggestion of a single-object
example case and throw out some example code for people to shoot down of
different approachs.  After a few iterations of that, with comments from
all (KaiGai, you're welcome to comment on it too, of course), we'll turn
you loose on it again to implement fully (if you're still willing to).

Following along that though, as we'd like this to be the design forum,
when you come across problems or issues implementing it, could you come
back to this forum and explain the issue and why it doesn't fit, as soon
as you hit it?  What you tend to do is disappear for a while, then come
back, instead, with a full patch which works, but has places where
you've gone down an unexpected path because you found a case which
wasn't covered, designed a solution for it which kinda fits and then
implemented it immediately.

I feel that's something I've encouraged you to do, unfortunately, and my
apologies for that.  I'm trying to get time from my employer (and my
wife) to dedicate to this, to hopefully avoid that happening in the
future.

 I'm going to vote fairly strongly against inserting function pointers
 at the outset.  I think that we should look at the first phase of this
 project as an attempt to restructure the code so that the access
 control decisions are isolated from the rest of the code.  *If* we can
 do that successfully, I think it will represent good progress all on
 its own.  Inserting function pointers is something that can be done
 later if it turns out to be useful with a very small, self-contained
 patch.

Re: [HACKERS] Adding support for SE-Linux security

* Robert Haas (robertmh...@gmail.com) wrote:
 On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote:
  Does that help clarify my example case?
 
 That case doesn't seem terribly problematic to me.  It seems clear
 that we'll want to pass some information about both x and y.  What is
 less clear is exactly what the argument types will be, and the right
 answer probably depends somewhat on the structure of the existing
 code, which I have not looked at.

Allow me to assist- y is never in a structure once you're out of the
parser:

ATExecChangeOwner(Oid relationOid, Oid newOwnerId, bool recursing)

I expect you'll find this is more the rule than the exception to alot of
the existing PG security model, because much of it's responsibility is
in what I'll call the DDL (under commands/) aspects.  The DML pieces
(under the executor) are a bit better about this, specifically you could
pass in a RangeTblEntry, exactly how ExecCheckRTEPerms handles it.

Actually, in a fit of barely-contained mirth, it strikes me that PG
really has already done what you're suggesting for the 'hard' part- and
the RangeTblEntry plus ExecCheckRTEPerms is exactly it.  It's all the
*other* checks we do for the PG security model, under commands/, that
are the problem here.  The parts of the code that, to be honest, I think
all us database geeks have historically cared alot less about.

 What I'm more concerned about is if
 the access control decision in this case were based on u for PG DAC, v
 for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System.
 If that's the case, and our function signature looks like (x,y,u,v,w),
 the we should worry.

Right, I understand that.  What I offer in reply is that we focus our
attention on using the OID+SubOID approach, as I'm suggesting, to the
fullest extent possible through that mechanism, and appreciate that the
other arguments passed to the function are derived specifically from the
parser and therefore unlikely to be changed until and unless we change
the base syntax of the command and calling function, at which time we
may have to add arguments to the function signature.  This would
continue at least until we get to the point where we decide that the
caller needs to be changed because it's got a huge function sig, and
move it to something like the structure of the executor and the
equivilant of ExecCheckRTEPerms() would get updated along with it, at
that time.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] Adding support for SE-Linux security


Stephen Frost wrote:

I agree with this- one issue is, unfortunately, an overabundance from
KaiGai of code-writing man-power.  This is an odd situation for this
community, in general, so we're having a hard time coming to grasp with
it.
There are plenty of parallels to when Zdenek was writing a ton of 
in-place upgrade code faster than anyone else was fully consuming it.  
The do it right or don't do it at all approach of the PG community 
seems particularly hard to reconcile with larger patches from people we 
don't get enough face time with.  It's easy to get deadlocked and not 
have a good way to navigate out when faced with a set of difficult 
decisions and only electronic communications between participants.  
Shoot, you and Robert have spent time doing technical arguments in 
person and we still got a
little rough patch on-list this week out of the debate.  I hate to even 
use this terminology, but these big patches seem to need a project 
manager advocate sometimes:  someone who knows everyone well enough to 
clear these stalled spots, smooth over any personality conflicts, and is 
motivated to intervene because they need the feature.  I see it as a 
sort of scaling problem that might be a recurring one.  It would be nice 
if we could all get better at identifying when it does happen, and 
perhaps find someone to help with planning before we waste so much time 
chasing code that isn't going to be accepted yet again.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager


Tom Lane wrote:

It's amazing to me that we've never
gone back and improved on the original quick-and-dirty
MemoryContextStats mechanism.  I certainly find myself using that a
lot for issues like tracking down memory leaks.
That code hasn't really gone anywhere since Neil tweaked the indentation 
two years ago.  What sorts of improvements were you looking for?


I started on trying to improve this area at one point but didn't get 
far.  My first step was going to be just wrapping the call into a UDF to 
make it easier to reach without having to set loose gdb.  I thought that 
might expand the possible uses for MemoryContextStats to help find a 
memory leak safely on the production box, and therefore attact more 
attention to improving it.  People really don't like running gdb on 
production, but a UDF dumping the same data wouldn't seem so risky.  As 
Daniel Farina pointed out to me one day in a duh moment, that idea is 
quite obviously doomed by the fact that people want dumps from processes 
that you won't be talking to in a UDF context.  You won't be able to 
find a leak in the background writer by dumping the context the UDF can 
see.  There would need to be some sort of IPC/signaling mechanism in 
each process if you wanted it to work everywhere, and once that 
realization hit I realized this was a deeper problem than it appeared.


If you want to make it easier to export into user space, it seems to me 
the memory context stats data either needs to get pushed somewhere it's 
easier to get to (the way the regular stats are), or there needs to be 
some way of polling any random PG process to find it--presumably by 
passing the request the pid you want to instrument.  Neither of which 
may really be a reasonable idea at all, particularly since that in most 
cases, you're using a debugger to track your leak down only after 
reproducing a test case for it.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

I just did a round of integrating some of the big-picture feedback that 
has shown up here since the meeting into 
http://wiki.postgresql.org/wiki/SEPostgreSQL_Review_at_the_BWPUG , 
mainly supplementing the references in the Works outside of SELinux 
section with the new suggested reading here suggested by Stephen Smalley 
and Joshua Brindle.  I'm trying to keep that a fairly readable intro to 
the controversial parts rather than going deeply technical. 

What I'm not going to try to track is all the low-level implementation 
details that are bouncing around right now, my brain is too full this 
week to cram more about OID trivia into it right now.  That would be a 
good idea for someone to summarize eventually and then throw that onto 
the wiki somewhere else, so that it's easier to remember the context of 
what/why decisions were made.  The way Simon has been keeping an ongoing 
log at http://wiki.postgresql.org/wiki/Hot_Standby shows a reasonable 
way to organize such a thing from a similarly complicated patch.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 5:36 PM, Stephen Frost sfr...@snowman.net wrote:
 * Robert Haas (robertmh...@gmail.com) wrote:
 On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote:
  Does that help clarify my example case?

 That case doesn't seem terribly problematic to me.  It seems clear
 that we'll want to pass some information about both x and y.  What is
 less clear is exactly what the argument types will be, and the right
 answer probably depends somewhat on the structure of the existing
 code, which I have not looked at.

 Allow me to assist- y is never in a structure once you're out of the
 parser:

Well this is why you're writing the patch and not me.  :-)

 ATExecChangeOwner(Oid relationOid, Oid newOwnerId, bool recursing)

 I expect you'll find this is more the rule than the exception to alot of
 the existing PG security model, because much of it's responsibility is
 in what I'll call the DDL (under commands/) aspects.  The DML pieces
 (under the executor) are a bit better about this, specifically you could
 pass in a RangeTblEntry, exactly how ExecCheckRTEPerms handles it.

 Actually, in a fit of barely-contained mirth, it strikes me that PG
 really has already done what you're suggesting for the 'hard' part- and
 the RangeTblEntry plus ExecCheckRTEPerms is exactly it.  It's all the
 *other* checks we do for the PG security model, under commands/, that
 are the problem here.  The parts of the code that, to be honest, I think
 all us database geeks have historically cared alot less about.

Hmm, interesting.

 What I'm more concerned about is if
 the access control decision in this case were based on u for PG DAC, v
 for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System.
 If that's the case, and our function signature looks like (x,y,u,v,w),
 the we should worry.

 Right, I understand that.  What I offer in reply is that we focus our
 attention on using the OID+SubOID approach, as I'm suggesting, to the
 fullest extent possible through that mechanism, and appreciate that the
 other arguments passed to the function are derived specifically from the
 parser and therefore unlikely to be changed until and unless we change
 the base syntax of the command and calling function, at which time we
 may have to add arguments to the function signature.  This would
 continue at least until we get to the point where we decide that the
 caller needs to be changed because it's got a huge function sig, and
 move it to something like the structure of the executor and the
 equivilant of ExecCheckRTEPerms() would get updated along with it, at
 that time.

What exactly do you mean by a SubOID?  I'm not really following that part.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

Stephen Frost wrote:
 In my cosmetic preference, ace_ is better than ac_. The 'e' means
 extendable, and ace feels like something cool. :-)
 
 No complaints here..  I just hope this doesn't end up being *exactly*
 the same as your original PGACE patches..  I'd feel terrible if we
 weren't able to at least improve something with this extremely long and
 drawn our process!

Please forget old PGACE. We're talking about our future. :-)

 And I'd like to store these files in backend/security/ace/*.c, because
 backend/security will also store other security modules!
 
 This is perfectly reasonable in my view.  No complaints here.
 
 src/
  + backend/
 + security/
+ ace/ ... access control framework
+ sepgsql/ ... selinux specific code
+ smack/   ... (upcoming?) smack specific code
+ solaristx/   ... (upcoming?) solaris-tx specific code
 
 Looks good to me.  Perhaps we'll have a smack/ patch showing up very
 shortly as well..

It's a welcome feature.

 The reason why I prefer the default PG check + one enhanced security at
 most is simplification of the security label management.
 If two label-based enhanced securities are enabled at same time,
 we need two field on pg_class and others to store security context.
 
 Ah, yes, I see your point must more clearly now.  This sounds reasonable
 to me.
 
 In addition, OS allows to choose one enhanced security at most eventually.

 In my image, the hook should be as:

   Value *
   ac_database_create([arguments ...])
   {
   /*
* The default PG checks here.
* It is never disabled
*/

   /* enhanced security as follows */
   #if defined(HAVE_SELINUX)
   if (sepgsql_is_enabled())
   return sepgsql_database_create(...);
   #elif defined(HAVE_SMACK)
   if (smack_is_enabled())
   return smack_database_create(...);
   #elif defined(HAVE_SOLARISTX)
   if (soltx_is_enabled())
   return soltx_database_create(...);
   #endif
   return NULL;
   }

 We can share a common image image?
 
 If all of these security modules make sense as only-more-restrictive,
 then I have no problem with this approach.  Honestly, I'm fine with the
 initial hooks looking as above in any case, since it provides a clear
 way to deal with switching out the 'default PG checks' if someone
 desires it- you could #if around that block as well.  As it's the
 principle/primary, and I could see only-more-restrictive being more
 popular, I don't mind having that code in-line in the hook.  The check
 itself is still being brought out and into the security/ framework.

Ahh, indeed, #elif does not allow a single binary to provide multiple
options. You are right.

I'd like to rewrite it as follows:

  Value *
  ac_database_create([arguments ...])
  {
  Value *retval = NULL;

  /*
   * The default PG checks here.
   * It is never disabled
   */

  switch (ace_feature)
  {
  #ifdef HAVE_SELINUX
  case ACE_SELINUX_SUPPORT:
  if (sepgsql_is_enabled())
  retval = sepgsql_database_create(...);
  break;
  #endif
  #ifdef HAVE_SMACK
  case ACE_SMACK_SUPPORT:
  if (smack_is_enabled())
  retval = smack_database_create(...);
  break;
  #endif
  #ifdef HAVE_SOLARISTX
  case ACE_SOLARISTX_SUPPORT:
  if (soltx_is_enabled())
  retval = soltx_database_create(...);
  break;
  #endif
  default:
  break;
  }

  return retval;
  }

 I do think that, technically, there's no reason we couldn't allow for
 multiple only-more-restrictive models to be enabled and built in a
 single binary for systems which support it.  As such, I would make those
 just #if defined() rather than #elif.  Let it be decided at runtime
 which are actually used, otherwise it becomes a much bigger problem for
 packagers too.

The reason why I don't want to support multiple enhanced securities at
same time is complexity of security label management, not access control
model.

Unlike X-server, RDBMS have to manage the security label of database
object persistently on the disk. It naturally needs enhancement of data
structure, such as pg_class.relsecon field in the latest SE-PgSQL/Lite.

If the third enhanced security does not use security label, basically,
I think it is feasible. In fact, the default PG check is a security
provider without security label, and it can coexist with SELinux.

Thanks,
-- 
KaiGai Kohei kai...@kaigai.gr.jp

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SE-PostgreSQL/Lite Review

Stephen Frost wrote:
 Josh,
 
 * Joshua Brindle (met...@manicmethod.com) wrote:
 Stephen Frost wrote:
 I do think that, technically, there's no reason we couldn't allow for
 multiple only-more-restrictive models to be enabled and built in a
 single binary for systems which support it.  As such, I would make those
 just #if defined() rather than #elif.  Let it be decided at runtime
 which are actually used, otherwise it becomes a much bigger problem for
 packagers too.
 It isn't just a case of using #if and it magically working. You'd need a  
 system to manage multiple labels on each object that can be addressed by  
 different systems. So instead of having an object mapped only to  
 system_u:object_r:mydb_t:s15 you'd also have to have it mapped to,  
 eg., ^ for Smack.
 
 I'm not sure I see that being a problem..  We're going to have
 references in our object managers which make sense to us (eg: table
 OIDs) and then a way of mapping those to some label (or possibly a set
 of labels, as you describe).  We might want to reconsider the catalog
 structure a bit if we want to support more than one at a time, but I
 don't see it as a huge problem to support more than one label existing
 for a given object.

If we allow multiple security labels on a database object, we have to
expand the structure of system catalog whenever a new security feature
will come in. I think it against to the purpose of the framework.

Even if we store them external relations to reference the object by OID,
we have to provide multiple interface to import/export a security label
for each enhanced securities. For example, it requires much complex patch
to the pg_dump.

My preference is all the enhanced securities shares a common facility
to manage security label, a common statement support and a common
backup/restore support.

Thanks,
-- 
KaiGai Kohei kai...@kaigai.gr.jp

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Robert Haas robertmh...@gmail.com writes:
 What exactly do you mean by a SubOID?  I'm not really following that part.

I assume he's talking about the object reference representation used in
pg_depend, which is actually class OID + object OID + sub-object ID.
The only object type that has sub-objects at the moment is tables,
wherein the sub-objects are columns and the sub-object IDs are column
numbers.  The sub-object ID is zero for all other cases.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security


Robert Haas wrote:

On Fri, Dec 11, 2009 at 4:26 PM, Stephen Frost sfr...@snowman.net wrote:

Hrm, I thought I had given a specific example.  Didn't do a good job of
it, apparently.  Let me try to be a bit more clear:

ALTER TABLE x OWNER TO y;

If given the table OID, there's a ton of information we can then pull
about the table- the tablespace, the owner, the schema, the columns, the
privileges, etc, etc.

What we can't possibly figure out from the OID is the value of y.  Yet,
in the PG security model, the value of y matters!  You have to know what
y is to check if y has 'create' rights on the schema.  If it doesn't
(and the user executing the command isn't the superuser) then the
request (under the PG model) is denied.

Does that help clarify my example case?


That case doesn't seem terribly problematic to me.  It seems clear
that we'll want to pass some information about both x and y.  What is
less clear is exactly what the argument types will be, and the right
answer probably depends somewhat on the structure of the existing
code, which I have not looked at.  What I'm more concerned about is if
the access control decision in this case were based on u for PG DAC, v
for SE-PostgreSQL, and w for Robert Haas's Personal Defensive System.
If that's the case, and our function signature looks like (x,y,u,v,w),
the we should worry.


Theoretically, (x,y,u) - (x,y,u,v) - (x,y,u,v,w) may happen.

However, if the new suggested security model is too far from the known
security model, we should consider whether it performs correctly at first
before commit it.
As long as we describe a security model corresponding to database objects,
it is hard to consider a possibility of infinite increasing in the future.

Thanks,
--
KaiGai Kohei kai...@kaigai.gr.jp

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] status update on Hot Standby and Streaming Replication

I'm not quite sure where we stand with these two patches.  My
impression is that many of the outstanding TODO items in Hot Standby
have been fixed, and I'm not sure what remains.  Streaming Replication
I think reviewing is not as far advanced, but I'm not sure.  Any
chance that Hot Standby can get committed in the next few days before
we bundle alpha3?  Otherwise, I think it will not get as much test
coverage as it really deserves.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] LDAP where DN does not include UID attribute

On Sun, Dec 6, 2009 at 11:29 PM, Greg Smith g...@2ndquadrant.com wrote:
 Magnus Hagander wrote:

 On Sun, Nov 29, 2009 at 13:05, Magnus Hagander mag...@hagander.net wrote:


 I'll be happy to work on this to get it ready for commit, or do you
 want to do the updates?


 Here's a patch with my work so far. Major points missing is the
 sanitizing of the username and the docs.


 It looks like you did an initial review here already.  Since we haven't
 heard from Robert in a while and you seem interested in the patch, I just
 updated this one to list you as the committer and marked it ready for
 committer.  You can commit it or bounce it from there, but it's obvious
 none of our other reviewers are going to be able to do anything with it.

I think we should mark this returned with feedback, since it sounds
like there are still open items.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Tom Lane wrote:
 Robert Haas robertmh...@gmail.com writes:
  Unlike Tom (I think), I do believe that there is demand (possibly only
  from a limited number of people, but demand all the same) for this
  feature.
 
 Please note that I do not think there is *zero* demand for the feature.
 There is obviously some.  What I find highly dubious is whether there is
 enough demand to justify the amount of effort, both short- and long-term,
 that the community would have to put into it.

Well, the bottom line is that this effort should grow the development
and user community of Postgres --- it if doesn't, it is a failure.

  And I also believe that most people in our community are
  generally supportive of the idea, but only a minority are willing to
  put in time to make it happen.  So I have no problem saying to the
  people who want the feature - none of our committers feel like working
  on this.  Sorry.  On the other hand, I also have no problem telling
  them - good news, Bruce Momjian thinks this is a great feature and
  wants to help you get it done.  I *do* have a problem with saying - we
  don't really know whether anyone will ever want to work on this with
  you or not.
 
 If I thought that Bruce could go off in a corner and make this happen
 and it would create no demands on anybody but him and KaiGai-san, I
 would say fine, if that's where you want to spend your time, go for
 it.  But even to state that implied claim is to see how false it is.
 Bruce is pointing to the Windows port, but he didn't make it happen
 by himself, or any close approximation of that.  Everybody who works
 on this project has been affected by that, and we're *still* putting
 significant amounts of time into Windows compatibility, over five years
 later.

The Windows port was primiarly done by Magnus, Claudio Natoli, and
Andrew Dunstan.  The good thing about that group is that their
involvement in Win32 did not take them away from existing Postgres work
--- in fact I think it increased Magnus's and Andrew's involvement.  As
I stated above, I expect the SE-Postgres work to be done mostly by new
people and to expand our development team.  KaiGai is certainly a new
addition, and I think there is already an indication that new people are
getting involved.  

Of course, our existing people will have to help too, but as I stated a
few days ago, I expect security-specific stuff to be maintained mostly
by new people, and our existing folks are going to have to help with
hooks, plus adding things like mandatory access control and row-level
security to base Postgres.  (I do think it is inevitable that those will
be added some day.  I agree the security folks will be accelerating
that.  Hopefully we will get more good out of this than the
inconvenience of this accelerated security stuff.)

 My guess is that a credible SEPostgres offering will require a long-term
 amount of work at least equal to, and very possibly a good deal more
 than, what it took to make a native Windows port.  If SEPostgres could
 bring us even 10% as many new users as the Windows port did, it'd
 probably be a worthwhile use of our resources.  But again, that's an
 assumption that's difficult to type without bursting into laughter.

Odds are SEPostgres will add perhaps 1% new users compared to Win32, but
perhaps very important, energetic, and visible users.  As stated
earlier, the base Postgres security additions like row-level security
are going to be inconvenient, but I do think we will eventually need
them anyway, so I don't see them as SE-Postgres burdens.

I am not replying to many of these emails so I don't appear to be
brow-beating (forcing) the community into accepting this features.  I
might be brow-beating the community, but I don't want to _appear_ to be
brow-beating.  ;-)

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] dtrace probes for memory manager

On Fri, Dec 11, 2009 at 3:50 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Zdenek Kotala zdenek.kot...@sun.com writes:
 Tom Lane píše v pá 11. 12. 2009 v 15:11 -0500:
 If we go this route it would be nice to think about making a facility
 that has some usefulness for non-DTrace platforms too.

 Do you mean general facility for switching memory allocator?

 No, I was thinking of some sort of memory allocation stats collection
 that doesn't depend on DTrace.  It's amazing to me that we've never
 gone back and improved on the original quick-and-dirty
 MemoryContextStats mechanism.  I certainly find myself using that a
 lot for issues like tracking down memory leaks.  While palloc has a lot
 of advantages, the fact that you can't easily plug in a debug-friendly
 substitute malloc package is not one of them :-(

This might be nice to have, but I don't think it's necessarily a
prerequisite for a committable patch.  For that, I think we just need
something that uses the dispatch mechanism to avoid the overhead in
the normal case.

However, as this is a substantial redesign and there are 4 days left
in the CommitFest, I am going to mark this patch Returned with
Feedback for now.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

On Fri, Dec 11, 2009 at 8:41 PM, Bruce Momjian br...@momjian.us wrote:
 I am not replying to many of these emails so I don't appear to be
 brow-beating (forcing) the community into accepting this features.  I
 might be brow-beating the community, but I don't want to _appear_ to be
 brow-beating.  ;-)

LOL.  At least you're honest...

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

Ron Mayer wrote:
 Bruce Momjian wrote:
  Well, the bottom line is that this effort should grow the development
  and user community of Postgres --- it if doesn't, it is a failure.
 
 Really?  Even if it only allows existing Postgres users and companies to
 expand their use into higher security applications IMHO it's a success.
 
 If a main goal were increasing users, implementing MySQL-isms
 and MSFTSqlServer-isms would seem the biggest bang for the buck.

Well, it should expand the user-base because right now we don't have
many current users who are looking for these features --- if we did, it
would be an easy decision to add these features.  Second, it should add
new developers because we have limited developer resources, and I am not
sure adding additional security is at the top of our priority list, and
we don't have many security-expert developers so by definition we need
more to implement this reliably.

These might be unpleasant opinions, but I believe they are accurate.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

2009-12-11 Thread Ron Mayer

Bruce Momjian wrote:
 Well, the bottom line is that this effort should grow the development
 and user community of Postgres --- it if doesn't, it is a failure.

Really?  Even if it only allows existing Postgres users and companies to
expand their use into higher security applications IMHO it's a success.

If a main goal were increasing users, implementing MySQL-isms
and MSFTSqlServer-isms would seem the biggest bang for the buck.




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

2009-12-11 Thread Ashish


I am thinking about starting with the following TODO item:

-- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual 
row counts differ by a specified percentage.

I picked this because it is somewhat related to query processing which is what I am 
most interested in. It also seems like a good start up project for a newbie 
like me. Before I start looking into what this would involve and start a conversation 
on designing a solution - I wanted to know what you guys think about this particular 
TODO, and it suitability to a newbie. Looking forward to your comments...

Thanks
Ashish

On Mon, 7 Dec 2009, Josh Berkus wrote:


On 12/7/09 4:41 PM, Ashish wrote:

Albe  Joshua, thanks for the advice. I am in the process of deciding
what to work on and am looking at the TODO list. I definitely do not
intend to work in a vacuum :-) I am really excited about this and look
forward to being challenged and learning a lot.


When you decide what you want to work on, let us know and we'll try to
find you an appropriate mentor.

--Josh Berkus





--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

Ashish wrote:
 I am thinking about starting with the following TODO item:
 
 -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated
 and actual row counts differ by a specified percentage.
 
 I picked this because it is somewhat related to query processing
 which is what I am most interested in. It also seems like a
 good start up project for a newbie like me. Before I start
 looking into what this would involve and start a conversation
 on designing a solution - I wanted to know what you guys think
 about this particular TODO, and it suitability to a newbie.
 Looking forward to your comments...

I even have a sample patch you can use as a start, attached.

--
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +
Index: doc/src/sgml/ref/explain.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/ref/explain.sgml,v
retrieving revision 1.38
diff -c -c -r1.38 explain.sgml
*** doc/src/sgml/ref/explain.sgml	18 Sep 2006 19:54:01 -	1.38
--- doc/src/sgml/ref/explain.sgml	22 Dec 2006 17:09:05 -
***
*** 64,72 
para
 The literalANALYZE/literal option causes the statement to be actually executed, not only
 planned.  The total elapsed time expended within each plan node (in
!milliseconds) and total number of rows it actually returned are added to
!the display.  This is useful for seeing whether the planner's estimates
!are close to reality.
/para
  
important
--- 64,72 
para
 The literalANALYZE/literal option causes the statement to be actually executed, not only
 planned.  The total elapsed time expended within each plan node (in
!milliseconds) and total number of rows it actually returned and variance are added to
!the display.  A sign of the variance indicates whether the estimate was too high or too low.
!This is useful for seeing how close the planner's estimates are to reality.
/para
  
important
***
*** 222,229 
  
 QUERY PLAN
  -
!  HashAggregate  (cost=39.53..39.53 rows=1 width=8) (actual time=0.661..0.672 rows=7 loops=1)
!-gt;  Index Scan using test_pkey on test  (cost=0.00..32.97 rows=1311 width=8) (actual time=0.050..0.395 rows=99 loops=1)
   Index Cond: ((id gt; $1) AND (id lt; $2))
   Total runtime: 0.851 ms
  (4 rows)
--- 222,229 
  
 QUERY PLAN
  -
!  HashAggregate  (cost=39.53..39.53 rows=1 width=8) (actual time=0.661..0.672 rows=7 var=-6.00 loops=1)
!-gt;  Index Scan using test_pkey on test  (cost=0.00..32.97 rows=1311 width=8) (actual time=0.050..0.395 rows=99 var=+12.24 loops=1)
   Index Cond: ((id gt; $1) AND (id lt; $2))
   Total runtime: 0.851 ms
  (4 rows)
Index: src/backend/commands/explain.c
===
RCS file: /cvsroot/pgsql/src/backend/commands/explain.c,v
retrieving revision 1.152
diff -c -c -r1.152 explain.c
*** src/backend/commands/explain.c	4 Oct 2006 00:29:51 -	1.152
--- src/backend/commands/explain.c	22 Dec 2006 17:09:09 -
***
*** 57,62 
--- 57,63 
  static void show_sort_keys(Plan *sortplan, int nkeys, AttrNumber *keycols,
  			   const char *qlabel,
  			   StringInfo str, int indent, ExplainState *es);
+ static double ExplainVariance(double estimate, double actual);
  
  /*
   * ExplainQuery -
***
*** 704,713 
  	{
  		double		nloops = planstate-instrument-nloops;
  
! 		appendStringInfo(str,  (actual time=%.3f..%.3f rows=%.0f loops=%.0f),
  		 1000.0 * planstate-instrument-startup / nloops,
  		 1000.0 * planstate-instrument-total / nloops,
  		 planstate-instrument-ntuples / nloops,
  		 planstate-instrument-nloops);
  	}
  	else if (es-printAnalyze)
--- 705,716 
  	{
  		double		nloops = planstate-instrument-nloops;
  
! 		appendStringInfo(str,  (actual time=%.3f..%.3f rows=%.0f var=%+.2f loops=%.0f),
  		 1000.0 * planstate-instrument-startup / nloops,
  		 1000.0 * planstate-instrument-total / nloops,
  		 planstate-instrument-ntuples / nloops,
+ 		 ExplainVariance(plan-plan_rows,
+ 	planstate-instrument-ntuples / nloops),
  		 planstate-instrument-nloops);
  	}
  	else if (es-printAnalyze)
***
*** 1205,1207 
--- 1208,1225 
  
  	appendStringInfo(str, \n);
  }
+ 
+ 
+ static double ExplainVariance(double estimate, double

Re: [HACKERS] EXPLAIN BUFFERS

On Fri, Dec 11, 2009 at 11:36 AM, Euler Taveira de Oliveira
eu...@timbira.com wrote:
 Robert Haas escreveu:
 On Thu, Dec 10, 2009 at 9:35 PM, Takahiro Itagaki
 itagaki.takah...@oss.ntt.co.jp wrote:
 Anyway, a revised patch according to the comments is attached.
 The new text format is:
  Buffers: shared hit=675 read=968, temp read=1443 written=1443
    * Zero values are omitted. (Non-text formats could have zero values.)
    * Rename Blocks: to Buffers:.
    * Remove parentheses and add a comma between shared, local and temp.

 I did a bit of copy-editing of your doc changes to make the English a
 bit more correct and idiomatic.  Slightly revised patch attached for
 your consideration.  The output format looks really nice (thanks for
 bearing with me), and the functionality is great.

 Please, document that zero values are omitted in the text format. It seems
 intuitive but could be surprise because zero values are in non-text formats.

OK, done, see attached.  I also noticed when looking through this that
the documentation says that auto_explain.log_buffers is ignored unless
auto_explain.log_analyze is set.  That is true and seems right to me,
but for some reason explain_ExecutorEnd() had been changed to set
es.analyze if either log_analyze or log_buffers was set.  It actually
didn't have any effect unless log_analyze was set, but only because
explain_ExecutorStart doesn't set queryDesc-doInstrument in that
case.  So I've reverted that here for clarity.

...Robert
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
index f0d907d..491f479 100644
--- a/contrib/auto_explain/auto_explain.c
+++ b/contrib/auto_explain/auto_explain.c
@@ -22,6 +22,7 @@ PG_MODULE_MAGIC;
 static int	auto_explain_log_min_duration = -1; /* msec or -1 */
 static bool auto_explain_log_analyze = false;
 static bool auto_explain_log_verbose = false;
+static bool auto_explain_log_buffers = false;
 static int	auto_explain_log_format = EXPLAIN_FORMAT_TEXT;
 static bool auto_explain_log_nested_statements = false;
 
@@ -93,6 +94,16 @@ _PG_init(void)
 			 NULL,
 			 NULL);
 
+	DefineCustomBoolVariable(auto_explain.log_buffers,
+			 Log buffers usage.,
+			 NULL,
+			 auto_explain_log_buffers,
+			 false,
+			 PGC_SUSET,
+			 0,
+			 NULL,
+			 NULL);
+
 	DefineCustomEnumVariable(auto_explain.log_format,
 			 EXPLAIN format to be used for plan logging.,
 			 NULL,
@@ -221,6 +232,7 @@ explain_ExecutorEnd(QueryDesc *queryDesc)
 			ExplainInitState(es);
 			es.analyze = (queryDesc-doInstrument  auto_explain_log_analyze);
 			es.verbose = auto_explain_log_verbose;
+			es.buffers = (es.analyze  auto_explain_log_buffers);
 			es.format = auto_explain_log_format;
 
 			ExplainBeginOutput(es);
diff --git a/doc/src/sgml/auto-explain.sgml b/doc/src/sgml/auto-explain.sgml
index dd3f3fd..1b9d4d9 100644
--- a/doc/src/sgml/auto-explain.sgml
+++ b/doc/src/sgml/auto-explain.sgml
@@ -104,6 +104,25 @@ LOAD 'auto_explain';
 
varlistentry
 term
+ varnameauto_explain.log_buffers/varname (typeboolean/type)
+/term
+indexterm
+ primaryvarnameauto_explain.log_buffers/ configuration parameter/primary
+/indexterm
+listitem
+ para
+  varnameauto_explain.log_buffers/varname causes commandEXPLAIN
+  (ANALYZE, BUFFERS)/ output, rather than just commandEXPLAIN/ 
+  output, to be printed when an execution plan is logged. This parameter is 
+  off by default. Only superusers can change this setting. This
+  parameter has no effect unless varnameauto_explain.log_analyze/
+  parameter is set.
+ /para
+/listitem
+   /varlistentry
+
+   varlistentry
+term
  varnameauto_explain.log_format/varname (typeenum/type)
 /term
 indexterm
diff --git a/doc/src/sgml/ref/explain.sgml b/doc/src/sgml/ref/explain.sgml
index 0d03469..6c68afd 100644
--- a/doc/src/sgml/ref/explain.sgml
+++ b/doc/src/sgml/ref/explain.sgml
@@ -31,7 +31,7 @@ PostgreSQL documentation
 
  refsynopsisdiv
 synopsis
-EXPLAIN [ ( { ANALYZE replaceable class=parameterboolean/replaceable | VERBOSE replaceable class=parameterboolean/replaceable | COSTS replaceable class=parameterboolean/replaceable | FORMAT { TEXT | XML | JSON | YAML } } [, ...] ) ] replaceable class=parameterstatement/replaceable
+EXPLAIN [ ( { ANALYZE replaceable class=parameterboolean/replaceable | VERBOSE replaceable class=parameterboolean/replaceable | COSTS replaceable class=parameterboolean/replaceable | BUFFERS replaceable class=parameterboolean/replaceable | FORMAT { TEXT | XML | JSON | YAML } } [, ...] ) ] replaceable class=parameterstatement/replaceable
 EXPLAIN [ ANALYZE ] [ VERBOSE ] replaceable class=parameterstatement/replaceable
 /synopsis
  /refsynopsisdiv
@@ -140,6 +140,24 @@ ROLLBACK;
/varlistentry
 
varlistentry
+termliteralBUFFERS/literal/term
+listitem
+ para
+  Include information on buffer usage. Specifically, include the number of
+  shared

Re: [HACKERS] [PATCH] dtrace probes for memory manager

Greg Smith g...@2ndquadrant.com writes:
 Tom Lane wrote:
 It's amazing to me that we've never
 gone back and improved on the original quick-and-dirty
 MemoryContextStats mechanism.

 That code hasn't really gone anywhere since Neil tweaked the indentation 
 two years ago.  What sorts of improvements were you looking for?

Not anything like what you were thinking about, apparently ;-).  What
I wish for the most often is a way to track palloc usage down to the
particular file/line of the alloc call.  This is obviously not something
that we'd pay the overhead for in a standard build, but I would love to
be able to build a debug version that could do it when I needed.  There
are malloc substitutes that can do this, but the palloc layer means that
they're not especially useful for debugging the PG backend.

Of course something like that doesn't have to have anything to do with
what Zdenek was wishing for, but it just struck me that creating a
stats-oriented shim layer in the memory context stuff is playing on
pretty much the same turf.

 I started on trying to improve this area at one point but didn't get 
 far.  My first step was going to be just wrapping the call into a UDF to 
 make it easier to reach without having to set loose gdb.  I thought that 
 might expand the possible uses for MemoryContextStats to help find a 
 memory leak safely on the production box, and therefore attact more 
 attention to improving it.

Just for context: the reason MemoryContextStats is designed as it is
is so that it has a reasonable chance of telling you something useful
about an out-of-memory failure.  When you're down to your last ten
bytes, it's not going to help if your reporting mechanism wants to
allocate a data structure to put its results in.  So it writes to stderr
and nothing else.  It may well be that that constraint means we can
never do anything really creative with MemoryContextStats per se.
But I see the use-case for other mechanisms as being to get some info
before we reach the point of having our backs to the wall.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

On Fri, Dec 11, 2009 at 9:00 PM, Ashish abin...@u.washington.edu wrote:
 I am thinking about starting with the following TODO item:

 -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual
 row counts differ by a specified percentage.

 I picked this because it is somewhat related to query processing which is
 what I am most interested in. It also seems like a good start up project
 for a newbie like me. Before I start looking into what this would involve
 and start a conversation on designing a solution - I wanted to know what you
 guys think about this particular TODO, and it suitability to a newbie.
 Looking forward to your comments...

If we're going to do this, I think we should implement this as an
optional behavior controlled by a new EXPLAIN option (maybe VARIANCE,
following Bruce's patch?) and generate the output using
ExplainPropertysome-data-type.  We could possibly make the option
take an optional threshold indicating how much variance is required
before the variance gets displayed, and display the variance for every
node if VARIANCE is specified without an argument.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

On Fri, Dec 11, 2009 at 9:05 PM, Bruce Momjian br...@momjian.us wrote:
 Ashish wrote:
 I am thinking about starting with the following TODO item:

 -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated
 and actual row counts differ by a specified percentage.

 I picked this because it is somewhat related to query processing
 which is what I am most interested in. It also seems like a
 good start up project for a newbie like me. Before I start
 looking into what this would involve and start a conversation
 on designing a solution - I wanted to know what you guys think
 about this particular TODO, and it suitability to a newbie.
 Looking forward to your comments...

 I even have a sample patch you can use as a start, attached.

Interesting.  The logic in ExplainVariance() doesn't look right to me
- the cases where one argument is zero seem like they will produce a
differently-scaled result than otherwise.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] status update on Hot Standby and Streaming Replication

2009-12-11 Thread Simon Riggs

On Fri, 2009-12-11 at 20:38 -0500, Robert Haas wrote:

 I'm not quite sure where we stand with these two patches.  My
 impression is that many of the outstanding TODO items in Hot Standby
 have been fixed, and I'm not sure what remains.  Streaming Replication
 I think reviewing is not as far advanced, but I'm not sure.  Any
 chance that Hot Standby can get committed in the next few days before
 we bundle alpha3?  Otherwise, I think it will not get as much test
 coverage as it really deserves.

It's possible that HS will get committed in next few days.

I'm aiming to release a final patch within 48 hours and commit by Wed.

The main TODO items are fixed, but I'm now going through testing and
polishing.

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

Robert Haas wrote:
 On Fri, Dec 11, 2009 at 9:05 PM, Bruce Momjian br...@momjian.us wrote:
  Ashish wrote:
  I am thinking about starting with the following TODO item:
 
  -- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated
  and actual row counts differ by a specified percentage.
 
  I picked this because it is somewhat related to query processing
  which is what I am most interested in. It also seems like a
  good start up project for a newbie like me. Before I start
  looking into what this would involve and start a conversation
  on designing a solution - I wanted to know what you guys think
  about this particular TODO, and it suitability to a newbie.
  Looking forward to your comments...
 
  I even have a sample patch you can use as a start, attached.
 
 Interesting.  The logic in ExplainVariance() doesn't look right to me
 - the cases where one argument is zero seem like they will produce a
 differently-scaled result than otherwise.

Yea, it is just a starting point for him.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.