[HACKERS] Streaming replication, some small issues
A couple of small issues spotted while reviewing the streaming replication patch: - Because sentPtr is initialized to zeros, GetOldestWALSendPointer will return zero before a just-launched WAL sender has sent its first message. That can lead to WAL files that are still needed by another standby to be deleted prematurely. - If a WAL file is not found in the master for some reason, standby goes into an infinite loop retrying it: ERROR: could not read xlog records: FATAL: could not open file pg_xlog/0001 (log file 0, segment 0): No such file or directory ERROR: could not read xlog records: FATAL: could not open file pg_xlog/0001 (log file 0, segment 0): No such file or directory ERROR: could not read xlog records: FATAL: could not open file pg_xlog/0001 (log file 0, segment 0): No such file or directory ... - It's possible to shut down master, change max_wal_senders to 0, restart and do an operation like CLUSTER which then skips WAL-logging. Then shutdown, change max_wal_senders back to non-zero. All this while the standby is running. Leads to a corrupt standby. I've also pushed a couple of small cosmetic changes to replication branch at git://git.postgresql.org/git/users/heikki/postgres.git I'll continue reviewing... -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] executor and slru dtrace probes
Dne 8.12.09 00:27, Bernd Helmle napsal(a): --On 13. November 2009 23:29:41 +0100 Zdenek Kotala zdenek.kot...@sun.com wrote: t contains two DTrace probe groups. One is related to monitoring SLRU and second is about executor nodes. I merged it with the head. Original end of mail thread is here: http://archives.postgresql.org/pgsql-hackers/2009-04/msg00148.php I've started to review this. It seems to me the attached patch wasn't adjusted or discussed again to address Tom's complaints? At least the executor probes contained here hold still the same issues mentioned by Tom in the discussion linked here. I did not make any change. I only revival patch and merge it with head. I think that SLRU probes are OK and acceptable. Tom's issues with executor probes are still there and I expect discussion about them. IIRC Theo uses these probes in production. If you think that it is better I could split patch into two separate patches and both can be reviewed separately. thanks Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EXPLAIN BUFFERS
On Dec 8, 2009, at 12:05 AM, Greg Smith g...@2ndquadrant.com wrote: Robert Haas wrote: I could live with the equals signs, but the use of parentheses seems weird and inconsistent with normal english usage (which permits parentheses as a means of making parenthetical comments). But it is consistent with people seeing: Seq Scan on foo (cost=0.00..155.00 rows=1 width=4) Which seems to be what was being emulated here. I though that was pretty reasonable given this is a related feature. It's not the same at all. The essence of a parenthetical phrase is that it can be omitted without turning what's left into nonsense - and in fact we have COSTS OFF, which does just that. Omitting only the parenthesized portions of the proposed output would not be sensible. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication, some small issues
On Tue, Dec 8, 2009 at 5:30 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: A couple of small issues spotted while reviewing the streaming replication patch: Thanks for the review! - Because sentPtr is initialized to zeros, GetOldestWALSendPointer will return zero before a just-launched WAL sender has sent its first message. That can lead to WAL files that are still needed by another standby to be deleted prematurely. Oops! I fixed that (in my git repository, see the bottom of this mail). - If a WAL file is not found in the master for some reason, standby goes into an infinite loop retrying it: ERROR: could not read xlog records: FATAL: could not open file pg_xlog/0001 (log file 0, segment 0): No such file or directory http://archives.postgresql.org/pgsql-hackers/2009-09/msg01393.php walreceiver shouldn't die on connection error, just to be restarted by startup process. Can we add error handling a la bgwriter and have a retry loop within walreceiver? As the result of your current and previous comment, you mean that walreceiver should always retry connecting to the primary after a connection error occurs in PQgetXLogData/PQputXLogRecPtr, and exit after the other errors occur? Though I'm not sure whether we can determine the error type precisely. - It's possible to shut down master, change max_wal_senders to 0, restart and do an operation like CLUSTER which then skips WAL-logging. Then shutdown, change max_wal_senders back to non-zero. All this while the standby is running. Leads to a corrupt standby. I've regarded this case as a restriction. But, how do you think we should cope with it? 1. Restriction: only documentation is required? 2. Needs safe guard: - forbid the primary to perform such operations while the standby is running? - emit PANIC error on the standby if the primary which lost sync restarts? 3. Full solution: automatic resync mechanism is required? I've also pushed a couple of small cosmetic changes to replication branch at git://git.postgresql.org/git/users/heikki/postgres.git Your changes seem good. I pulled and merged your changes into my repository: git://git.postgresql.org/git/users/fujii/postgres.git branch: replication And, I pushed the capability of replication of a backup history file into the repository. I'll continue reviewing... Thanks a lot! Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication, some small issues
On Tue, Dec 8, 2009 at 8:30 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: - It's possible to shut down master, change max_wal_senders to 0, restart and do an operation like CLUSTER which then skips WAL-logging. Then shutdown, change max_wal_senders back to non-zero. All this while the standby is running. Leads to a corrupt standby. The same thing is possible with archived logs as well, no? I suspect we should have a WAL record to say unlogged operation performed here which a standby database would recognize and throw a large warning up. The only reason I say warning is because it might be reasonable if the relation is some temporary ETL table which isn't needed in the standby. Perhaps if we note the relation affected we could throw an error iff the standby is activated with the relation still existing. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication, some small issues
Greg Stark wrote: On Tue, Dec 8, 2009 at 8:30 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: - It's possible to shut down master, change max_wal_senders to 0, restart and do an operation like CLUSTER which then skips WAL-logging. Then shutdown, change max_wal_senders back to non-zero. All this while the standby is running. Leads to a corrupt standby. The same thing is possible with archived logs as well, no? Yeah, I think you're right. I suspect we should have a WAL record to say unlogged operation performed here which a standby database would recognize and throw a large warning up. +1. Seems like a very simple solution. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] questions about concurrency control in Postgresql
2009/12/8 黄晓骋 huangxcl...@gmail.com: From the above, I think the tuple lock is unnecessary, because it uses transaction lock. Besides, tuple lock is unlocked after the tuple is updated but not after the transaction commits. I mean it's not 2PL. It's a two step process. An update marks the tuple locked. Another transaction which comes along and wants to lock the tuple waits on the transaction marked on the tuple. When the first transaction commits or aborts then the second transaction can proceed and lock the tuple itself. The reason we need both locks is because the first transaction cannot go around the whole database finding every tuple it ever locked to unlock it, firstly that could be a very large list and secondly there would be no way to do that atomically. Tuple locks and all user-visible locks are indeed held until the end of the transaction. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML
On Mon, Dec 07, 2009 at 07:07:13PM -0500, Andrew Dunstan wrote: Josh Berkus wrote: Not everything is sanely convertible into some sort of plugin. A plugin mechanism for this would be FAR more trouble that it is worth, IMNSHO. We are massively over-egging this pudding (as a culinary blogger you should appreciate this analogy). OK, then let's just accept it. It's small, has a maintainer, is useful to some people, and doesn't create any wierd complications. I think, given the knowledge that YAML is now a subdialect of JSON it could potentially be made smaller, but I can't say how at the moment. Actually, it's the other way, JSON is a subset of YAML. I've no contribution to the main topic, but I'd like to point out that the JSON is a subset of YAML meme is not without controversy: http://search.cpan.org/~mlehmann/JSON-XS-2.26/XS.pm#JSON_and_YAML It may not be relevant in your use-case, but I thought it worth a mention. Tim. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML
Tim Bunce wrote: I've no contribution to the main topic, but I'd like to point out that the JSON is a subset of YAML meme is not without controversy: http://search.cpan.org/~mlehmann/JSON-XS-2.26/XS.pm#JSON_and_YAML It may not be relevant in your use-case, but I thought it worth a mention. Ouch. Thanks for bringing it to our attention. Well, if we're going to commit this, as now appears likely, we should have some language lawyers go over our code for both YAML and JSON with a fine tooth comb to make sure what we are producing is strictly According To Hoyle. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML
On Tue, Dec 8, 2009 at 9:13 AM, Andrew Dunstan and...@dunslane.net wrote: Well, if we're going to commit this, as now appears likely, we should have some language lawyers go over our code for both YAML and JSON with a fine tooth comb to make sure what we are producing is strictly According To Hoyle. +1. I'm a little concerned about the bit about the YAML specification changing, too, but at least if we can ensure that we're compliant with the spec that is current at the time the code goes in we have a leg to stand on. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Streaming replication and non-blocking I/O
I find the backend libpq changes related to non-blocking I/O quite complex. Can we find a simpler solution? The problem we're trying to solve is that while the walsender backend sends a lot of WAL records to the client, the client can send a lot of messages to the backend. If volume of the messages from client to server exceeds both the input buffer in the server and the output buffer in the client, the client will block until the server has read some data. But if the client is blocked, it will not process incoming data from the server, and eventually the server will block too. And we have a deadlock. This: http://florin.bjdean.id.au/docs/omnimark/omni55/docs/html/concept/717.htm is a pretty good description of the problem. The first question is: do we really need to be prepared for that? The XLogRecPtr acknowledgment messages the client sends are very small, and if the client is mindful about not sending them too often - perhaps max 1 ack per 1 received XLOG message - the receive buffer in the backend should never fill up in practice. If that's deemed not good enough, we could modify just internal_flush() so that it uses secure_poll to wait for the possibility to either read or write, instead of blocking for just write. Whenever there's incoming data, read them into PqRecvBuffer for later processing, which keeps the OS input buffer from filling up. If PqRecvBuffer fills up, it can be extended, or we can start dropping old XLogRecPtr messages from it. In any case, we'll need something like pq_wait to check if a message can be read without blocking, but that's just a small additional function as opposed to a whole new API for assembling and sending messages without blocking. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A sniffer for the buffer
Thanks for the answers.
I will change it for PostgreSQL 8.4 and try to use DBT-2.
But, I'm not quite sure if DTrace will give me the workload that I want.
Since, i want to trace the Workload that is above the Buffer Layer. With
workload I mean two fields (operation: read/write and Block Number). Then
all blocks that PostgreSQL request to the buffer layer I will capture.
Do DTrace can give my this information?
Thanks in advance,
Jonas Jeske
2009/12/8 Greg Smith g...@2ndquadrant.com
Jonas J wrote:
I took a look in the code again and made some changes. For the readBuffer
im doing now:
ReadBuffer(Relation reln, BlockNumber blockNum)
fprintf(fp,r%u\n,(unsigned int) blockNum); //as defined in header,
typedef uint32 BlockNumber;
and from the write pages:
write_buffer(Buffer buffer, bool unpin){
fprintf(fp,w%d\n,BufferGetBlockNumber(buffer)); //get the blockNumber of
this buffer
It's better to keep this discussion going on the list so you can get
alternate suggestions besides mine. The above is better. You're still
missing the relation number, which you're really going to want eventually.
And I think you're still vulnerable to printing the information out before
the block is locked properly in the write_buffer case.
So, I had never used DTrace, where can I find some good paper to start
studying it ?? Also, do it work with linux, or only solaris ??
You can get it to work on Linux for PostgreSQL use if you use systemtap;
there's some people who've posted suggestions about that around.
P.S.: I'm using PostgreSQL 8.1.4 to run with TPCC-UVA benchmarks tests...
Ah. PostgreSQL 8.1 is significantly slower than the current versions, and
you're not going to get as much help with issues related to the source code
as if you're using a newer one. For example, I don't know the 8.1 buffer
implementation code nearly as well because I didn't really start tinkering
with it until 8.2, so some of the things you're asking about I don't have
easy access or recollection of. Also, the DTrace stuff is only really going
to be helpful if you're starting with 8.4.
There may be some work to get TPCC-UVA working with 8.4 though,
particularly due to some changes made in 8.3 related to casting data to
other types. There is a TPC-C implementation that's used pretty often by
developers here named dbt-2: http://wiki.postgresql.org/wiki/DBT-2 that
will work with a newer version.
--
Greg Smith2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com www.2ndQuadrant.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New PostgreSQL Committers
On Mon, Dec 07, 2009 at 10:49:13AM +, Dave Page wrote: On behalf of the core team, I'm pleased to announce that the Congratulations! +1 Congrats to you all, and thanks for the contributions, both past and future. As an aside, this sort of thing is one of the best signs to an external user of the health of the PostgreSQL project: the 'orderly transfer of power' as it were. I'm always cautious about adopting a project with a limited set of core developers (often one) no matter how good the software. Ross -- Ross Reedstrom, Ph.D. reeds...@rice.edu Systems Engineer Admin, Research Scientistphone: 713-348-6166 The Connexions Project http://cnx.orgfax: 713-348-3665 Rice University MS-375, Houston, TX 77005 GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A sniffer for the buffer
Greg Smith wrote: Jonas J wrote: P.S.: I'm using PostgreSQL 8.1.4 to run with TPCC-UVA benchmarks tests... Ah. PostgreSQL 8.1 is significantly slower than the current versions, and you're not going to get as much help with issues related to the source code as if you're using a newer one. For example, I don't know the 8.1 buffer implementation code nearly as well because I didn't really start tinkering with it until 8.2, so some of the things you're asking about I don't have easy access or recollection of. 8.1 didn't have lock partitioning, so BufMappingLock was a serious problem back then. Completely different from the current code. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Mon, 2009-12-07 at 22:25 -0500, Greg Smith wrote: David P. Quigley wrote: Not to start a flame war here about access control models but you gave 3 different examples one of which I don't think has any means to do anything productive here. You won't be starting a flame war for the same reason some of the community members are so concerned about this patch. There aren't enough people familiar with this part of the security field within our database developer community to even be able to answer fairly basic questions like the one you just clarified. If you can help bring more qualified reviewers to bear on that, it would be extremely helpful. I even tried to organize a meetup between PostgreSQL hackers working in this area and the security people I knew around here (Baltimore/DC) last year, but just couldn't find any interested enough to show. Other than a brief visit on this list from some of the Tresys guys, we haven't seen much input here beyond that offered by the patch author, who's obviously qualified but at the end of the day is still only one opinion. He's also not in a good position to tell other people their ideas are misinformed either. I can't make any guarantees on who I can drag to a meeting but if you wanted to try to organize another meeting between the Postgres people and some of us I can try to organize it on our end. One of my coworkers that does a lot of commenting on stuff like this is on leave at the moment but when he gets back I'll discuss it with him. I'll also talk with some of the other people in the area on our end to see what I can arrange. If you have any questions in the meantime feel free to ask. If there are any specific parts of the patch that you'd like discussed I can do that as well. I do have to agree though that I'd rather see KaiGai's original security plugin framework go in and then merge a particular security module after that.From what I see it would require at least the hook framework and the label storage mechanism. I feel bad saying that knowing the KaiGai spent a lot of time ripping all of that out. However if you are concerned about supporting more than just SELinux as a MAC model then the plugin framework he originally proposed is the better solution. I'd be willing to take a look at the framework and see if it really is SELinux centric. If it is we can figure out if there is a way to accomodate something like SMACK and FMAC. I'd like to hear from someone with more extensive experience with Solaris Trusted Extensions about how TX would make use of this. I have a feeling it would be similar to the way it deals with NFS which is by having the process exist in the global zone as a privileged process and then multi-plexes it to the remaining zones. That way their getpeercon would get a label derived from the zone. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, Dec 8, 2009 at 10:07 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: I'd be willing to take a look at the framework and see if it really is SELinux centric. If it is we can figure out if there is a way to accomodate something like SMACK and FMAC. I'd like to hear from someone with more extensive experience with Solaris Trusted Extensions about how TX would make use of this. I have a feeling it would be similar to the way it deals with NFS which is by having the process exist in the global zone as a privileged process and then multi-plexes it to the remaining zones. That way their getpeercon would get a label derived from the zone. Well, the old patches should still be available in the mailing list archives. Maybe going back and looking at that code would be a good place to start. The non-ripped-out code has been cleaned up a lot since then, but at least it's a place to start. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
I wrote: Frankly, I'd be amazed if there was a performance regression, OK, I'm amazed. While it apparently helps some cases dramatically (Andres had a case where run time was reduced by 93.2%), I found a pretty routine case where run time was increased by 3.1%. I tweaked the code and got that down to a 2.5% run time increase. I'm having troubles getting it any lower than that. And yes, this is real, not noise -- the slowest unpatched time for this test is faster than the fastest time with any version of the patch. :-( Andres, could you provide more information on the test which showed the dramatic improvement? In particular, info on OS, CPU, character set, encoding scheme, and what kind of data was used for the test. I'll do some more testing and try to figure out how the patch is slowing things down and post with details. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
On Tuesday 08 December 2009 16:23:11 Kevin Grittner wrote: I wrote: Frankly, I'd be amazed if there was a performance regression, OK, I'm amazed. While it apparently helps some cases dramatically (Andres had a case where run time was reduced by 93.2%), I found a pretty routine case where run time was increased by 3.1%. I tweaked the code and got that down to a 2.5% run time increase. I'm having troubles getting it any lower than that. And yes, this is real, not noise -- the slowest unpatched time for this test is faster than the fastest time with any version of the patch. :-( Andres, could you provide more information on the test which showed the dramatic improvement? In particular, info on OS, CPU, character set, encoding scheme, and what kind of data was used for the test. I'll do some more testing and try to figure out how the patch is slowing things down and post with details. Could you show your testcase? I dont see why it could get slower? I tested with various data, the one benefiting most was some changelog where each entry was signed by an email. OS: Debian Sid, Core2 Duo, UTF-8, and I tried both C and de_DE.UTF8. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML
Robert Haas robertmh...@gmail.com writes: +1. I'm a little concerned about the bit about the YAML specification changing, too, but at least if we can ensure that we're compliant with the spec that is current at the time the code goes in we have a leg to stand on. If the spec is in flux, that seems like More Than Sufficient reason to reject the patch for the time being. It can be resubmitted when it's no longer shooting at a moving target. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sought after architectures for the PostgreSQL buildfarm?
Peter St. Onge wrote privately: With the local server room renovations, there may be some older equipment being retired. Are there any architectures that you would like to see added to the buildfarm? Hackers, Any nominations? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 If the spec is in flux, that seems like More Than Sufficient reason to reject the patch for the time being. It can be resubmitted when it's no longer shooting at a moving target. Saying that it is in flux is a bit of a stretch. Even if it were, the parts that do change are nothing that will affect us. We're doing dirt-simple YAML (and JSON) generation. Basically, 'name: value' pairs plus some list building via indents and dashes. I'm completely not worried about our usage ever falling afoul of future YAML or JSON spec changes. (This goes for the person on this list concerned about the output being too hard to parse. Yes, YAML has lots of tiny corner cases and elaborate syntax, but we're not using any of those, so parsing should be quite possible for any YAML parser out there). - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation PGP Key: 0x14964AC8 200912081104 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkseeZQACgkQvJuQZxSWSsjNlACg3j5zNPnGzNiXtRG0r9OZnlY3 qjkAoOvzcq+S9qLGQIMbZ0BH55P+TtH/ =icE3 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sought after architectures for the PostgreSQL buildfarm?
Andrew Dunstan and...@dunslane.net writes: Peter St. Onge wrote privately: With the local server room renovations, there may be some older equipment being retired. Are there any architectures that you would like to see added to the buildfarm? Any nominations? Sure, what's being offered? I've noticed a depressing trend towards Intel-conformity in the buildfarm lately. Some more non-Linux OSes wouldn't hurt either. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Andres Freund and...@anarazel.de wrote: Could you show your testcase? OK. I was going to try to check other platforms first, and package up the information better, but here goes. I created 1 lines with random IP-based URLs for a test. The first few lines are: create table t1 (c1 int not null primary key, c2 text); insert into t1 values (2, 'http://255.102.51.212/*/quick/brown/fox?jumpsover*lazydog.html http://204.56.222.143/*/quick/brown/fox?jumpsover*lazydog.html http://138.183.168.227/*/quick/brown/fox?jumpsover*lazydog.html Actually, the special character was initially the word the, but I wanted to see if having non-ASCII characters in the value made any difference. It didn't. Unfortunately, I was testing at home last night and forgot to bring the exact test query with me, but it was this or something close to it: \timing select to_tsvector(c2) from t1, (select generate_series(1,200)) x where c1 = 2; I was running on Ubuntu 9.10, an AMD dual core CPU (don't have the model number handy), UTF-8, en_US.UTF8. I dont see why it could get slower? I don't either. The best I can tell, following the pointer from orig to any of its elements seems to be way more expensive than I would ever have guessed. The only thing that seemed to improve the speed was minimizing that by using a local variable to capture any element referenced more than once. (Although, there is overlap between the timings for the original patch and the one which seemed a slight improvement; I would need to do more testing to really rule out noise and have complete confidence that my changes actually are an improvement on the original patch.) Perhaps it is some quirk of using 32 bit pointers on the 64 bit AMD CPU? (I'm looking forward to testing this today on a 64 bit build on an Intel CPU.) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
On Tuesday 08 December 2009 17:15:36 Kevin Grittner wrote: Andres Freund and...@anarazel.de wrote: Could you show your testcase? Will hopefully look into this later. I dont see why it could get slower? I don't either. The best I can tell, following the pointer from orig to any of its elements seems to be way more expensive than I would ever have guessed. The only thing that seemed to improve the speed was minimizing that by using a local variable to capture any element referenced more than once. (Although, there is overlap between the timings for the original patch and the one which seemed a slight improvement; I would need to do more testing to really rule out noise and have complete confidence that my changes actually are an improvement on the original patch.) Perhaps it is some quirk of using 32 bit pointers on the 64 bit AMD CPU? (I'm looking forward to testing this today on a 64 bit build on an Intel CPU.) Did you test that with a optimized build? Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Andres Freund and...@anarazel.de wrote: Did you test that with a optimized build? After running a series of tests with --enable-cassert to confirm that nothing squawked, I disabled that before doing any performance testing. Going from memory, I used --enable-debug --with-libxml and --prefix. I didn't explicitly use or disable any compiler optimizations. Possibly relevant is that I was using Greg Smith's peg tool (as best I could): http://github.com/gregs1104/peg/ I'm not aware of it doing anything to the compile options, but I didn't look for that, either. I guess I should poke around that some more to be sure. After having peg do the checkout and set up the directories, did my own ./configure and make runs so that I could be sure of my configure settings. Are there any settings which you feel I should be using which PostgreSQL doesn't set up as you would recommend during the ./configure run? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, Dec 8, 2009 at 10:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Mon, 2009-12-07 at 17:57 -0500, Robert Haas wrote: On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian br...@momjian.us wrote: As Alvaro mentioned, the original patch used ACE but it added too much code so the community requested its removal from the patch. It could be re-added if we have a need. Well, there's no point in putting that framework back in unless we can make it sufficiently general that it could be used to serve the needs of more than one security model. And so far, the signs have not been promising. David Quigley suggests downthread that making a truly general model isn't really possible, and he may be right, or not. I was just mentioning that it's an angle I have been thinking about investigating, but it may be a dead end. The real issue is making the code committable, and then maintaining it, as Tom rightly says, forever. We've got to make sure that we're willing to take that on before we do it, and I don't think it's a small task. It isn't so much whether we want the feature as whether the level of effort is proportionate to the benefit. ...Robert So the issue with generality is that path name based MAC has a different set of requirements than label based does. If you want to acomodate several types of label based MAC models then a framework can be developed for that similar to the one in the Linux Kernel. I asked around after I sent the email yesterday and from what I understand AppArmor does not have the concept of a userspace object manager so I don't know what they'd do in this area. I'm sure they could come up with a scheme to write policy for the database but I don't know how useful it would be. If you look at the LSM framework in the Linux Kernel recently there have been hooks added to accomodate path based MAC systems so it can be done but adds another set of hooks. The important goal here though in designing a framework is to make sure that you have a comprehensive list of the objects you want to mediate and make sure you put the proper enforcement points in. Someone may come along later and want to mediate a new object or some new functionality may come along that introduces a new object so a hook may then need to be added. Something to realize as well is that a security model may not want to implement all of the hooks and it doesn't have to. In the case of no module being loaded or someone not wanting the loadable security module framework I'm sure we can provide an option to reduce overhead or compile the framework out completely. I'll take a look at his patches for the framework that KaiGai originally proposed. It sounds like the pathname-based schemes are not really designed to work inside of a database anyway, so my first thought is we shouldn't worry about them. The label-based schemes are by their nature designed to apply in an arbitrary context being applied to arbitrary objects. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, 2009-12-08 at 11:48 -0500, Robert Haas wrote: On Tue, Dec 8, 2009 at 10:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Mon, 2009-12-07 at 17:57 -0500, Robert Haas wrote: On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian br...@momjian.us wrote: As Alvaro mentioned, the original patch used ACE but it added too much code so the community requested its removal from the patch. It could be re-added if we have a need. Well, there's no point in putting that framework back in unless we can make it sufficiently general that it could be used to serve the needs of more than one security model. And so far, the signs have not been promising. David Quigley suggests downthread that making a truly general model isn't really possible, and he may be right, or not. I was just mentioning that it's an angle I have been thinking about investigating, but it may be a dead end. The real issue is making the code committable, and then maintaining it, as Tom rightly says, forever. We've got to make sure that we're willing to take that on before we do it, and I don't think it's a small task. It isn't so much whether we want the feature as whether the level of effort is proportionate to the benefit. ...Robert So the issue with generality is that path name based MAC has a different set of requirements than label based does. If you want to acomodate several types of label based MAC models then a framework can be developed for that similar to the one in the Linux Kernel. I asked around after I sent the email yesterday and from what I understand AppArmor does not have the concept of a userspace object manager so I don't know what they'd do in this area. I'm sure they could come up with a scheme to write policy for the database but I don't know how useful it would be. If you look at the LSM framework in the Linux Kernel recently there have been hooks added to accomodate path based MAC systems so it can be done but adds another set of hooks. The important goal here though in designing a framework is to make sure that you have a comprehensive list of the objects you want to mediate and make sure you put the proper enforcement points in. Someone may come along later and want to mediate a new object or some new functionality may come along that introduces a new object so a hook may then need to be added. Something to realize as well is that a security model may not want to implement all of the hooks and it doesn't have to. In the case of no module being loaded or someone not wanting the loadable security module framework I'm sure we can provide an option to reduce overhead or compile the framework out completely. I'll take a look at his patches for the framework that KaiGai originally proposed. It sounds like the pathname-based schemes are not really designed to work inside of a database anyway, so my first thought is we shouldn't worry about them. The label-based schemes are by their nature designed to apply in an arbitrary context being applied to arbitrary objects. ...Robert So I was reading through a set of slides that KaiGai has and he mentioned a May commitfest link and I looked for the comments related to his PGACE patches. I've been crawling through the commitfest paces so I can figure out what the latest version of the pgace patch is. Does anyone know when the patch was reduced to what it is today? Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On 12/8/09 11:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Tue, 2009-12-08 at 11:48 -0500, Robert Haas wrote: On Tue, Dec 8, 2009 at 10:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Mon, 2009-12-07 at 17:57 -0500, Robert Haas wrote: On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian br...@momjian.us wrote: As Alvaro mentioned, the original patch used ACE but it added too much code so the community requested its removal from the patch. It could be re-added if we have a need. Well, there's no point in putting that framework back in unless we can make it sufficiently general that it could be used to serve the needs of more than one security model. And so far, the signs have not been promising. David Quigley suggests downthread that making a truly general model isn't really possible, and he may be right, or not. I was just mentioning that it's an angle I have been thinking about investigating, but it may be a dead end. The real issue is making the code committable, and then maintaining it, as Tom rightly says, forever. We've got to make sure that we're willing to take that on before we do it, and I don't think it's a small task. It isn't so much whether we want the feature as whether the level of effort is proportionate to the benefit. ...Robert So the issue with generality is that path name based MAC has a different set of requirements than label based does. If you want to acomodate several types of label based MAC models then a framework can be developed for that similar to the one in the Linux Kernel. I asked around after I sent the email yesterday and from what I understand AppArmor does not have the concept of a userspace object manager so I don't know what they'd do in this area. I'm sure they could come up with a scheme to write policy for the database but I don't know how useful it would be. If you look at the LSM framework in the Linux Kernel recently there have been hooks added to accomodate path based MAC systems so it can be done but adds another set of hooks. The important goal here though in designing a framework is to make sure that you have a comprehensive list of the objects you want to mediate and make sure you put the proper enforcement points in. Someone may come along later and want to mediate a new object or some new functionality may come along that introduces a new object so a hook may then need to be added. Something to realize as well is that a security model may not want to implement all of the hooks and it doesn't have to. In the case of no module being loaded or someone not wanting the loadable security module framework I'm sure we can provide an option to reduce overhead or compile the framework out completely. I'll take a look at his patches for the framework that KaiGai originally proposed. It sounds like the pathname-based schemes are not really designed to work inside of a database anyway, so my first thought is we shouldn't worry about them. The label-based schemes are by their nature designed to apply in an arbitrary context being applied to arbitrary objects. ...Robert So I was reading through a set of slides that KaiGai has and he mentioned a May commitfest link and I looked for the comments related to his PGACE patches. I've been crawling through the commitfest paces so I can figure out what the latest version of the pgace patch is. Does anyone know when the patch was reduced to what it is today? Dave I'm another SELinux developer and I'd like to help out where I can here. I'm a bit confused by this discussion of PGACE. I thought the postgresql community agreed that they wanted this removed in order to make the patch size smaller. Has that changed? Is the increase in patch size now acceptable? Sorry if I'm joining the conversation late. Thanks, Chad Sellers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, Dec 8, 2009 at 12:16 PM, Chad Sellers csell...@tresys.com wrote: On 12/8/09 11:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Tue, 2009-12-08 at 11:48 -0500, Robert Haas wrote: On Tue, Dec 8, 2009 at 10:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Mon, 2009-12-07 at 17:57 -0500, Robert Haas wrote: On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian br...@momjian.us wrote: As Alvaro mentioned, the original patch used ACE but it added too much code so the community requested its removal from the patch. It could be re-added if we have a need. Well, there's no point in putting that framework back in unless we can make it sufficiently general that it could be used to serve the needs of more than one security model. And so far, the signs have not been promising. David Quigley suggests downthread that making a truly general model isn't really possible, and he may be right, or not. I was just mentioning that it's an angle I have been thinking about investigating, but it may be a dead end. The real issue is making the code committable, and then maintaining it, as Tom rightly says, forever. We've got to make sure that we're willing to take that on before we do it, and I don't think it's a small task. It isn't so much whether we want the feature as whether the level of effort is proportionate to the benefit. ...Robert So the issue with generality is that path name based MAC has a different set of requirements than label based does. If you want to acomodate several types of label based MAC models then a framework can be developed for that similar to the one in the Linux Kernel. I asked around after I sent the email yesterday and from what I understand AppArmor does not have the concept of a userspace object manager so I don't know what they'd do in this area. I'm sure they could come up with a scheme to write policy for the database but I don't know how useful it would be. If you look at the LSM framework in the Linux Kernel recently there have been hooks added to accomodate path based MAC systems so it can be done but adds another set of hooks. The important goal here though in designing a framework is to make sure that you have a comprehensive list of the objects you want to mediate and make sure you put the proper enforcement points in. Someone may come along later and want to mediate a new object or some new functionality may come along that introduces a new object so a hook may then need to be added. Something to realize as well is that a security model may not want to implement all of the hooks and it doesn't have to. In the case of no module being loaded or someone not wanting the loadable security module framework I'm sure we can provide an option to reduce overhead or compile the framework out completely. I'll take a look at his patches for the framework that KaiGai originally proposed. It sounds like the pathname-based schemes are not really designed to work inside of a database anyway, so my first thought is we shouldn't worry about them. The label-based schemes are by their nature designed to apply in an arbitrary context being applied to arbitrary objects. ...Robert So I was reading through a set of slides that KaiGai has and he mentioned a May commitfest link and I looked for the comments related to his PGACE patches. I've been crawling through the commitfest paces so I can figure out what the latest version of the pgace patch is. Does anyone know when the patch was reduced to what it is today? Dave I'm another SELinux developer and I'd like to help out where I can here. I'm a bit confused by this discussion of PGACE. I thought the postgresql community agreed that they wanted this removed in order to make the patch size smaller. Has that changed? Is the increase in patch size now acceptable? Sorry if I'm joining the conversation late. Thanks, Chad Sellers Nothing's been decided. We're just trying to figure out the best way to tackle the problem. I think the main question here is who if anyone from among the committers is willing to put in the time and effort to maintain this feature over the short and long haul, but that's sort of an internal project issue. I was just thinking out loud about whether it was possible and/or desirable to try to modularize this a bit so that it could be used for more than just SE-Linux. Obviously, the labeling stuff could be generalize to accomodate a label from an arbitrary security system, but that's only a small piece of the problem. One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design
Re: [HACKERS] Sought after architectures for the PostgreSQL buildfarm?
On 12/8/09 7:51 AM, Andrew Dunstan wrote: Peter St. Onge wrote privately: With the local server room renovations, there may be some older equipment being retired. Are there any architectures that you would like to see added to the buildfarm? Got any Sparc machines? We're likely about to lose our sparc buildfarm. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sought after architectures for the PostgreSQL buildfarm?
I might be able to help with: Sparc PA-Risc (HP-UX) IA64 -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683E-Mail: l...@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Josh Berkus Sent: Tuesday, December 08, 2009 11:43 AM To: Andrew Dunstan Cc: PostgreSQL-development; Peter St. Onge Subject: Re: [HACKERS] Sought after architectures for the PostgreSQL buildfarm? On 12/8/09 7:51 AM, Andrew Dunstan wrote: Peter St. Onge wrote privately: With the local server room renovations, there may be some older equipment being retired. Are there any architectures that you would like to see added to the buildfarm? Got any Sparc machines? We're likely about to lose our sparc buildfarm. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sought after architectures for the PostgreSQL buildfarm?
On Tue, Dec 08, 2009 at 10:51:01AM -0500, Andrew Dunstan wrote: Peter St. Onge wrote privately: With the local server room renovations, there may be some older equipment being retired. Are there any architectures that you would like to see added to the buildfarm? Hackers, Any nominations? Do they have anything running z/OS? It has a truly amazing compiler called c89 which implements exactly that spec and nothing else. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Robert Haas robertmh...@gmail.com writes: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design yet. I don't think that it's about SELinux. The real issue here is that KaiGai-san is about a mile out in front of the PG hackers community in terms of his ambitions for the scope of what can be controlled by security policy. If the patch were only doing what the community has actually agreed to, there would be little need for it to touch anything but the aclcheck functions. Now I recognize that a large part of the potential attraction in this for the security community is exactly the idea of having fine-grain security control. But if you ever want anything significantly different from SQL-standard permission mechanisms, there's going to have to be a whole lot more work done. Basically, nobody in the PG community has got any confidence either in the overall design or the implementation details for locking things down that aren't already controlled by SQL permission mechanisms. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Kevin Grittner wrote: After running a series of tests with --enable-cassert to confirm that nothing squawked, I disabled that before doing any performance testing. Going from memory, I used --enable-debug --with-libxml and --prefix. I didn't explicitly use or disable any compiler optimizations. Possibly relevant is that I was using Greg Smith's peg tool (as best I could): http://github.com/gregs1104/peg/ I'm not aware of it doing anything to the compile options, but I didn't look for that, either. Now that you ask, of course I just spotted a bug in there such that the documented behavior for the PGDEBUG feature doesn't actually work. If you were using that to turn off asserts, that may not have worked as expected. Don't know what you did there exactly. Fix now pushed to the repo. I general, when doing performance testing, now matter how careful I've been I try to do a: show debug_assertions; To confirm I'm not running with those on. Andres, are using any optimization flags when you're testing? Would have preferred that the first mention of my new project didn't involve a bug report, but that's software I guess. For everyone here who's not on the reviewers mailing list: peg is a scripting tool I've put together recently that makes it easier to setup the environment (source code, binaries, database) for testing PostgreSQL in a development content. It takes care of grabbing the latest source, building, starting the database, all that boring stuff. I tried to make it automate the most tedious parts of both development and patch review. Documentation and the program itself are at the git repo Kevin mentioned. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] More broken links in documentation
I wrote a small script to check our SGML, and the following links are returning not found in our docs: ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf in charset.sgml, detailed info about some charsets. Do we just remove this? Anybody know if similar info is available elsewhere? http://db.cs.berkeley.edu/~jmh/ reference to a professor at berkeley whos work it's based on. I suggest we just remove the link, but obviously keep the name. http://garcia.me.berkeley.edu/~adong/rtree reference to a student at berkeley in the same place (cube.sgml). Same suggestion as above. ftp://ftp.sco.com/skunkware Link to sco skunkware which supposedly is available there if you don't have the CD, in the installation notes for Unixware and OpenServer. Which it isn't. Suggest we just remove this sentence, but leave the reference to the CD. http://www.cs.ut.ee/~helger/crypto/ General technical references link in pgcrypto with no more details. Suggest we just remove it. Anybody objecting to just removing this stuff per the list above? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Greg Smith g...@2ndquadrant.com wrote: Now that you ask, of course I just spotted a bug in there such that the documented behavior for the PGDEBUG feature doesn't actually work. If you were using that to turn off asserts, that may not have worked as expected. Don't know what you did there exactly. Fix now pushed to the repo. Thanks. I was going to reply to your original message with my experiences (and probably still will), but it seemed like it might be relevant here. I did check pg_config results before doing anything, and saw that the debug and cassert weren't set, so that's why I did explicit configure and make commands. Even without that covered, peg was a nice convenience -- I can say that it saved me more time already than it took to install and read the docs. Nice work! Anyway, I'm not sure whether your reply directly answers the point I was raising -- peg doesn't do anything with the compiler optimization flags under the covers, does it? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, Dec 8, 2009 at 1:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design yet. I don't think that it's about SELinux. The real issue here is that KaiGai-san is about a mile out in front of the PG hackers community in terms of his ambitions for the scope of what can be controlled by security policy. If the patch were only doing what the community has actually agreed to, there would be little need for it to touch anything but the aclcheck functions. Now I recognize that a large part of the potential attraction in this for the security community is exactly the idea of having fine-grain security control. But if you ever want anything significantly different from SQL-standard permission mechanisms, there's going to have to be a whole lot more work done. Basically, nobody in the PG community has got any confidence either in the overall design or the implementation details for locking things down that aren't already controlled by SQL permission mechanisms. I think that's basically right. Further, I think this is basically a resource issue. If you were inclined to spend a large amount of your time on this problem, you could either gain confidence in the present design and implementation or come up with a new one in which you did have confidence. But it doesn't seem important enough to you (or your employer) for the amount of time it would take, so you're not. I think there are other committers and community members in a similar situation - basically all of them. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Install chapter broken link
2009/12/7 Greg Smith g...@2ndquadrant.com: Magnus Hagander wrote: Our instalation chapter (15.2) has a link to http://developer.postgresql.org/~petere/bsd-gettext/ to get gettext on other systems. this link is broken. Can somebody provide a proper one, or should we remove it? Goals of his version described at http://archives.postgresql.org/pgsql-hackers/2001-05/msg01114.php lest anyone wonder why there was a personal port here (I know I did). The version most like that available now would be the ones listed at ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/gettext/README.html The only popular platform I'm aware of that doesn't include it nowadays is OS X: http://blog.doughellmann.com/2009/06/installing-gnu-gettext-for-use-with.html which does mean the need to install your own hasn't completely gone away. Looks like all the various free BSDs are using GNU gettext now; that's what I found in the NetBSD link above, and at http://www.freebsd.org/cgi/cvsweb.cgi/ports/devel/gettext/ Even Solaris aims to be compatible with the GNU version as of 2004. I think http://www.gnu.org/software/gettext/ is the appropriate new link destination. Seems reasonable, thanks for the investigation! Change applied. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On 12/8/09 12:36 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Dec 8, 2009 at 12:16 PM, Chad Sellers csell...@tresys.com wrote: On 12/8/09 11:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Tue, 2009-12-08 at 11:48 -0500, Robert Haas wrote: On Tue, Dec 8, 2009 at 10:51 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Mon, 2009-12-07 at 17:57 -0500, Robert Haas wrote: On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian br...@momjian.us wrote: As Alvaro mentioned, the original patch used ACE but it added too much code so the community requested its removal from the patch. It could be re-added if we have a need. Well, there's no point in putting that framework back in unless we can make it sufficiently general that it could be used to serve the needs of more than one security model. And so far, the signs have not been promising. David Quigley suggests downthread that making a truly general model isn't really possible, and he may be right, or not. I was just mentioning that it's an angle I have been thinking about investigating, but it may be a dead end. The real issue is making the code committable, and then maintaining it, as Tom rightly says, forever. We've got to make sure that we're willing to take that on before we do it, and I don't think it's a small task. It isn't so much whether we want the feature as whether the level of effort is proportionate to the benefit. ...Robert So the issue with generality is that path name based MAC has a different set of requirements than label based does. If you want to acomodate several types of label based MAC models then a framework can be developed for that similar to the one in the Linux Kernel. I asked around after I sent the email yesterday and from what I understand AppArmor does not have the concept of a userspace object manager so I don't know what they'd do in this area. I'm sure they could come up with a scheme to write policy for the database but I don't know how useful it would be. If you look at the LSM framework in the Linux Kernel recently there have been hooks added to accomodate path based MAC systems so it can be done but adds another set of hooks. The important goal here though in designing a framework is to make sure that you have a comprehensive list of the objects you want to mediate and make sure you put the proper enforcement points in. Someone may come along later and want to mediate a new object or some new functionality may come along that introduces a new object so a hook may then need to be added. Something to realize as well is that a security model may not want to implement all of the hooks and it doesn't have to. In the case of no module being loaded or someone not wanting the loadable security module framework I'm sure we can provide an option to reduce overhead or compile the framework out completely. I'll take a look at his patches for the framework that KaiGai originally proposed. It sounds like the pathname-based schemes are not really designed to work inside of a database anyway, so my first thought is we shouldn't worry about them. The label-based schemes are by their nature designed to apply in an arbitrary context being applied to arbitrary objects. ...Robert So I was reading through a set of slides that KaiGai has and he mentioned a May commitfest link and I looked for the comments related to his PGACE patches. I've been crawling through the commitfest paces so I can figure out what the latest version of the pgace patch is. Does anyone know when the patch was reduced to what it is today? Dave I'm another SELinux developer and I'd like to help out where I can here. I'm a bit confused by this discussion of PGACE. I thought the postgresql community agreed that they wanted this removed in order to make the patch size smaller. Has that changed? Is the increase in patch size now acceptable? Sorry if I'm joining the conversation late. Thanks, Chad Sellers Nothing's been decided. Sorry, my mistake. This mailing list moves pretty quick so it's hard to catch up with everything. We're just trying to figure out the best way to tackle the problem. I think the main question here is who if anyone from among the committers is willing to put in the time and effort to maintain this feature over the short and long haul, but that's sort of an internal project issue. I was just thinking out loud about whether it was possible and/or desirable to try to modularize this a bit so that it could be used for more than just SE-Linux. So, a generalized framework is nice in that it supports multiple models, but it does bring with it an additional maintenance burden. I'm sure that's been discussed at length already. I will say that it's almost impossible to place hooks properly for imaginary users. So, if you create a framework, it's probably just a placeholder with hooks for the one user (SEPostgreSQL) that will later have
Re: [HACKERS] More broken links in documentation
2009/12/8 Magnus Hagander mag...@hagander.net I wrote a small script to check our SGML, and the following links are returning not found in our docs: ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf in charset.sgml, detailed info about some charsets. Do we just remove this? Anybody know if similar info is available elsewhere? http://db.cs.berkeley.edu/~jmh/ reference to a professor at berkeley whos work it's based on. I suggest we just remove the link, but obviously keep the name. The link http://db.cs.berkeley.edu/jmh/ works. Could you replace it with that? http://www.cs.ut.ee/~helger/crypto/ General technical references link in pgcrypto with no more details. Suggest we just remove it. I've found http://research.cyber.ee/~lipmaa/crypto/ works. Looking at the Google cache for the original link, it appears to contain the same stuff. Thom
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Hi, On Tuesday 08 December 2009 20:09:22 Greg Smith wrote: Andres, are using any optimization flags when you're testing? I was testing with and without debug/cassert - and did not get adverse results in both... Unfortunately it looks like I wont get to test today, but only tomorrow morning its 9pm and I am not yet fully finished with work Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More broken links in documentation
Magnus Hagander wrote: ftp://ftp.sco.com/skunkware Link to sco skunkware which supposedly is available there if you don't have the CD, in the installation notes for Unixware and OpenServer. Which it isn't. Suggest we just remove this sentence, but leave the reference to the CD. This is now at http://www.sco.com/skunkware/ -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More broken links in documentation
2009/12/8 Thom Brown thombr...@gmail.com: 2009/12/8 Magnus Hagander mag...@hagander.net I wrote a small script to check our SGML, and the following links are returning not found in our docs: ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf in charset.sgml, detailed info about some charsets. Do we just remove this? Anybody know if similar info is available elsewhere? http://db.cs.berkeley.edu/~jmh/ reference to a professor at berkeley whos work it's based on. I suggest we just remove the link, but obviously keep the name. The link http://db.cs.berkeley.edu/jmh/ works. Could you replace it with that? Clearly my google-fu was failing me badly. I have no idea why. Changed to that. Also, the rtree one appears to be at http://best.berkeley.edu/~adong/rtree/index.html - first hit in my second attempt to google it, no idea how I missed it. http://www.cs.ut.ee/~helger/crypto/ General technical references link in pgcrypto with no more details. Suggest we just remove it. I've found http://research.cyber.ee/~lipmaa/crypto/ works. Looking at the Google cache for the original link, it appears to contain the same stuff. Yeah, same comment about google-fu. Thanks! -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More broken links in documentation
2009/12/8 Greg Smith g...@2ndquadrant.com: Magnus Hagander wrote: ftp://ftp.sco.com/skunkware Link to sco skunkware which supposedly is available there if you don't have the CD, in the installation notes for Unixware and OpenServer. Which it isn't. Suggest we just remove this sentence, but leave the reference to the CD. This is now at http://www.sco.com/skunkware/ That's it, I'm out of excuses :-) Thanks! That leaves just the ora.com link. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Kevin Grittner wrote: Anyway, I'm not sure whether your reply directly answers the point I was raising -- peg doesn't do anything with the compiler optimization flags under the covers, does it? Not really. It does this: PGDEBUG=--enable-cassert --enable-debug ./configure --prefix=$PGINST/$PGPROJECT --enable-depend --enable-thread-safety $PGDEBUG Which are pretty standard options. The idea is that you'd use the default normally, then just set PGDEBUG= for a non-debug build--or to otherwise change the configure flags but still get things done automatically for you. If it's set before the script starts, it doesn't change it. I did try to design things so that you could do any step in the automated series manually and not have that screw things up. There's hundreds of lines of code in there just for things like figuring out whether configure has been run or not yet when it decides you need to build, so it can try to do the right thing in either case. My hope was that anyone who tried peg out would find it a net positive time savings after a single use, glad to hear I accomplished that in your case. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More broken links in documentation
Magnus Hagander wrote: ftp://ftp.ora.com/pub/examples/nutshell/ujip/doc/cjk.inf in charset.sgml, detailed info about some charsets. Do we just remove this? Anybody know if similar info is available elsewhere? Now http://examples.oreilly.com/cjkvinfo/doc/cjk.inf -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Greg Smith g...@2ndquadrant.com writes: Kevin Grittner wrote: Anyway, I'm not sure whether your reply directly answers the point I was raising -- peg doesn't do anything with the compiler optimization flags under the covers, does it? Not really. It does this: PGDEBUG=--enable-cassert --enable-debug ./configure --prefix=$PGINST/$PGPROJECT --enable-depend --enable-thread-safety $PGDEBUG --enable-cassert might have a noticeable performance impact. We usually try to not have that on when doing performance testing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More broken links in documentation
2009/12/8 Magnus Hagander mag...@hagander.net: 2009/12/8 Greg Smith g...@2ndquadrant.com: Magnus Hagander wrote: ftp://ftp.sco.com/skunkware Link to sco skunkware which supposedly is available there if you don't have the CD, in the installation notes for Unixware and OpenServer. Which it isn't. Suggest we just remove this sentence, but leave the reference to the CD. This is now at http://www.sco.com/skunkware/ That's it, I'm out of excuses :-) Thanks! That leaves just the ora.com link. Found a mirror at http://examples.oreilly.com/cjkvinfo/doc/cjk.inf, which says the master should be on that ftp site. They're both oreilly sites, but they seem to have dropped that book from the list on the ftp site. Do we dare link to that examples site, or should we expect that one to go away as well? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Tom Lane wrote: --enable-cassert might have a noticeable performance impact. We usually try to not have that on when doing performance testing. All covered in the tool's documentation, and it looks like Kevin did the right thing during his tests by checking the pg_config output to confirm the right flags were used. I think we can rule out simple pilot error here and work under the assumption Kevin found a mild performance regression under some circumstances with the patch. Hopefully Andres will be able to replicate the problem, and it sounds like Kevin might be able to provide more information about it tonight too. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
Tom Lane t...@sss.pgh.pa.us wrote: --enable-cassert might have a noticeable performance impact. We usually try to not have that on when doing performance testing. Right. I turned it on for some initial tests to confirm that we had no assertion failures; then turned it off for the performance testing. I did leave --enable-debug on during performance testing. My understanding is that debug info doesn't affect performance, but I guess it never hurts to try an empirical test to confirm. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, 2009-12-08 at 14:22 -0500, Robert Haas wrote: On Tue, Dec 8, 2009 at 1:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design yet. I don't think that it's about SELinux. The real issue here is that KaiGai-san is about a mile out in front of the PG hackers community in terms of his ambitions for the scope of what can be controlled by security policy. If the patch were only doing what the community has actually agreed to, there would be little need for it to touch anything but the aclcheck functions. Now I recognize that a large part of the potential attraction in this for the security community is exactly the idea of having fine-grain security control. But if you ever want anything significantly different from SQL-standard permission mechanisms, there's going to have to be a whole lot more work done. Basically, nobody in the PG community has got any confidence either in the overall design or the implementation details for locking things down that aren't already controlled by SQL permission mechanisms. I think that's basically right. Further, I think this is basically a resource issue. If you were inclined to spend a large amount of your time on this problem, you could either gain confidence in the present design and implementation or come up with a new one in which you did have confidence. But it doesn't seem important enough to you (or your employer) for the amount of time it would take, so you're not. I think there are other committers and community members in a similar situation - basically all of them. ...Robert I have to agree with Chad (downthread) that I don't see much in KaiGai's hook patch that prevents its use by other security models. I will say though one thing that might have been done wrong was with how it was presented. In actuality his patch set is two projects (at least). The first is the framework. So I think the goal should have been to get the framework integrated first and then work on the SELinux module after that. The framework patch alone consists of at least 4 sets of logical changes that could have been separated to make review easier. Once the framework was in, work could be done to get the SELinux module in while reducing overhead from the case where no module is loaded. So with regard to confidence in the design I think that part of the reason for the skepticism in the fact that it was such a large code drop. Even the separated parts were very large. I think if the framework patches are broken up more logically and in a way that is easier to discuss then that might help the community get a grasp on what it is trying to accomplish. In terms of documentation I was reading through the wiki at sepgsql.googlecode.com and aside from some wordsmithing/grammar things it is pretty solid with describing what it is trying to accomplish. One problem that I see is that at first glance it does appear to be very SELinux centric. It describes access based on types and SELinux contexts which is understandable based on the fact that it describes the framework and the module. Something to note is that the documentation describes an object model for the program. I think that would be a good place to focus the discussion with respect to a framework if we decide to pursue it. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
* Robert Haas (robertmh...@gmail.com) wrote: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. I've really got to take exception to this. I've only been following along and not really participating because, to be honest, I'm frustrated with how this has gone down. In the end there were at least two patches, in my view, that *didn't* involve adding SE-Linux-specific checks everywhere. The patch that I reviewed that got thrown out by Tom, and the original PGACE framework. Both of those added alot of *hooks*, because they were necessary, but nothing made those hooks particularly SELinux-specifc. We're hearing alot about things being SELinux-specific from people who also profess to not know SELinux. Indeed, as I recall, the patch I reviewed was primairly trying to just addess pulling out the hooks necessary for the existing PostgreSQL security model. Very little of it was SE-Linux specific *anything*. I contend that while the specific hooks which would be added *in places that don't already have checks* (in most places, the hook was added to replace an existing check) are hooks that then make sense for SELinux, they would also make sense for other frameworks. They may also not be a complete list, but once the *framework* is in place, adding new hooks (presuming another model would like a hook somewhere that SELinux and the existing PG security framework don't) should not require some kind of forklift upgrade. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design yet. The problem is that the existing permissions-checking is done all over the place. That's not ideal from the get-go, in my opinion. Unfortuantely, when we moved all of the permissions-checking to a single place, it required knowing about alot of the backend, which Tom took exception to. I agree that's frustrating, but I don't see it as a particular reason to throw out the entire concept of a modular security framework. Perhaps we need to better expose just those pieces the security framework cares about from the other parts of the backend- it's entirely likely that the reason it has to know about everything is that, due to where all the existing security checks are, they just (ab)used the fact that they had access to that information at hand for that object type. Consequently there are checks spread throughout the code, which definitely complicates both validation and maintenance. One question I have is - are the places where those checks are placed specific to SE-Linux, or would they be applicable to any sort of label-based MAC? The patch which I worked with Kaigai on was specifically geared to first try to get a patch which addressed the existing PG security model in a modular way, to allow additional hooks to be added later in places which needed them, and to provide the information for other models to make a decision about the permission. I don't feel it was particularly SE-Linux specific at all, but rather a first step towards being able to support something beyond the model we have today (anything..). Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Adding support for SE-Linux security
On Tue, Dec 8, 2009 at 2:50 PM, David P. Quigley dpqu...@tycho.nsa.gov wrote: On Tue, 2009-12-08 at 14:22 -0500, Robert Haas wrote: On Tue, Dec 8, 2009 at 1:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design yet. I don't think that it's about SELinux. The real issue here is that KaiGai-san is about a mile out in front of the PG hackers community in terms of his ambitions for the scope of what can be controlled by security policy. If the patch were only doing what the community has actually agreed to, there would be little need for it to touch anything but the aclcheck functions. Now I recognize that a large part of the potential attraction in this for the security community is exactly the idea of having fine-grain security control. But if you ever want anything significantly different from SQL-standard permission mechanisms, there's going to have to be a whole lot more work done. Basically, nobody in the PG community has got any confidence either in the overall design or the implementation details for locking things down that aren't already controlled by SQL permission mechanisms. I think that's basically right. Further, I think this is basically a resource issue. If you were inclined to spend a large amount of your time on this problem, you could either gain confidence in the present design and implementation or come up with a new one in which you did have confidence. But it doesn't seem important enough to you (or your employer) for the amount of time it would take, so you're not. I think there are other committers and community members in a similar situation - basically all of them. I have to agree with Chad (downthread) that I don't see much in KaiGai's hook patch that prevents its use by other security models. I will say though one thing that might have been done wrong was with how it was presented. In actuality his patch set is two projects (at least). The first is the framework. So I think the goal should have been to get the framework integrated first and then work on the SELinux module after that. The framework patch alone consists of at least 4 sets of logical changes that could have been separated to make review easier. Once the framework was in, work could be done to get the SELinux module in while reducing overhead from the case where no module is loaded. I can say from experience that this project is very skeptical of frameworks that aren't accompanied by at least one, and preferably multiple, working implementations. So there is a bit of a chicken and egg problem here. What can sometimes work is to say, look, here's a place where I can put a hook and later I will do something complex with it but here are a couple of relatively simple but useful examples to get started. The examples form the justification for the commit because they are independently useful, but they are much simpler than what the framework may eventually end up being used for. I don't believe that this approach is feasible for SE-PostgreSQL. A simple version of label security is still going to be very complicated, and there are probably even fewer customers for such a thing than there are for SE-PostgreSQL. So with regard to confidence in the design I think that part of the reason for the skepticism in the fact that it was such a large code drop. Even the separated parts were very large. Definitely true. I think if the framework patches are broken up more logically and in a way that is easier to discuss then that might help the community get a grasp on what it is trying to accomplish. Maybe, maybe not. Nobody's going to commit anything unless it's a complete feature. It can be a cut-down feature, but it has to be of independent use. If there are parts that can be peeled off of SE-PostgreSQL and committed independently to some good benefit, then I think that's a great idea. But if it's just a separation of the patch for clarity, I don't think there's much value in that. In terms of documentation I was reading through the wiki at sepgsql.googlecode.com and aside from some wordsmithing/grammar things it is pretty solid with describing what it is trying to accomplish. One problem that I see is that at first glance it does appear to be very SELinux centric. It describes access based on types and SELinux contexts which is understandable based on the fact that it describes the framework and the module. Something to note is that the documentation describes an object
Re: [HACKERS] [PATCH] dtrace probes for memory manager
--On 13. November 2009 17:16:15 -0500 Robert Haas robertmh...@gmail.com wrote: Don't think. Benchmark. :-) (If you can measure it at all, it's too much, at least IMHO.) I've tried to benchmark this now on my (fairly slow compared to server hardware) MacBook and see some negative trend for those memory probes in pgbench. Running dozens of rounds with pgbench (scale 150, 10 clients / 1000 transactions) gives the following numbers (untuned PostgreSQL config): AVG(tps) with dtrace memory probes: 31.62 tps AVG(tps) without dtrace memory probes: 38.24 tps So there seems to be a minor slowdown at least on *my* machine. However, it would be fine if someone can prove these numbers.. What do you guys think, what other tests/parameters can be invoked to test for an impact ? -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
Bernd Helmle wrote: I've tried to benchmark this now on my (fairly slow compared to server hardware) MacBook and see some negative trend for those memory probes in pgbench. Running dozens of rounds with pgbench (scale 150, 10 clients / 1000 transactions) That makes for a 5.5 minute test, which is unfortunately close to the default checkpoint period. You're going to want a pgbench configuration that's doing thousands of operations per second to measure this overhead I think, and let it run a bit longer. The difference you're seeing could easily be just that that the with probes result had more checkpoints happen during testing than the other one--if it got even a single checkpoint more, that could be enough to throw results off using the default test and such low TPS results. Try this instead, which will give you a test where checkpoints have a minimal impact, but lots of memory will be thrown around: pgbench -i -s 10 db pgbench -S -c 10 -T 600 db That will do just SELECT statements against a much smaller database (about 160MB) and will run for 10 minutes each time. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] dtrace probes for memory manager
--On 8. Dezember 2009 15:51:52 -0500 Greg Smith g...@2ndquadrant.com wrote: Try this instead, which will give you a test where checkpoints have a minimal impact, but lots of memory will be thrown around: pgbench -i -s 10 db pgbench -S -c 10 -T 600 db Thanks for the input, will try -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ECPG patch 2, SQLDA support
On Mon, Nov 16, 2009 at 5:59 AM, Boszormenyi Zoltan z...@cybertec.at wrote: New version: rebased to current CVS. This one no longer applies to HEAD, could you update it please? -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, 2009-12-08 at 15:24 -0500, Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. I've really got to take exception to this. I've only been following along and not really participating because, to be honest, I'm frustrated with how this has gone down. In the end there were at least two patches, in my view, that *didn't* involve adding SE-Linux-specific checks everywhere. The patch that I reviewed that got thrown out by Tom, and the original PGACE framework. Both of those added alot of *hooks*, because they were necessary, but nothing made those hooks particularly SELinux-specifc. We're hearing alot about things being SELinux-specific from people who also profess to not know SELinux. Indeed, as I recall, the patch I reviewed was primairly trying to just addess pulling out the hooks necessary for the existing PostgreSQL security model. Very little of it was SE-Linux specific *anything*. I contend that while the specific hooks which would be added *in places that don't already have checks* (in most places, the hook was added to replace an existing check) are hooks that then make sense for SELinux, they would also make sense for other frameworks. They may also not be a complete list, but once the *framework* is in place, adding new hooks (presuming another model would like a hook somewhere that SELinux and the existing PG security framework don't) should not require some kind of forklift upgrade. It would be nice if it were possible to use the exist permissions-checking functions and have them check a few more things while they're at it, but it's looking like that won't be feasible, or at least no one's come up with a plausible design yet. The problem is that the existing permissions-checking is done all over the place. That's not ideal from the get-go, in my opinion. Unfortuantely, when we moved all of the permissions-checking to a single place, it required knowing about alot of the backend, which Tom took exception to. I agree that's frustrating, but I don't see it as a particular reason to throw out the entire concept of a modular security framework. Perhaps we need to better expose just those pieces the security framework cares about from the other parts of the backend- it's entirely likely that the reason it has to know about everything is that, due to where all the existing security checks are, they just (ab)used the fact that they had access to that information at hand for that object type. Consequently there are checks spread throughout the code, which definitely complicates both validation and maintenance. One question I have is - are the places where those checks are placed specific to SE-Linux, or would they be applicable to any sort of label-based MAC? The patch which I worked with Kaigai on was specifically geared to first try to get a patch which addressed the existing PG security model in a modular way, to allow additional hooks to be added later in places which needed them, and to provide the information for other models to make a decision about the permission. I don't feel it was particularly SE-Linux specific at all, but rather a first step towards being able to support something beyond the model we have today (anything..). Thanks, Stephen I think what makes people think that the changes are SELinux specific is that the examples given use SELinux contexts. I think it should be made clear that saying that in the PG there is a database object and we want to mediate access to create database objects is not SELinux or even MAC model specific (asside from labels). However to say that a program labeled myapp_t can connect to the database and create a table and when its created its labeled mytable_t that would be something SELinux specific. The framework idea separates that. It says hey here are the objects in the system and here are the actions on them that we want to mediate. I will admit that since SELinux is the driving MAC model for the framework you're going to find that the objects and permissions are the ones that it wants to control. However like Steven said, adding a hook later on or having a model not use a hook is not a Herculean task. We've proven time and time again with the LSM framework that when a feature is added it is trivial to introduce a new hook to mediate it or to place an existing hook. I understand that PostgreSQL is a fast moving target with a large developer base but so is the Linux Kernel and a similar framework has been working there for years now. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On mån, 2009-12-07 at 17:33 +0100, Martijn van Oosterhout wrote: On Mon, Dec 07, 2009 at 01:09:59PM -0300, Alvaro Herrera wrote: Given the extreme patience and diligence exhibited by KaiGai, I hesitate to say this, but it seems to me that this would be critically important for the long term success of this feature. I have no idea how much work it would be to make the interface to the external security system pluggable, but if it's at all feasible, I think it should be done. This is how the code was developed initially -- the patch was called PGACE and SELinux was but the first implementation on top of it. I find it astonishing that after SE-PgSQL was implemented on top of a pluggable system (PGACE) and this system was removed at request of the community [1] that at this late phase people are suggesting it needs to be added back again. Havn't the goalposts been moved enough times? PGACE wasn't a plugin system. It was an API inside the core code. If it had been a plugin system, this would have been much easier, because the plugin itself could have been developed independently. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On mån, 2009-12-07 at 11:45 -0500, Chris Browne wrote: I feel about the same way about this as I did about the adding of native Windows support; I'm a bit concerned that this could be a destabilizing influence. I was wrong back then; the Windows support hasn't had the ill effects I was concerned it might have. Windows support certainly has had ill effects, but the benefits of supporting Windows clearly outweigh that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, Dec 8, 2009 at 3:24 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. I've really got to take exception to this. I've only been following along and not really participating because, to be honest, I'm frustrated with how this has gone down. In the end there were at least two patches, in my view, that *didn't* involve adding SE-Linux-specific checks everywhere. The patch that I reviewed that got thrown out by Tom, and the original PGACE framework. Both of those added alot of *hooks*, because they were necessary, but nothing made those hooks particularly SELinux-specifc. We're hearing alot about things being SELinux-specific from people who also profess to not know SELinux. Sorry. I spent a lot of time for both CommitFest 2008-11 and CommitFest 2009-07 in the hopes of getting something committable, and I wasn't successful. I'm just at the end of my rope. It seems fairly clear that Tom isn't going to commit any piece of SE-PostgreSQL at all, ever. So who's going to do it? It doesn't make any sense to continue trucking along with this patch into the indefinite future if it has no hope of being committed. Frankly, I think this comes down to money. There are several PostgreSQL companies which employ very capable PostgreSQL committers. When someone is willing to pony up enough money to get those people interested (as, I gather, has happened with block-checksumming) then this will happen. Until then, I don't believe anyone is going to volunteer to be responsible for a 10,000-line patch in their free time. Tom is the only one crazy enough for that, and he said no. The next time someone submits a huge, unsolicited patch to do ANYTHING, we should do them a favor and tell them this up front, rather than a year and a half later. Then they could have the appropriate conversations with the appropriate people and determine whether to budget for it or give up. What has happened with this patch has not served KaiGai well, or improved the image of this community. I agree that's frustrating, but I don't see it as a particular reason to throw out the entire concept of a modular security framework. I don't either. There were certainly technical things in the previous patch that could stand to have been improved, and I think the general approach has some potential. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, 2009-12-08 at 15:26 -0500, Robert Haas wrote: [snip...] I can say from experience that this project is very skeptical of frameworks that aren't accompanied by at least one, and preferably multiple, working implementations. So there is a bit of a chicken and egg problem here. What can sometimes work is to say, look, here's a place where I can put a hook and later I will do something complex with it but here are a couple of relatively simple but useful examples to get started. The examples form the justification for the commit because they are independently useful, but they are much simpler than what the framework may eventually end up being used for. I don't believe that this approach is feasible for SE-PostgreSQL. A simple version of label security is still going to be very complicated, and there are probably even fewer customers for such a thing than there are for SE-PostgreSQL. Ok if it is a chicken and egg problem then we need to find a smaller egg. I agree that having a huge patch set isn't ideal which is why I understand the desire to have KaiGai cut back his patches. However the patches he is putting forward from what I've gathered have no row based labeling and no MAC enforcement. I can understand if you want to cut back the object model so you're not mediating every object in the system but cutting those two features make it unusable by the customers we have who want to use it. So with regard to confidence in the design I think that part of the reason for the skepticism in the fact that it was such a large code drop. Even the separated parts were very large. Definitely true. I think if the framework patches are broken up more logically and in a way that is easier to discuss then that might help the community get a grasp on what it is trying to accomplish. Maybe, maybe not. Nobody's going to commit anything unless it's a complete feature. It can be a cut-down feature, but it has to be of independent use. If there are parts that can be peeled off of SE-PostgreSQL and committed independently to some good benefit, then I think that's a great idea. But if it's just a separation of the patch for clarity, I don't think there's much value in that. I don't think it would be just for clarity. I know that not all open source communities are the same so I'll try to leave the anecdotal evidence light. When submit patches for the Linux Kernel we take a single feature and structure them as self contained logical changes. Separating the changes logically doesn't just improve clarity but also makes it easy to cherry pick features that can be useful on their own. Just because the framework and the SEPostgreSQL features would be two patch sets doesn't mean they aren't being developed in parallel. It looks to me that we're in the same boat we were in with SELinux. We had the feature and proposed it and someone brought up well what about other security models. So time was spent developing a framework that everyone could use and as that was being done the SELinux patches were modified to use this new framework. They were two separate features developed in tandem. If we do it right the SEPostgreSQL code will be isolated from the framework and we can spend time putting the framework in and just plug in the specific security module when time comes. The X-ACE work went in to X before the corresponding SELinux Flask module for it and I don't believe LSM and SELinux went into the same merge window as well. I have proofread earlier versions of the docs and found them inadequate. There were language issues, but the bigger problem was that they were both too specific and yet not sufficiently detailed. For example, they'd say install X package on Red Hat. Well, what if I'm not on Red Hat? The on the other hand they'd say this GUC enables mcstrans which means nothing to me. I think this has been improved somewhat in more recent version, but clearly we need (1) good developer documentation, so someone other than KaiGai has a chance of maintaining the code and keeping it correct as other things are changed; (2) user documentation, so that someone can read the feature story for this capability; and (3) reference documentation, cross-referenced with the user documentation. Having said that, fixing the documentation won't get this patch committed, though it's certainly a step in the right direction. For that we need a committer-owner. True but hopefully proper documentation will help someone decide that they have enough information to take on that position. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Peter Eisentraut pete...@gmx.net writes: PGACE wasn't a plugin system. It was an API inside the core code. If it had been a plugin system, this would have been much easier, because the plugin itself could have been developed independently. Well, it should certainly have used function pointers or something to allow better pluggability, but that would have been a trivial change. I don't believe that doing so would have made development any easier. The real problem in all this is to answer the question do we have the right hooks in the right places?. Whether the hooks lead to function pointers or hard-wired calls doesn't enter into that. Moreover, since we can confidently say that all the early answers will be no, it would be a serious mistake to try to develop the plugin independently. Having to keep two independent sets of source code in sync would waste a lot of effort every time you realized you needed to adjust the hook definitions. Once you'd gotten to a releasable state maybe you could assume the hook definitions would become stable, but right now I have no confidence in that at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tsearch parser inefficiency if text includes urls or emails - new version
I wrote: Perhaps it is some quirk of using 32 bit pointers on the 64 bit AMD CPU? (I'm looking forward to testing this today on a 64 bit build on an Intel CPU.) The exact same test on 64 bit OS (SuSE Enterprise Server) on Intel gave very different results. With 10 runs each of 200 iterations of parsing the 1 URLs, the patch Andres submitted ran 0.4% faster than HEAD, and my attempt to improve on it ran 0.6% slower than HEAD. I'll try to run the numbers to get the percentage chance that a random distribution would have generated a spread as large as either of those; but I think it's safe to say that the submitted patch doesn't hurt there and that my attempt to improve on it was misdirected. :-/ I would like to independently confirm the dramatic improvement reported by Andres. Could I get a short snippet from the log which was used for that, along with an indication of the size of the text parsed in that test? (Since the old code looks like it might have O(N^2) performance in some situations, while the patch changes that to O(N), I might not be testing with a big enough N.) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
On Tue, 2009-12-08 at 16:51 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: PGACE wasn't a plugin system. It was an API inside the core code. If it had been a plugin system, this would have been much easier, because the plugin itself could have been developed independently. Well, it should certainly have used function pointers or something to allow better pluggability, but that would have been a trivial change. I don't believe that doing so would have made development any easier. The real problem in all this is to answer the question do we have the right hooks in the right places?. Whether the hooks lead to function pointers or hard-wired calls doesn't enter into that. Moreover, since we can confidently say that all the early answers will be no, it would be a serious mistake to try to develop the plugin independently. Having to keep two independent sets of source code in sync would waste a lot of effort every time you realized you needed to adjust the hook definitions. Once you'd gotten to a releasable state maybe you could assume the hook definitions would become stable, but right now I have no confidence in that at all. regards, tom lane I disagree with several of your statements above. While the question of whether or not the hooks are in the right place is up for debate what the hooks should be is something that I think has been explored pretty well. The hooks should reflect the object model. Where you need to put them to enforce permission is a different story. Also maintaining a module and a framework is not a waste of time. If your security module has a tremendous code churn when the interface to the program is modified you should reevaluate your design. Both SELinux and the Flask X-ACE modules are examples where an existing extension had a modular framework developed after the fact and easily integrated into the project. The majority of the work is moving to a framework not changing function prototypes in code that's already in the framework. Dave -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Robert Haas wrote: On Tue, Dec 8, 2009 at 10:07 AM, David P. Quigley dpqu...@tycho.nsa.gov wrote: I'd be willing to take a look at the framework and see if it really is SELinux centric. If it is we can figure out if there is a way to accomodate something like SMACK and FMAC. I'd like to hear from someone with more extensive experience with Solaris Trusted Extensions about how TX would make use of this. I have a feeling it would be similar to the way it deals with NFS which is by having the process exist in the global zone as a privileged process and then multi-plexes it to the remaining zones. That way their getpeercon would get a label derived from the zone. Well, the old patches should still be available in the mailing list archives. Maybe going back and looking at that code would be a good place to start. The non-ripped-out code has been cleaned up a lot since then, but at least it's a place to start. We can see old branches here: http://code.google.com/p/sepgsql/source/browse/branches/pgsql-8.3.x/sepgsql/src/backend/security/pgaceHooks.c But I don't provide this framework for the 8.4.x/8.5.x, because this idea was rejected in the earlier discussion. Please consider it represent just a concept. Thanks. -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Robert Haas wrote: Sorry. I spent a lot of time for both CommitFest 2008-11 and CommitFest 2009-07 in the hopes of getting something committable, and I wasn't successful. I'm just at the end of my rope. It seems fairly clear that Tom isn't going to commit any piece of SE-PostgreSQL at all, ever. So who's going to do it? It doesn't make any sense to continue trucking along with this patch into the indefinite future if it has no hope of being committed. Frankly, I think this comes down to money. There are several PostgreSQL companies which employ very capable PostgreSQL committers. When someone is willing to pony up enough money to get those people interested (as, I gather, has happened with block-checksumming) then this will happen. Until then, I don't believe anyone is going to volunteer to be responsible for a 10,000-line patch in their free time. Tom is the only one crazy enough for that, and he said no. I have offered to review/commit the patch. I don't promise my effort will be pretty, but I will get the job done. I have not started yet because I think we are still unclear if the feature is worth the additional code maintenance. I frankly think the patch should be thought of as the SE-Linux-specific directory files, which KaiGai can maintain, and the other parts, which I think I can handle. The next time someone submits a huge, unsolicited patch to do ANYTHING, we should do them a favor and tell them this up front, rather than a year and a half later. Then they could have the appropriate conversations with the appropriate people and determine whether to budget for it or give up. What has happened with this patch has not served KaiGai well, or improved the image of this community. Yes, this has not been our finest hour. :-( I think the causes have been explained already: o early patches did not have community buy-in o we are unclear about the size of the user community o we are unclear what the end user will want o the feature is complex o the features is in an unfamiliar problem-domain -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication, some small issues
On Tue, Dec 8, 2009 at 9:05 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I suspect we should have a WAL record to say unlogged operation performed here which a standby database would recognize and throw a large warning up. +1. Seems like a very simple solution. Sounds good. This is not just a problem of SR, so I'll implement it as self-contained feature later. Design: - If relation is not temp and archiving (and streaming replication) is enabled, we log the unlogged OP record including relfilenode of the relation. - If unlogged OP record is found during archive recovery, we register its relfilenode to the hashtable which tracks maybe corrupted relations. If the registered relfilenode is brandnew, we emit warning. Also, the log record indicating DROP TABLE etc is found, we remove its relfilenode from the hashtable. - When restartpoint occurs, we write all the registered relfilenodes to the flat file. - At the end of archive recovery, if there is relfilenode in the hashtable, we emit FATAL error to prevent the server from being brought up. XXX: But this might be too conservative. I believe that some people want to complete archive recovery even if a relation is corrupted, and drop that relation after the server has been activated. So I'm going to provide new recovery.conf parameter specifying whether to let archive recovery fail when some relations might be corrupted. Thought? Am I missing something? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication, some small issues
Fujii Masao masao.fu...@gmail.com writes: Thought? Am I missing something? This seems terribly overdesigned. Just emit a warning when you see the unlogged op record and have done. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The first question is: do we really need to be prepared for that? The XLogRecPtr acknowledgment messages the client sends are very small, and if the client is mindful about not sending them too often - perhaps max 1 ack per 1 received XLOG message - the receive buffer in the backend should never fill up in practice. It's OK to drop that feature. If that's deemed not good enough, we could modify just internal_flush() so that it uses secure_poll to wait for the possibility to either read or write, instead of blocking for just write. Whenever there's incoming data, read them into PqRecvBuffer for later processing, which keeps the OS input buffer from filling up. If PqRecvBuffer fills up, it can be extended, or we can start dropping old XLogRecPtr messages from it. Extending PqRecvBuffer seems better because XLogRecPtr message has some types (i.e., we cannot just drop old message without parsing all messages in the buffer). Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication, some small issues
On Wed, Dec 9, 2009 at 10:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: Thought? Am I missing something? This seems terribly overdesigned. Just emit a warning when you see the unlogged op record and have done. Sounds quite simple. OK, I'll do so. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bug: fuzzystrmatch levenshtein is wrong
On Mon, Dec 7, 2009 at 8:33 AM, marcin mank marcin.m...@gmail.com wrote:
The current behavior of levenshtein(text,text,int,int,int) is wrong. Consider:
leki_dev=# select levenshtein('','a',2,4,5);
levenshtein
-
1
(1 row)
leki_dev=# select levenshtein('a','',2,4,5);
levenshtein
-
1
(1 row)
leki_dev=# select levenshtein('aa','a',2,4,5);
levenshtein
-
1
(1 row)
leki_dev=# select levenshtein('a','aa',2,4,5);
levenshtein
-
1
(1 row)
versus (after patch)
postgres=# select levenshtein('','a',2,4,5);
levenshtein
-
2
(1 row)
postgres=# select levenshtein('a','',2,4,5);
levenshtein
-
4
(1 row)
postgres=# select levenshtein('aa','a',2,4,5);
levenshtein
-
4
(1 row)
postgres=# select levenshtein('a','aa',2,4,5);
levenshtein
-
2
(1 row)
patch attached.
I cannot get this patch to apply for anything. All 4 hunks fail, both
on HEAD and on the the pre-8.4-pgindent version of fuzzystrmatch.c.
Either I'm doing something wrong here, or there's something wrong with
this patch file.
...Robert
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bug: fuzzystrmatch levenshtein is wrong
Robert Haas wrote: patch attached. I cannot get this patch to apply for anything. All 4 hunks fail, both on HEAD and on the the pre-8.4-pgindent version of fuzzystrmatch.c. Either I'm doing something wrong here, or there's something wrong with this patch file. The author converted tabs to spaces --- there is not a single tab in the diff file. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
David P. Quigley wrote: So I was reading through a set of slides that KaiGai has and he mentioned a May commitfest link and I looked for the comments related to his PGACE patches. I've been crawling through the commitfest paces so I can figure out what the latest version of the pgace patch is. Does anyone know when the patch was reduced to what it is today? I could salvage a legacy PGACE patch: http://sepgsql.googlecode.com/files/sepostgresql-pgace-8.4devel-3-r739.patch However, its code base was v8.4devel, so conflictable to the latest CVS HEAD. In addition, it contains various kind of concepts within a single patch. * comprehensive security hooks * a facility to store security identifier on the header of each row * a facility to translate between security identifier (int) and security context (text) * security_context writable system column support. From the current perspective, we can understand these features should be proposed as separated features. But I was young. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Can I ask the final decision whether the YAML formatter should be applied or rejected? As far as I read the discussion, we can apply it because serveral users want it and we don't have a plan to support extensible formatters in the core. Greg Smith g...@2ndquadrant.com wrote: -The patch is small to apply -Having one more legacy format to satisfy would actually improve the odds that a future modular EXPLAIN would be robustly designed -While it's questionable whether it's strictly a majority on voting here, it's close, which suggests there is plenty of support for wanting this feature -Since nothing is removed the people who aren't in favor of this format aren't negatively impacted by it being committed Comments from additional discussion: - YAML format is the second (or the best for some people) human-unreadabe format for EXPLAIN output. - JSON is not always a subset of YAML. It is not recommended to generate/parse YAML with a JSON generator/parser or vice versa. - We won't add any core hooks and plug-in to format EXPLAIN output. Client tools may convert the output if needed. (ex. using XLST) - The spec of YAML might be changed in the feature, but we use only the basic parts. The parts will not be changed. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
David P. Quigley wrote: I understand that PostgreSQL is a fast moving target with a large developer base but so is the Linux Kernel and a similar framework has been working there for years now. It sounds like how you're thinking about this project's development model is inverted from the reality, and it's import to sort that out because it really impacts why there's so much resistance here. One reason the Linux kernel moves so fast because they don't care one lick for many types of backward compatibility, they'll just change interfaces as necessary to work around problems. To use the terms of the old 2.4 Linux kernel, there is no stable branch development happens against anymore; just the fast changing unstable one, that if everyone is lucky eventually converges into some usable versions anyway. Here, there is zero tolerance for any commit that destabilizes the codebase even temporarily. Until you get the whole thing sorted out usefully enough to be shippable quality, you don't go into the main (and only official) branch. Also, the PostgreSQL developers like to deliver a stable release and then change *nothing* to it except to fix bugs, supporting that release for years. We consider a released piece of software like a contract to the users, and everyone goes through lots of work to avoid changing anything unless absolutely necessary. A very large component of the concern here comes from that mindset. If this goes out, and there's a fundamental problem with it, this community doesn't even have a good process to fix that until the next major release, around a year later. In general, there is no such thing as an upgrade to a stable version that includes a serious behavior change. Having that happen is considered a major breakdown in the process, and there's only been a very small number of such situations in the past, when some just impossible to work around bug was introduced. Instead, just the absolute minimum of changes needed to fix bugs are applied to the stable, shipped versions. If a version ships with a broken enough behavior, it's quite possible that fixing it cannot be done in a way that can be distributed and applied via the standard minor version upgrade procedure used at all. The idea here is not if it's not quite right, development is fast so it will get sorted out eventually. Instead it's if it's not believed to be free of non-trivial bugs, don't commit it at all. SEPostgres has lived in this state for a while now. And it's not known bugs that are the problem, it's that almost all of the Postgres community can't even accurately gauge whether there are bugs or not in the code/design, which is really uncomfortable given how things work here. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] Installing PL/pgSQL by default
Tom Lane wrote:
Andrew Dunstan and...@dunslane.net writes:
Before we go too far with this, I'd like to know how we will handle the
problems outlined here:
http://archives.postgresql.org/pgsql-hackers/2008-02/msg00916.php
Hm, I think that's only a problem if we define it to be a problem,
and I'm not sure it's necessary to do so. Currently, access to PL
languages is controlled by superusers. You are suggesting that if
plpgsql is installed by default, then access to it should be controlled
by non-superuser DB owners instead. Why do we have to move the
goalposts in that direction? It's not like we expect that DB owners
should control access to other built-in facilities, like int8 or
pg_stat_activity for example. The argument against having plpgsql
always available is essentially one of security risks, and I would
expect that most installations think that security risks are to be
managed by superusers.
I installed PL/pgSQL by default via initdb with the attached patch. The
only problem is that pg_dump still dumps out the language creation:
CREATE PROCEDURAL LANGUAGE plpgsql;
ALTER PROCEDURAL LANGUAGE plpgsql OWNER TO postgres;
What is odd is that I used the same process that initdb uses to create
other objects. Does anyone know why this is happening?
--
Bruce Momjian br...@momjian.ushttp://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: doc/src/sgml/installation.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/installation.sgml,v
retrieving revision 1.331
diff -c -c -r1.331 installation.sgml
*** doc/src/sgml/installation.sgml 8 Dec 2009 20:08:30 - 1.331
--- doc/src/sgml/installation.sgml 9 Dec 2009 02:02:33 -
***
*** 2262,2275
is commandcreatelang/command failing with unusual errors.
For example, running as the owner of the PostgreSQL installation:
screen
! -bash-3.00$ createlang plpgsql template1
! createlang: language installation failed: ERROR: could not load library /opt/dbs/pgsql748/lib/plpgsql.so: A memory address is not in the address space for the process.
/screen
Running as a non-owner in the group posessing the PostgreSQL
installation:
screen
! -bash-3.00$ createlang plpgsql template1
! createlang: language installation failed: ERROR: could not load library /opt/dbs/pgsql748/lib/plpgsql.so: Bad address
/screen
Another example is out of memory errors in the PostgreSQL server
logs, with every memory allocation near or greater than 256 MB
--- 2262,2275
is commandcreatelang/command failing with unusual errors.
For example, running as the owner of the PostgreSQL installation:
screen
! -bash-3.00$ createlang plperl template1
! createlang: language installation failed: ERROR: could not load library /opt/dbs/pgsql748/lib/plperl.so: A memory address is not in the address space for the process.
/screen
Running as a non-owner in the group posessing the PostgreSQL
installation:
screen
! -bash-3.00$ createlang plperl template1
! createlang: language installation failed: ERROR: could not load library /opt/dbs/pgsql748/lib/plperl.so: Bad address
/screen
Another example is out of memory errors in the PostgreSQL server
logs, with every memory allocation near or greater than 256 MB
Index: src/bin/initdb/initdb.c
===
RCS file: /cvsroot/pgsql/src/bin/initdb/initdb.c,v
retrieving revision 1.177
diff -c -c -r1.177 initdb.c
*** src/bin/initdb/initdb.c 14 Nov 2009 15:39:36 - 1.177
--- src/bin/initdb/initdb.c 9 Dec 2009 02:02:36 -
***
*** 176,181
--- 176,182
static void setup_privileges(void);
static void set_info_version(void);
static void setup_schema(void);
+ static void load_plpgsql(void);
static void vacuum_db(void);
static void make_template0(void);
static void make_postgres(void);
***
*** 1893,1898
--- 1894,1924
}
/*
+ * load PL/pgsql server-side language
+ */
+ static void
+ load_plpgsql(void)
+ {
+ PG_CMD_DECL;
+
+ fputs(_(loading PL/pgSQL server-side language ... ), stdout);
+ fflush(stdout);
+
+ snprintf(cmd, sizeof(cmd),
+ \%s\ %s template1 %s,
+ backend_exec, backend_options,
+ DEVNULL);
+
+ PG_CMD_OPEN;
+
+ PG_CMD_PUTS(CREATE LANGUAGE plpgsql;\n);
+
+ PG_CMD_CLOSE;
+
+ check_ok();
+ }
+
+ /*
* clean everything up in template1
*/
static void
***
*** 3125,3130
--- 3151,3158
setup_schema();
+ load_plpgsql();
+
vacuum_db();
make_template0();
Index: src/test/regress/GNUmakefile
===
RCS file: /cvsroot/pgsql/src/test/regress/GNUmakefile,v
retrieving revision 1.79
diff -c -c -r1.79 GNUmakefile
***
Re: [HACKERS] bug: fuzzystrmatch levenshtein is wrong
On Tue, Dec 8, 2009 at 9:08 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: patch attached. I cannot get this patch to apply for anything. All 4 hunks fail, both on HEAD and on the the pre-8.4-pgindent version of fuzzystrmatch.c. Either I'm doing something wrong here, or there's something wrong with this patch file. The author converted tabs to spaces --- there is not a single tab in the diff file. Ah. I knew there had to be a reason. I'm attaching my version of this patch. Barring objections, I am going to apply this to HEAD and backpatch to 8.4, where this feature (and the associated bug) were introduced. ...Robert levenshtein.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Takahiro Itagaki wrote: Can I ask the final decision whether the YAML formatter should be applied or rejected? As far as I read the discussion, we can apply it because serveral users want it and we don't have a plan to support extensible formatters in the core. The path I thought made sense at this point was to mark the patch ready for a committer, since it sounds like everyone is done with it now, and have another committer besides yourself do a final review as part of that. At this point, I think we've justified the feature and confirmed the feature works. Given the controversy, I think another set of eyes to make sure it's not going to be a maintenance headache moving forward should (as usual) have the final say on whether the code goes in or not, because that's only drawback to it left to committing it I see at this point. To be clear about which version we're talking about: http://archives.postgresql.org/message-id/20091130123456.4a03.52131...@oss.ntt.co.jp is the candidate for commit that includes the cleanup you've already done. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64 [repost]
Magnus, Just in case anybody was wondering, I've added myself as a reviewer of this one for next commitfest - I doubt that's very surprising :-) Others are of course more than welcome to chip in! Greg Smith wrote: There are also a couple of patches that for various reasons have yet to get a first review done. We've been reassigning for those the last couple of days, and I expect that all those will also be looked at by Tuesday as well (except for SE-PostgreSQL/Lite, which we all know is a bit more complicated). This will leave some time for their authors to respond to feedback before we close up here, and of course we still have one more CommitFest left for patches that are returned with feedback to be resubmitted for. Now that Greg is going to close the Nov Commit Festa, I think he is requesting initial reviews for the patches. Did you have a chance to review the Windows x64 patches? -- Tatsuo Ishii SRA OSS, Inc. Japan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64 [repost]
Tatsuo Ishii wrote: Now that Greg is going to close the Nov Commit Festa, I think he is requesting initial reviews for the patches. While Magnus might take a look anyway, in general we'll all be taking a break from review until January 15th, when the final CommitFest for this version starts. That way everyone has some time to work on their development instead of looking at other people's. You should also be aware that if there are any problems, you may discover the patch just gets put to the sidelines until the next version. We're trying to encourage people to submit major patches *before* the final CommitFest, so that there's time to do a round of feedback on them followed by major cleanup before they get committed. Unfortunately, something as big was a new platform port could easily find itself postponed until the next major version--it's not something development is going to wait for if it doesn't get finished up before the CommitFest is over. You're basically at Magnus's mercy though, if he wants to dedicate enough time to get it done that certainly can happen. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Greg Smith wrote: Takahiro Itagaki wrote: Can I ask the final decision whether the YAML formatter should be applied or rejected? As far as I read the discussion, we can apply it because serveral users want it and we don't have a plan to support extensible formatters in the core. The path I thought made sense at this point was to mark the patch ready for a committer, since it sounds like everyone is done with it now, and have another committer besides yourself do a final review as part of that. That brings us back to where this conversation started ;-) I'll pick it up. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ProcessUtility_hook
It looks like the last round of issues on this patch only came from Tom's concerns, so I'm not sure if anyone but Tom (or a similarly picky alternate committer) is going to find anything else in a re-review. It looks to me like all the issues were sorted out anyway. Euler, you're off the hook for this one; it looks ready for committer to me. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64 [repost]
Tatsuo Ishii wrote: Now that Greg is going to close the Nov Commit Festa, I think he is requesting initial reviews for the patches. While Magnus might take a look anyway, in general we'll all be taking a break from review until January 15th, when the final CommitFest for this version starts. That way everyone has some time to work on their development instead of looking at other people's. You should also be aware that if there are any problems, you may discover the patch just gets put to the sidelines until the next version. We're trying to encourage people to submit major patches *before* the final CommitFest, so that there's time to do a round of feedback on them followed by major cleanup before they get committed. Unfortunately, something as big was a new platform port could easily find itself postponed until the next major version--it's not something development is going to wait for if it doesn't get finished up before the CommitFest is over. You're basically at Magnus's mercy though, if he wants to dedicate enough time to get it done that certainly can happen. Ok. Your suggestion is very helpfull. In general Tsutomu will wait for feedbacks come in, probably until Jan 15th. BTW, is there anyone who wishes the patches get in 8.5? Apparently Tstutomu, Magnus and I are counted in the group:-) But I'd like to know how other people are interested in the patches. -- Tatsuo Ishii SRA OSS, Inc. Japan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EXPLAIN BUFFERS
Euler Taveira de Oliveira wrote: If there is no more objections, I'll flag the patch 'ready for committer' I just executed that. Note that there are two bits of subjective tweaking possible to do with this one when it's committed: slimming down the width of the display, and Euler's suggestion's for rewording. I linked to both of those messages in the CF app, labeled as notes the committer might want to consider, but that the patch hasn't been updated to include yet. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64 [repost]
On Tue, Dec 8, 2009 at 10:30 PM, Tatsuo Ishii is...@postgresql.org wrote: Tatsuo Ishii wrote: Now that Greg is going to close the Nov Commit Festa, I think he is requesting initial reviews for the patches. While Magnus might take a look anyway, in general we'll all be taking a break from review until January 15th, when the final CommitFest for this version starts. That way everyone has some time to work on their development instead of looking at other people's. You should also be aware that if there are any problems, you may discover the patch just gets put to the sidelines until the next version. We're trying to encourage people to submit major patches *before* the final CommitFest, so that there's time to do a round of feedback on them followed by major cleanup before they get committed. Unfortunately, something as big was a new platform port could easily find itself postponed until the next major version--it's not something development is going to wait for if it doesn't get finished up before the CommitFest is over. You're basically at Magnus's mercy though, if he wants to dedicate enough time to get it done that certainly can happen. Ok. Your suggestion is very helpfull. In general Tsutomu will wait for feedbacks come in, probably until Jan 15th. Of course there's also no rule that you couldn't review these sooner - that might help get the ball rolling! ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] pg_ctl init extension
Zdenek Kotala wrote: thanks for your useful comments. I attached new doc patch version. I removed example changes and add link to create database cluster (I hope) everywhere. Please, look on it and let me know if there is still something what should be changed. That looks much better. There's only one bit that sticks out oddly now: + Note: The commandinitdb/command might be invoked by + commandpg_ctl initdb/command and commandinitdb/command cannot be in + default path on a productnamePostgreSQL/productname installations. What is that supposed to mean exactly? -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Greg Smith g...@2ndquadrant.com writes: To be clear about which version we're talking about: http://archives.postgresql.org/message-id/20091130123456.4a03.52131...@oss.ntt.co.jp is the candidate for commit that includes the cleanup you've already done. I suppose this is subject to the same auto_explain problem already noticed for JSON, but I would suggest that we commit this first and then fix both together, rather than create merge issues. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64 [repost]
Ok. Your suggestion is very helpfull. In general Tsutomu will wait for feedbacks come in, probably until Jan 15th. Of course there's also no rule that you couldn't review these sooner - that might help get the ball rolling! Of course I did before he publishes the patches. (I and he are working for the same company). However I'm not a Windows programmer by no means. So my suggestion was mainly for designs... -- Tatsuo Ishii SRA OSS, Inc. Japan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ProcessUtility_hook
On Tue, Dec 8, 2009 at 10:12 PM, Greg Smith g...@2ndquadrant.com wrote: It looks like the last round of issues on this patch only came from Tom's concerns, so I'm not sure if anyone but Tom (or a similarly picky alternate committer) is going to find anything else in a re-review. It looks to me like all the issues were sorted out anyway. Euler, you're off the hook for this one; it looks ready for committer to me. Since Itagaki Takahiro is now a committer, I sort of assumed he would be committing his own patches. I am not really sure what the etiquette is in this area, but there seems to be an implication here someone else will be committing this, which isn't necessarily my understanding of how it works. Certainly, unless someone has a contrary opinion, I think he can go ahead if he wishes. On the other hand, if it's helpful, I'm more than happy to pick up this one and/or EXPLAIN BUFFERS for final review and commit. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64 [repost]
On Tue, Dec 8, 2009 at 10:48 PM, Tatsuo Ishii is...@postgresql.org wrote: Ok. Your suggestion is very helpfull. In general Tsutomu will wait for feedbacks come in, probably until Jan 15th. Of course there's also no rule that you couldn't review these sooner - that might help get the ball rolling! Of course I did before he publishes the patches. (I and he are working for the same company). However I'm not a Windows programmer by no means. So my suggestion was mainly for designs... Ah, OK. Makes sense. Sorry, I have not seen too much of your code so I don't know what areas you specialize in... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] Installing PL/pgSQL by default
Bruce Momjian br...@momjian.us writes: I installed PL/pgSQL by default via initdb with the attached patch. The only problem is that pg_dump still dumps out the language creation: CREATE PROCEDURAL LANGUAGE plpgsql; ALTER PROCEDURAL LANGUAGE plpgsql OWNER TO postgres; What is odd is that I used the same process that initdb uses to create other objects. Does anyone know why this is happening? I think pg_dump pays attention to what schema the objects are in, and that's most likely creating them in PUBLIC. Try adding set search_path = pg_catalog. It's not impossible that we'll have to tweak pg_dump a bit; it's never had to deal with languages that shouldn't be dumped ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ProcessUtility_hook
Robert Haas wrote: Since Itagaki Takahiro is now a committer, I sort of assumed he would be committing his own patches. Maybe, but I wasn't going to be the one to suggest that Tom get cut out of the loop after he raised a list of issues with the patch already. I think the situation for EXPLAIN BUFFERS is much simpler, given that the last round of feedback was only quibbling over the line formatting and docs. What I think is a reasonable general guideline is to route submitted patches back to their author to commit only when the patch has been recently free of code issues during its review. If we've already had another committer chime in with concerns, they should probably confirm themselves that the updated version is suitable to commit, and do so instead of the author. That just seems like a reasonable risk-reduction workflow approach to me, similar to how the sign-off practice works on some other projects. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
David P. Quigley wrote: On Tue, 2009-12-08 at 15:26 -0500, Robert Haas wrote: [snip...] I can say from experience that this project is very skeptical of frameworks that aren't accompanied by at least one, and preferably multiple, working implementations. So there is a bit of a chicken and egg problem here. What can sometimes work is to say, look, here's a place where I can put a hook and later I will do something complex with it but here are a couple of relatively simple but useful examples to get started. The examples form the justification for the commit because they are independently useful, but they are much simpler than what the framework may eventually end up being used for. I don't believe that this approach is feasible for SE-PostgreSQL. A simple version of label security is still going to be very complicated, and there are probably even fewer customers for such a thing than there are for SE-PostgreSQL. Ok if it is a chicken and egg problem then we need to find a smaller egg. I agree that having a huge patch set isn't ideal which is why I understand the desire to have KaiGai cut back his patches. However the patches he is putting forward from what I've gathered have no row based labeling and no MAC enforcement. I can understand if you want to cut back the object model so you're not mediating every object in the system but cutting those two features make it unusable by the customers we have who want to use it. The coverage of access controls are tradeoff between amount of the changeset and valuable functionalities, in generally. Ultimately it is a role of developer to decide what is unseparatable core feture and what is separatable secondary one. It is my decision to restrict types of database objects to reduce burden of reviewers, so the latest one apply SELinux specific access controls on only databases, schemas, tables and columns. Even if we have a framework to host label-based MAC, its coverage shall be unchanged, because comprehensive support at once obviously makes harder to review. So with regard to confidence in the design I think that part of the reason for the skepticism in the fact that it was such a large code drop. Even the separated parts were very large. Definitely true. I think if the framework patches are broken up more logically and in a way that is easier to discuss then that might help the community get a grasp on what it is trying to accomplish. Maybe, maybe not. Nobody's going to commit anything unless it's a complete feature. It can be a cut-down feature, but it has to be of independent use. If there are parts that can be peeled off of SE-PostgreSQL and committed independently to some good benefit, then I think that's a great idea. But if it's just a separation of the patch for clarity, I don't think there's much value in that. I don't think it would be just for clarity. I know that not all open source communities are the same so I'll try to leave the anecdotal evidence light. When submit patches for the Linux Kernel we take a single feature and structure them as self contained logical changes. Separating the changes logically doesn't just improve clarity but also makes it easy to cherry pick features that can be useful on their own. Just because the framework and the SEPostgreSQL features would be two patch sets doesn't mean they aren't being developed in parallel. It looks to me that we're in the same boat we were in with SELinux. We had the feature and proposed it and someone brought up well what about other security models. So time was spent developing a framework that everyone could use and as that was being done the SELinux patches were modified to use this new framework. They were two separate features developed in tandem. If we do it right the SEPostgreSQL code will be isolated from the framework and we can spend time putting the framework in and just plug in the specific security module when time comes. The X-ACE work went in to X before the corresponding SELinux Flask module for it and I don't believe LSM and SELinux went into the same merge window as well. It is necessary to deploy the framework and SELinux support at same time or the framework earlier than SELinux? We can organize a common framework when the second security module will come in. From our experience, it may need similar security hooks on same or similar strategic points. In the development history, the code size and complexity had been a matter for inclusion. I also think the idea to invoke security modules via mac framework is right approach, but it increases the initial code size compared to the current approach in spite of even user visible whorth. I have proofread earlier versions of the docs and found them inadequate. There were language issues, but the bigger problem was that they were both too specific and yet not sufficiently detailed. For example, they'd say install X package on Red Hat. Well, what if
Re: [HACKERS] EXPLAIN BUFFERS
Greg Smith g...@2ndquadrant.com wrote: I just executed that. Note that there are two bits of subjective tweaking possible to do with this one when it's committed: slimming down the width of the display, and Euler's suggestion's for rewording. I linked to both of those messages in the CF app, labeled as notes the committer might want to consider, but that the patch hasn't been updated to include yet. Sure, I should have merge all of the comments. Patch attached. - Updated documentation as Euler's suggestion, but I replaced the It of the second last sentence to This parameter. - Updated the output format as follows. I think this format is the most similar to existing lines. (actual by ANALYZE and Filter:). Note that the patch also removes buffer counters from log_statement_stats, but we only have brief descriptions about the options. Our documentation say nothing about buffer counters, so I didn't modify those lines in sgml. http://developer.postgresql.org/pgdocs/postgres/runtime-config-statistics.html#RUNTIME-CONFIG-STATISTICS-MONITOR IMHO, we could remove those options completely because we can use EXPLAIN BUFFERS and DTrace probes instead of them. =# EXPLAIN (BUFFERS, ANALYZE) SELECT * FROM pgbench_accounts a, pgbench_branches b WHERE a.bid = b.bid AND abalance 0 ORDER BY abalance; QUERY PLAN -- Sort (cost=2891.03..2891.04 rows=1 width=461) (actual time=22.494..22.494 rows=0 loops=1) Sort Key: a.abalance Sort Method: quicksort Memory: 25kB Blocks: (shared hit=96 read=1544 written=0) (local hit=0 read=0 written=0) (temp read=0 written=0) - Nested Loop (cost=0.00..2891.02 rows=1 width=461) (actual time=22.488..22.488 rows=0 loops=1) Join Filter: (a.bid = b.bid) Blocks: (shared hit=96 read=1544 written=0) (local hit=0 read=0 written=0) (temp read=0 written=0) - Seq Scan on pgbench_accounts a (cost=0.00..2890.00 rows=1 width=97) (actual time=22.486..22.486 rows=0 loops=1) Filter: (abalance 0) Blocks: (shared hit=96 read=1544 written=0) (local hit=0 read=0 written=0) (temp read=0 written=0) - Seq Scan on pgbench_branches b (cost=0.00..1.01 rows=1 width=364) (never executed) Blocks: (shared hit=0 read=0 written=0) (local hit=0 read=0 written=0) (temp read=0 written=0) Total runtime: 22.546 ms (13 rows) Regards, --- Takahiro Itagaki NTT Open Source Software Center explain_buffers_20091209.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench: new feature allowing to launch shell commands
Michael Paquier michael.paqu...@gmail.com wrote: Please find attached the latest version of the patch, with the threading bug corrected and the documentation updated as well. Why do you use BUFFER_SIZE-1 for snprintf? snprintf(commandShell, SHELL_COMMAND_SIZE-1, ...) Trailing nulls are also included in the length, so snprintf(commandShell, SHELL_COMMAND_SIZE, ...) would be ok. (removed -1) Other parts look fine, except an empty tag replaceable/ in the documentation. Is it a typo? Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
2009/12/9 Bruce Momjian br...@momjian.us: I frankly think the patch should be thought of as the SE-Linux-specific directory files, which KaiGai can maintain, and the other parts, which I think I can handle. I think that's a horribly bad idea. We have already got a similar issue with ECPG, which clearly stagnates whenever Michael is busy and don't have time to go through the patches. Because it's his code, and nobody else knows how to and/or cares to maintain it. And this is just a single piece of the frontend that doesn't affect anything else. If you want to do something similar for sepg, then sepg needs to be turned into a full plugin system, where the plugin is a completely separate thing. In which case the plugin can be developed separately, for example on pgfoundry (and be considered to merge later, if we want, but not necessarily ever since it has a narrow user base). I haven't looked at the patch properly for quite a while, but I imagine turning it into such a plugin is not feasible. Because if it is, why haven't this been done already? :) But if it is, perhaps that is something we should consider, since it lessens the maintenance burden into just the API (which is still a huge burden compared to many of our APIs, but it is a lot less than what the patch is now) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Stephen Frost wrote: * Robert Haas (robertmh...@gmail.com) wrote: One of the major and fundamental stumbling blocks we've run into is that every solution we've looked at so far seems to involve adding SE-Linux-specific checks in many places in the code. I've really got to take exception to this. I've only been following along and not really participating because, to be honest, I'm frustrated with how this has gone down. In the end there were at least two patches, in my view, that *didn't* involve adding SE-Linux-specific checks everywhere. The patch that I reviewed that got thrown out by Tom, and the original PGACE framework. Both of those added alot of *hooks*, because they were necessary, but nothing made those hooks particularly SELinux-specifc. We're hearing alot about things being SELinux-specific from people who also profess to not know SELinux. Indeed, as I recall, the patch I reviewed was primairly trying to just addess pulling out the hooks necessary for the existing PostgreSQL security model. Very little of it was SE-Linux specific *anything*. I contend that while the specific hooks which would be added *in places that don't already have checks* (in most places, the hook was added to replace an existing check) are hooks that then make sense for SELinux, they would also make sense for other frameworks. They may also not be a complete list, but once the *framework* is in place, adding new hooks (presuming another model would like a hook somewhere that SELinux and the existing PG security framework don't) should not require some kind of forklift upgrade. Basically, I don't think it is a bad idea to provide a framework to host multiple label-based security mechanisms, because they have same origin so they have similar features, such as security label and policy. However, we have a tradeoff betweem an ideal image (multiple enhanced security stuff can be launched via common hooks) and amount of changeset. The latest SE-PgSQL/Lite patch support only four kind of database objects (databases, schemas, tables and columns), so it puts security hooks on the strategic points corresponding to these objects, such as createdb(). If we simply replace the invocations by the common hooks, coverage of the framework will be restricted to these four objects. Is it possible to call them as a framework? If it will be comprehensive from the first, we shall face a nightmare again. I think we can replace the sepgsql_*() invocations by the framework when the second enhancement (such as Solaris-TX?) will come in. Note that label-based security has historically same origin, so we can expect much smaller arrangement than integration DAC and MAC. Consequently there are checks spread throughout the code, which definitely complicates both validation and maintenance. One question I have is - are the places where those checks are placed specific to SE-Linux, or would they be applicable to any sort of label-based MAC? The patch which I worked with Kaigai on was specifically geared to first try to get a patch which addressed the existing PG security model in a modular way, to allow additional hooks to be added later in places which needed them, and to provide the information for other models to make a decision about the permission. I don't feel it was particularly SE-Linux specific at all, but rather a first step towards being able to support something beyond the model we have today (anything..). Right, the default PG model has been already comprehensive, and has a different origin from label-based securities, so we needed massive changeset to organize both of DAC (the default PG model) and MAC into a common security framework. Thanks, -- OSS Platform Development Division, NEC KaiGai Kohei kai...@ak.jp.nec.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
Fujii Masao wrote: On Tue, Dec 8, 2009 at 11:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: If that's deemed not good enough, we could modify just internal_flush() so that it uses secure_poll to wait for the possibility to either read or write, instead of blocking for just write. Whenever there's incoming data, read them into PqRecvBuffer for later processing, which keeps the OS input buffer from filling up. If PqRecvBuffer fills up, it can be extended, or we can start dropping old XLogRecPtr messages from it. Extending PqRecvBuffer seems better because XLogRecPtr message has some types (i.e., we cannot just drop old message without parsing all messages in the buffer). True. Another idea I had was to introduce a callback that backend libpq can call when the buffer fills. Walsender would set the callback to ProcessStreamMsgs(). But if everyone is happy with just relying on the OS buffer to not fill up, let's just drop it. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XLogInsert
On Sun, Sep 13, 2009 at 10:42 PM, Jeff Janes jeff.ja...@gmail.com wrote: I've attached a patch which removes the iteration over the blocks to be backed-up from the critical section of XLogInsert. Now those blocks are only looped over in one piece of code which both computes the CRC and builds the linked list, rather than having parallel loops. ok, i started to review this one... what the patch is doing seems very obvious and doesn't break the original logic AFAIUI (i could be wrong, this seems like chinese to me ;) i made some tests archiving wal and recovering... crashing the server and restarting... every test was running the regression tests, don't know what else to do, ideas? i haven't made any performance tests but it should gain something :), maybe someone can make those tests? if not, i will mark the patch as ready for committer some time in the next hours... -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
