[HACKERS] PROPOSAL of xmlvalidate

2010-11-28 Thread Tomáš Pospíšil
Hi, I am working on patch adding xmlvalidate() functionality. LibXML 2.7.7 improved DTD, XSD, Relax-NG validation, so using that. I have idea of creating system table for holding DTDs, XSDs, Relax-NGs (similar as on ORACLE). Is that good idea? If so, how to implement that table? pg_attribute

Re: [HACKERS] ALTER OBJECT any_name SET SCHEMA name

2010-11-28 Thread Robert Haas
On Sat, Nov 27, 2010 at 2:17 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Thanks! The _oid variants will have to re-appear in the alter extension set schema patch, which is the last of the series. Meanwhile, I will have to merge head with the current extension patch (already overdue for

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Thu, Nov 4, 2010 at 6:35 AM, Stephen Frost sfr...@snowman.net wrote: * Jan Urbański (wulc...@wulczer.org) wrote: On 04/11/10 14:09, Robert Haas wrote: Hmm, I wonder how useful this is given that restriction. As

Re: [HACKERS] PROPOSAL of xmlvalidate

2010-11-28 Thread Andrew Dunstan
On 11/28/2010 05:33 AM, Tomáš Pospíšil wrote: Hi, I am working on patch adding xmlvalidate() functionality. LibXML 2.7.7 improved DTD, XSD, Relax-NG validation, so using that. I have idea of creating system table for holding DTDs, XSDs, Relax-NGs (similar as on ORACLE). Is that good idea?

Re: [HACKERS] PLy_malloc and plperl mallocs

2010-11-28 Thread Jan Urbański
On 28/11/10 05:23, Andrew Dunstan wrote: On 11/27/2010 10:28 PM, Tom Lane wrote: =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?=wulc...@wulczer.org writes: I noticed that PL/Python uses a simple wrapper around malloc that does ereport(FATAL) if malloc returns NULL. I find it a bit harsh, don't we

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Martijn van Oosterhout
On Sat, Nov 27, 2010 at 10:02:33PM -0500, Tom Lane wrote: In recent discussions of the plan-tree representation for KNNGIST index scans, there seemed to be agreement that it'd be a good idea to explicitly represent the expected sort ordering of the output. While thinking about how best to do

[HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Marti Raudsepp
Hi list, Often enough when developing PostgreSQL views and functions, I have pasted the CREATE OR REPLACE commands into the wrong window/shell and ran them there without realizing that I'm creating a function in the wrong database, instead of replacing. Currently psql does not provide any

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Josh Kupershmidt
On Fri, Nov 26, 2010 at 7:11 PM, Robert Haas robertmh...@gmail.com wrote: I'm not totally convinced that this is the correct behavior.  It seems a bit surprising that UPDATE privilege on a single column is enough to lock out all SELECT activity from the table.  It's actually a bit surprising

Re: [HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Tom Lane
Marti Raudsepp ma...@juffo.org writes: This patch returns command tag CREATE X or REPLACE X for LANGAUGE/VIEW/RULE/FUNCTION. This is done by passing completionTag to from ProcessUtility to more functions, and adding a 'bool *didUpdate' argument to some lower-level functions. I'm not sure if

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Simon Riggs
On Fri, 2010-11-26 at 19:11 -0500, Robert Haas wrote: 2010/11/25 KaiGai Kohei kai...@ak.jp.nec.com: (2010/10/16 4:49), Josh Kupershmidt wrote: [Moving to -hackers] On Fri, Oct 15, 2010 at 3:43 AM, Simon Riggssi...@2ndquadrant.com wrote: On Mon, 2010-10-11 at 09:41 -0400, Josh

Re: [HACKERS] ECPG question about PREPARE and EXECUTE

2010-11-28 Thread Michael Meskes
On Wed, Nov 10, 2010 at 11:44:52AM +0100, Boszormenyi Zoltan wrote: a question came to us in the form of a code example, which I shortened. Say, we have this structure: ... Any comment on why it isn't done? Missing feature. Originally the pure text based statement copying wasn't able to cope

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sat, Nov 27, 2010 at 11:18 PM, Bruce Momjian br...@momjian.us wrote: Not sure that information moves us forward.  If the postmaster cleared the memory, we would have COW in the child and probably be even slower. Well, we can determine the answers

Re: [HACKERS] PLy_malloc and plperl mallocs

2010-11-28 Thread Tom Lane
=?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= wulc...@wulczer.org writes: I'll do that for PL/Python for now. While on the topic of needless FATAL errors, if you try to create a Python 3 function in a session that already loaded Python 2, you get a FATAL error with an errhint of Start a new session to use

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Simon Riggs
On Sat, 2010-11-27 at 14:27 -0500, Tom Lane wrote: This is discouraging; it certainly doesn't make me want to expend the effort to develop a production patch. Perhaps. Why do this only for shared memory? Surely the majority of memory accesses are to private memory, so being able to allocate

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sat, 2010-11-27 at 14:27 -0500, Tom Lane wrote: This is discouraging; it certainly doesn't make me want to expend the effort to develop a production patch. Perhaps. Why do this only for shared memory? There's no exposed API for causing a

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Simon Riggs
On Sun, 2010-11-28 at 12:04 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Sat, 2010-11-27 at 14:27 -0500, Tom Lane wrote: This is discouraging; it certainly doesn't make me want to expend the effort to develop a production patch. Perhaps. Why do this only for

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sun, 2010-11-28 at 12:04 -0500, Tom Lane wrote: There's no exposed API for causing a process's regular memory to become hugepages. We could make all the palloc stuff into shared memory also (private shared memory that is). We're not likely to run

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Nov 27, 2010 at 11:18 PM, Bruce Momjian br...@momjian.us wrote: Not sure that information moves us forward.  If the postmaster cleared the memory, we would have COW in the

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Martijn van Oosterhout
On Sun, Nov 28, 2010 at 02:32:04PM -0500, Tom Lane wrote: Sure, but 4MB of memory is enough to require 1000 TLB entries, which is more than enough to blow the TLB even on a Nehalem. That can't possibly be right. I'm sure the chip designers have heard of programs using more than 4MB.

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: If you look closely at what we're doing with sort operators (get_ordering_op_properties pretty much encapsulates this), it turns out that a sort operator is shorthand for three pieces of information: 1. btree opfamily OID 2. specific input datatype for the

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Bruce Momjian
Robert Haas wrote: On Sat, Nov 27, 2010 at 11:18 PM, Bruce Momjian br...@momjian.us wrote: Not sure that information moves us forward. ?If the postmaster cleared the memory, we would have COW in the child and probably be even slower. Well, we can determine the answers to these questions

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: The more general issue here is what to do about our high backend startup costs. Beyond trying to recycle backends for new connections, as I've previous proposed and with all the problems it entails, the only thing that looks promising here is to try

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Tom Lane
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: So to fix these problems we'd need to replace sort operator OIDs in SortGroupClause and plan nodes with those three items. Obviously, this would be slightly bulkier, but the extra cost added to copying parse

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 11:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-11-26 at 19:11 -0500, Robert Haas wrote: 2010/11/25 KaiGai Kohei kai...@ak.jp.nec.com: (2010/10/16 4:49), Josh Kupershmidt wrote: [Moving to -hackers] On Fri, Oct 15, 2010 at 3:43 AM, Simon

[HACKERS] SSI using rw-conflict lists

2010-11-28 Thread Kevin Grittner
Well, it's been a productive holiday weekend. I've completed the switch of the SSI implementation from one conflict pointer in and one conflict pointer out per transaction to a list of conflicts between transactions. This eliminated all false positives in my dtester suite. The only test which

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Tom Lane
I wrote: (For some extra amusement, trace through where build_index_pathkeys' data comes from...) While I don't propose to implement right away the whole SortGroupClause and plan tree modification sketched above, I did look into fixing build_index_pathkeys so that it doesn't uselessly convert

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Kenneth Marshall
On Sat, Nov 27, 2010 at 02:27:12PM -0500, Tom Lane wrote: We've gotten a few inquiries about whether Postgres can use huge pages under Linux. In principle that should be more efficient for large shmem regions, since fewer TLB entries are needed to support the address space. I spent a bit of

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Jeff Janes
On Sun, Nov 28, 2010 at 5:38 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes jeff.ja...@gmail.com wrote: I haven' t thought of a way to test this, so I guess I'll just ask. If the attacking client just waits a few milliseconds for a response and then

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 3:53 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: The more general issue here is what to do about our high backend startup costs.  Beyond trying to recycle backends for new connections, as I've previous proposed and with all the

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: After our recent conversation about KNNGIST, it occurred to me to wonder whether there's really any point in pretending that a user can usefully add an AM, both due to hard-wired planner knowledge and due to lack of any sort of extensible XLOG

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread KaiGai Kohei
(2010/11/27 9:11), Robert Haas wrote: 2010/11/25 KaiGai Koheikai...@ak.jp.nec.com: (2010/10/16 4:49), Josh Kupershmidt wrote: [Moving to -hackers] On Fri, Oct 15, 2010 at 3:43 AM, Simon Riggssi...@2ndquadrant.com wrote: On Mon, 2010-10-11 at 09:41 -0400, Josh Kupershmidt wrote: On Thu,

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 5:41 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sun, Nov 28, 2010 at 5:38 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes jeff.ja...@gmail.com wrote: I haven' t thought of a way to test this, so I guess I'll just ask. If the

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 6:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: After our recent conversation about KNNGIST, it occurred to me to wonder whether there's really any point in pretending that a user can usefully add an AM, both due to hard-wired

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Jeff Janes
On Sun, Nov 28, 2010 at 3:57 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Nov 28, 2010 at 5:41 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sun, Nov 28, 2010 at 5:38 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes jeff.ja...@gmail.com wrote: I

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:10 PM, Jeff Janes jeff.ja...@gmail.com wrote: Oh, I wasn't complaining.  I think that having max_connections be charged for the duration even if the socket is dropped is the only reasonable thing to do, and wanted to verify that it did happen. Otherwise the module

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Kenneth Marshall k...@rice.edu writes: On Sat, Nov 27, 2010 at 02:27:12PM -0500, Tom Lane wrote: ... A bigger problem is that the shmem request size must be a multiple of the system's hugepage size, which is *not* a constant even though the test patch just uses 2MB as the assumed value. For a

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: One possible way to get a real speedup here would be to look for ways to trim the number of catcaches. BTW, it's not going to help to remove catcaches that have a small initial size, as the pg_am cache certainly does. If the bucket zeroing cost is

Re: [HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 11:29 AM, Tom Lane t...@sss.pgh.pa.us wrote: Marti Raudsepp ma...@juffo.org writes: This patch returns command tag CREATE X or REPLACE X for LANGAUGE/VIEW/RULE/FUNCTION. This is done by passing completionTag to from ProcessUtility to more functions, and adding a 'bool

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
BTW, this might be premature to mention pending some tests about mapping versus zeroing overhead, but it strikes me that there's more than one way to skin a cat. I still think the idea of statically allocated space sucks. But what if we rearranged things so that palloc0 doesn't consist of

Re: [HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I think more expessive command tags are in general a good thing. The idea that this particular change would be useful primarily for humans examining the psql output seems a bit weak to me, but I can easily see it being useful for programs. Right now

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Tom Lane
I wrote: Now, this loss of flexibility doesn't particularly bother me, because I know of no existing or contemplated btree-substitute access methods. If one did appear on the horizon, there are a couple of ways we could fix the problem, the cleanest being to let a non-btree opfamily declare

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Greg Stark
On Mon, Nov 29, 2010 at 12:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: I would expect that you can just iterate through the size possibilities pretty quickly and just use the first one that works -- no /proc groveling. It's not really that easy, because (at least on the kernel version I

[HACKERS] Feature request: INSERT .... ON DUPLICATE UPDATE ....

2010-11-28 Thread Yourfriend
Dear all, In our company, we use both PostgreSQL and MySQL, our developers include me think that the INSERT ... ON DUPLICATE UPDATE clause is a much more user friendly function,so, would you please add this liked function in PostgreSQL, I know we can write PROCEDURE or RULE to solve this

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Fujii Masao
On Sat, Nov 27, 2010 at 9:25 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 25, 2010 at 1:18 AM, KaiGai Kohei kai...@ak.jp.nec.com wrote: The attached patch is revised version. - Logging part within auth_delay was removed. This module now focuses on  injection of a few seconds delay

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Mon, Nov 29, 2010 at 12:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: Really you do want to scrape the value. Couldn't we just round the shared memory allocation down to a multiple of 4MB? That would handle all older architectures where the size is 2MB or

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Greg Stark
On Mon, Nov 29, 2010 at 12:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: The most portable way to do that would be to use calloc insted of malloc, and hope that libc is smart enough to provide freshly-mapped space. It would be good to look and see whether glibc actually does so, of course.  If not

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Mon, Nov 29, 2010 at 12:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: Another question that would be worth asking here is whether the hand-baked MemSet macro still outruns memset on modern architectures. I think it's been quite a few years since that was last

Re: [HACKERS] Patch to add a primary key using an existing index

2010-11-28 Thread Itagaki Takahiro
On Fri, Nov 26, 2010 at 05:58, Steve Singer ssin...@ca.afilias.info wrote: The attached version of the patch gets your regression tests to pass. I'm going to mark this as ready for a committer. I think we need more discussions about the syntax: ALTER TABLE table_name ADD PRIMARY KEY (...)

Re: [HACKERS] Assertion failure on hot standby

2010-11-28 Thread Simon Riggs
On Fri, 2010-11-26 at 01:11 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: That would mean running GetCurrentTransactionId() inside LockAcquire() if (lockmode = AccessExclusiveLock locktag-locktag_type == LOCKTAG_RELATION !RecoveryInProgress()) (void)

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: One possible way to get a real speedup here would be to look for ways to trim the number of catcaches. BTW, it's not going to help to remove catcaches that have a small initial size,

Re: [HACKERS] Patch to add a primary key using an existing index

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 8:06 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Fri, Nov 26, 2010 at 05:58, Steve Singer ssin...@ca.afilias.info wrote: The attached version of the patch gets your regression tests to pass. I'm going to mark this as ready for a committer. I think we need

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Robert Haas
2010/11/28 KaiGai Kohei kai...@ak.jp.nec.com: I'm not totally convinced that this is the correct behavior.  It seems a bit surprising that UPDATE privilege on a single column is enough to lock out all SELECT activity from the table.  It's actually a bit surprising that even full-table UPDATE

Re: [HACKERS] Feature request: INSERT .... ON DUPLICATE UPDATE ....

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:44 PM, Yourfriend doudou...@gmail.com wrote: In our company,  we use both PostgreSQL and MySQL, our developers include me think that the INSERT ... ON DUPLICATE UPDATE clause is a much more user friendly function,so, would you please add this liked function in

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:45 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks. I found the typo: I only have one? :-) Thanks, fixed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread KaiGai Kohei
(2010/11/29 10:43), Robert Haas wrote: 2010/11/28 KaiGai Koheikai...@ak.jp.nec.com: I'm not totally convinced that this is the correct behavior. It seems a bit surprising that UPDATE privilege on a single column is enough to lock out all SELECT activity from the table. It's actually a bit

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Yeah, very true. What's a bit frustrating about the whole thing is that we spend a lot of time pulling data into the caches that's basically static and never likely to change anywhere, ever. True. I wonder if we could do something like the relcache

[HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-11-28 Thread Noah Misch
I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a vacuum_defer_cleanup_age constituting roughly three hours of the master's transactions. Even so, I kept seeing recovery pause

Re: [HACKERS] pg_execute_from_file review

2010-11-28 Thread Itagaki Takahiro
On Fri, Nov 26, 2010 at 06:24, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Thanks for your review. Please find attached a revised patch where I've changed the internals of the function so that it's split in two and that the opr_sanity check passes, per comments from David Wheeler and Tom

Re: [HACKERS] Patch to add a primary key using an existing index

2010-11-28 Thread David Fetter
On Sun, Nov 28, 2010 at 08:40:08PM -0500, Robert Haas wrote: On Sun, Nov 28, 2010 at 8:06 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Fri, Nov 26, 2010 at 05:58, Steve Singer ssin...@ca.afilias.info wrote: The attached version of the patch gets your regression tests to pass.