Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Michael Meskes
It was proposed a while ago for libpq to support URI syntax for specifying the connection information: ... Now we're going to actually implement this. Do you know that we had this feature (more or less) in libpq for years but it was removed quite a while ago. It should still be there in the

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Florian Weimer
* Alexander Shulgin: This, in my opinion, is very similar to what we would like to achieve with the URI syntax, so the above could also be specified using a URI parameter like this: psql -d postgresql://example.net:5433/mydb How would you specifiy a local port/UNIX domain socket? Would

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Tue, Nov 22, 2011 at 7:25 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Nov 22, 2011 at 11:40 PM, Simon Riggs si...@2ndquadrant.com wrote: I think this is unsafe for shared catalogs. I think so too. Thats why it uses IsMVCCSnapshot() to confirm when it is safe to do so. Ah, you mean

Re: [HACKERS] Permissions checks for range-type support functions

2011-11-23 Thread Robert Haas
On Tue, Nov 22, 2011 at 6:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: 2. The ANALYZE option is flat out dangerous, because it allows any function with the signature f(internal) returns bool to be called as though it's a typanalyze function.  There are a couple of such functions in the catalogs

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 2:00 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 22, 2011 at 7:25 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Nov 22, 2011 at 11:40 PM, Simon Riggs si...@2ndquadrant.com wrote: I think this is unsafe for shared catalogs. I think so too. Thats why

Re: [HACKERS] Inlining comparators as a performance optimisation

2011-11-23 Thread Simon Riggs
On Fri, Nov 18, 2011 at 2:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On Fri, Nov 18, 2011 at 5:20 AM, Robert Haas robertmh...@gmail.com wrote: I think that we should really consider doing with this patch what Tom suggested upthread; namely, looking for

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Wed, Nov 23, 2011 at 9:25 AM, Simon Riggs si...@2ndquadrant.com wrote: Yeah.  This change would have the disadvantage of disabling HOT cleanup for shared catalogs; I'm not sure whether that's a good decision. No, it disables cleanup when being read. They are still VACUUMed normally. Note

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié nov 23 12:15:55 -0300 2011: And it effects shared catalogs only, which are all low traffic anyway. I think low traffic is the key point. I understand that you're not changing the VACUUM behavior, but you are making heap_page_prune_opt() not do

Re: [HACKERS] Permissions checks for range-type support functions

2011-11-23 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié nov 23 11:01:50 -0300 2011: On Tue, Nov 22, 2011 at 6:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: 2. The ANALYZE option is flat out dangerous, because it allows any function with the signature f(internal) returns bool to be called as though it's a

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Wed, Nov 23, 2011 at 10:20 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mié nov 23 12:15:55 -0300 2011: And it effects shared catalogs only, which are all low traffic anyway. I think low traffic is the key point.  I understand that you're not

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié nov 23 12:28:38 -0300 2011: Hmm, I'm not seeing any increase in the number of entries in pg_shdepend when I create either a temporary or permanent table: rhaas=# select sum(1) from pg_shdepend; sum - 2 (1 row) rhaas=# create temp

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Wed, Nov 23, 2011 at 10:35 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: That's because the owner is pinned (i.e. the bootstrap user).  Try with a different user.  I see new rows with both temp and non-temp tables. Oh, wow. I had no idea it worked like that. You learn something new

Re: [HACKERS] Permissions checks for range-type support functions

2011-11-23 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Robert Haas's message of mié nov 23 11:01:50 -0300 2011: It's always seemed mildly insane to me that we don't distinguish between different flavors of internal. That seems like an accident waiting to happen. Well, before we

Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays

2011-11-23 Thread Merlin Moncure
On Tue, Nov 22, 2011 at 6:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Oliver Jowett oli...@opencloud.com writes: On 23 November 2011 10:47, Mikko Tiihonen mikko.tiiho...@nitorcreations.com wrote: Here is a patch that adds a new flag to the protocol that is set when all elements of the array are

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 3:20 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mié nov 23 12:15:55 -0300 2011: And it effects shared catalogs only, which are all low traffic anyway. I think low traffic is the key point.  I understand that you're not

Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays

2011-11-23 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: On Tue, Nov 22, 2011 at 6:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: The only way that anything like this will go in is as part of a protocol version bump, Wire format changes can only be made with a protocol version bump? Is this a new policy? In

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of mié nov 23 13:14:04 -0300 2011: On Wed, Nov 23, 2011 at 3:20 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mié nov 23 12:15:55 -0300 2011: And it effects shared catalogs only, which are all low traffic

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Wed, Nov 23, 2011 at 3:20 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Maybe not pg_database or pg_tablespace and such, but I'm not so sure about pg_shdepend. (Do we record pg_shdepend entries for temp tables?) Normal catalog access does

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Pavan Deolasee
On Wed, Nov 23, 2011 at 9:44 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Nov 23, 2011 at 3:20 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mié nov 23 12:15:55 -0300 2011: And it effects shared catalogs only, which are all low traffic

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 5:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On Wed, Nov 23, 2011 at 3:20 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Maybe not pg_database or pg_tablespace and such, but I'm not so sure about pg_shdepend. (Do we record

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Wed, Nov 23, 2011 at 5:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Normal catalog access does not use HOT and never has. You are mistaken. Normal catalog access against shared catalogs via heap_scan does

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, Nov 22, 2011 at 11:40 PM, Simon Riggs si...@2ndquadrant.com wrote: I think this is unsafe for shared catalogs. I think so too. Thats why it uses IsMVCCSnapshot() to confirm when it is safe to do so. Ah, you mean access to shared catalogs

Re: [HACKERS] plpython SPI cursors

2011-11-23 Thread Jan Urbański
On 20/11/11 19:14, Steve Singer wrote: On 11-10-15 07:28 PM, Jan Urbański wrote: Hi, attached is a patch implementing the usage of SPI cursors in PL/Python. Currently when trying to process a large table in PL/Python you have slurp it all into memory (that's what plpy.execute does). J I

Re: [HACKERS] Inlining comparators as a performance optimisation

2011-11-23 Thread Robert Haas
On Tue, Nov 22, 2011 at 8:09 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: I wonder, is it worth qualifying that the Sort Method was a quicksort (fast path) sort within explain analyze output? This is a rather large improvement, so It might actually be something that people look out for, as

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 6:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: The real question is do we favour HOT cleanup on those small 8 tables, or do we favour HOT cleanup of every other table? No, the real question is why not think a little harder and see if we can come up with a solution that

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Wed, Nov 23, 2011 at 1:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: What I think might make more sense is to keep two variables, RecentGlobalXmin with its current meaning and RecentDatabaseWideXmin which considers only xmins of transactions in the current database. Then HOT cleanup could

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 7:57 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 23, 2011 at 1:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: What I think might make more sense is to keep two variables, RecentGlobalXmin with its current meaning and RecentDatabaseWideXmin which considers only

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Nov 23, 2011 at 1:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: What I think might make more sense is to keep two variables, RecentGlobalXmin with its current meaning and RecentDatabaseWideXmin which considers only xmins of transactions in the

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 8:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Nov 23, 2011 at 1:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: What I think might make more sense is to keep two variables, RecentGlobalXmin with its current meaning and

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Wed, Nov 23, 2011 at 3:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Nov 23, 2011 at 1:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: What I think might make more sense is to keep two variables, RecentGlobalXmin with its current meaning and

Re: [HACKERS] range_adjacent and discrete ranges

2011-11-23 Thread Tom Lane
I wrote: Attached is a draft patch for this. It passes regression tests but I've not tried to exercise it with a canonical function that actually does something different. I hacked up int4range_canonical to produce []-style ranges, and confirmed that this version of range_adjacent seems to

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 8:45 PM, Robert Haas robertmh...@gmail.com wrote: I've wondered a few times whether we could get rid of the RecentGlobalXmin computation from GetSnapshotData() altogether. You have to calculate an xmin, so its unavoidable. My patch actually improves the speed of

Re: [HACKERS] range_adjacent and discrete ranges

2011-11-23 Thread Tom Lane
I wrote: I did a little bit of performance testing on an x86_64 machine (Fedora 14), and found that the time to execute a clause like WHERE int4range(1,2) -|- int4range(x, 1000) (x being an integer Var) grows from 0.37 us to 0.56 us if we adopt the patched version of range_adjacent.

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: My patch actually improves the speed of snapshots, rather than slowing them as Tom's would. It can be arbitrarily fast if it doesn't have to get the right answer. Unfortunately, you're not producing the right answer. You can not exclude XIDs in other

Re: [HACKERS] [Review] Include detailed information about a row failing a CHECK constraint into the error message

2011-11-23 Thread Robert Haas
On Mon, Nov 21, 2011 at 8:59 AM, Jan Kundrát j...@flaska.net wrote: What is the suggested way to go form here? Shall I update the unit tests? Yes. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list

[HACKERS] Snapshot build updates

2011-11-23 Thread Magnus Hagander
The snapshots on the ftpsite have been down for a number of days, since hub.org upgraded the machine it used to be on and git stopped working there. Since we were planning to move it anyway, we didn't bother doing anything about it at the time. The snapshots are now auto-generated by buildfarm

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 9:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: My patch actually improves the speed of snapshots, rather than slowing them as Tom's would. It can be arbitrarily fast if it doesn't have to get the right answer. (LOL) - laughing

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Robert Haas
On Wed, Nov 23, 2011 at 5:43 PM, Simon Riggs si...@2ndquadrant.com wrote: Computing two cutoffs is overkill for the rare event of pruning a shared catalog page. I did think of that already and its not a good solution. I'm tempted by the idea of getting rid of multiple databases altogether. The

Re: [HACKERS] Not HOT enough

2011-11-23 Thread Simon Riggs
On Wed, Nov 23, 2011 at 10:47 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 23, 2011 at 5:43 PM, Simon Riggs si...@2ndquadrant.com wrote: Computing two cutoffs is overkill for the rare event of pruning a shared catalog page. I did think of that already and its not a good solution.

Re: [HACKERS] FlexLocks

2011-11-23 Thread Kevin Grittner
Kevin Grittner wrote: Robert Haas wrote: Updated patches attached. I have to admit I don't have my head around the extraWaits issue, so I can't personally vouch for that code, although I have no reason to doubt it, either. Everything else was something that I at least *think* I

Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays

2011-11-23 Thread Oliver Jowett
On 24 November 2011 05:36, Tom Lane t...@sss.pgh.pa.us wrote: Now it's possible we could do that without formally calling it a protocol version change, but I don't care at all for the idea of coming up with one-off hacks every time somebody decides that some feature is important enough that

Re: [HACKERS] [JDBC] Optimize postgres protocol for fixed size arrays

2011-11-23 Thread Kevin Grittner
Oliver Jowett wrote: Can we get a mechanism for minor protocol changes in this future version? Something as simple as exchanging a list of protocol features during the initial handshake (then use only features that are present on both sides) would be enough. The difficulty of making any

[HACKERS] PL/Python SQL error code pass-through

2011-11-23 Thread Mika Eloranta
Hi all, Here's a little SQL snippet that exposes an apparent regression in the 9.1.x PL/Python behavior: ---clip--- # cat foo.sql \set VERBOSITY 'verbose' CREATE table bar (a INTEGER CONSTRAINT hello CHECK (a 1)); CREATE OR REPLACE FUNCTION foo () RETURNS integer AS $$

[HACKERS] logging in high performance systems.

2011-11-23 Thread Theo Schlossnagle
We have a need for logging in systems where it isn't feasible to log to disk as it negatively impacts performance. I'd like to be able to creatively solve this problem without modifying the core, but today I cannot. So... here's my first whack at solving this with some flexibility. The first

[HACKERS] Obstacles to user-defined range canonicalization functions

2011-11-23 Thread Tom Lane
I got religion this evening about the potential usefulness of user-defined canonicalization functions --- the example that did it for me was thinking about a range type over timestamp that quantizes boundaries to hours, or half hours, or 15 minutes, or any scheduling unit that is standard in a

Re: [HACKERS] Obstacles to user-defined range canonicalization functions

2011-11-23 Thread Florian Pflug
On Nov24, 2011, at 04:33 , Tom Lane wrote: One possibility that just came to me is to decree that every discrete range type has to be based on an underlying continuous range type (with all the same properties except no canonicalization function), and then the discrete range's canonicalization

Re: [HACKERS] Obstacles to user-defined range canonicalization functions

2011-11-23 Thread David E. Wheeler
On Nov 23, 2011, at 10:33 PM, Tom Lane wrote: Now you could argue that for performance reasons everybody should write their canonicalization functions in C anyway, but I'm not sure I buy that --- at the very least, it'd be nice to write the functions in something higher-level while

Re: [HACKERS] logging in high performance systems.

2011-11-23 Thread Greg Smith
On 11/23/2011 09:28 PM, Theo Schlossnagle wrote: The second thing I did was write a sample use of those hooks to implement a completely non-blocking fifo logger. (if it would block, it drops the log line). The concept is that we could run this without risk of negative performance impact due to

Re: [HACKERS] pg_upgrade relation OID mismatches

2011-11-23 Thread Bruce Momjian
Bruce Momjian wrote: OK, that is a heap table. My only guess is that the heap is being created without binary_upgrade_next_heap_pg_class_oid being set. Looking at the code, I can't see how the heap could be created without this happening. Another idea is that pg_dumpall isn't output the

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Alexander Shulgin
Excerpts from Florian Weimer's message of Wed Nov 23 13:04:47 +0200 2011: * Alexander Shulgin: This, in my opinion, is very similar to what we would like to achieve with the URI syntax, so the above could also be specified using a URI parameter like this: psql -d

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Alexander Shulgin
Excerpts from Florian Weimer's message of Wed Nov 23 13:04:47 +0200 2011: * Alexander Shulgin: This, in my opinion, is very similar to what we would like to achieve with the URI syntax, so the above could also be specified using a URI parameter like this: psql -d

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Dmitriy Igrishin
Hey Alexander, 2011/11/24 Alexander Shulgin a...@commandprompt.com Excerpts from Florian Weimer's message of Wed Nov 23 13:04:47 +0200 2011: * Alexander Shulgin: This, in my opinion, is very similar to what we would like to achieve with the URI syntax, so the above could also be

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Alexander Shulgin
Excerpts from Dmitriy Igrishin's message of Thu Nov 24 09:19:02 +0200 2011: If host part of the URI points to localhost, the UNIX domain socket would be considered by libpq just as if you would pass -h localhost -p 5433. But what if the user wants to connect exactly via socket or TCP/IP

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Martijn van Oosterhout
On Thu, Nov 24, 2011 at 08:59:56AM +0200, Alexander Shulgin wrote: How would you specifiy a local port/UNIX domain socket? Missed that in my previous reply. If host part of the URI points to localhost, the UNIX domain socket would be considered by libpq just as if you would pass -h

Re: [HACKERS] Notes on implementing URI syntax for libpq

2011-11-23 Thread Dmitriy Igrishin
2011/11/24 Alexander Shulgin a...@commandprompt.com Excerpts from Dmitriy Igrishin's message of Thu Nov 24 09:19:02 +0200 2011: If host part of the URI points to localhost, the UNIX domain socket would be considered by libpq just as if you would pass -h localhost -p 5433. But