Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Itagaki Takahiro
On Wed, Oct 20, 2010 at 6:39 AM, Terry Laurenzo t...@laurenzo.org wrote: The answer may be to have both a jsontext and jsonbinary type as each will be optimized for a different case. I want to choose one format for JSON rather than having two types. It should be more efficient than other format

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Terry Laurenzo
Good points. In addition, any binary format needs to support object property traversal without having to do a deep scan of all descendants. BSON handles this with explicit lengths for document types (objects and arrays) so that entire parts of the tree can be skipped during sibling traversal. It

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: Robert Haas robertmh...@gmail.com writes: It seems good to do this in the normal case, but (1) if client_min_messages was already set higher than WARNING, we probably should not lower it and (2) we might want to have a way to lower it for troubleshooting

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: Stephen R. van den Berg s...@cuci.nl writes: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the columns, and the types of the columns available. Why don't we

[HACKERS] Installer Fix on some Windows 7 64-bit Systems

2010-10-20 Thread Humair Mohammed
One some Windows machines the command processor is not invoked properly when running the WScript.Run method. The same PostgreSQL 9.0.1-1 one click installer that worked perfrectly fine on one Windows 7 64-bit machine did not work as expected on another machine with same OS. Adding the command

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Roberto Mello wrote: On Tue, Oct 19, 2010 at 6:13 PM, Stephen R. van den Berg s...@cuci.nl wrote: Greg Stark wrote: premise this on the idea that you've lost everything in the catalog but not the data in other tables. Which seems like a narrow use case. It happens, more often than you'd think.

Re: [HACKERS] ISN patch that applies cleanly with git apply

2010-10-20 Thread Peter Geoghegan
I have committed the patch and the text proposed above. Can I take it that there is no need for a formal review, where I answer various questions per http://wiki.postgresql.org/wiki/Reviewing_a_Patch? -- Regards, Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Simplifying replication

2010-10-20 Thread Dimitri Fontaine
Josh Berkus j...@agliodbs.com writes: Well, one thing to be addressed is separating the PITR functionality from replication. PITR needs a lot of features -- timelines, recovery stop points, etc. -- which replication doesn't need or want. I think that focussing on streaming replication

Re: [HACKERS] ISN patch that applies cleanly with git apply

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:12 AM, Peter Geoghegan peter.geoghega...@gmail.com wrote: I have committed the patch and the text proposed above. Can I take it that there is no need for a formal review, where I answer various questions per http://wiki.postgresql.org/wiki/Reviewing_a_Patch? That is

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:22 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: In v6 patch, should client_min_messages or log_min_messages be lower than WARNING, they get set to WARNING for the script install context. We still dump the extension's script at each WARNING, but you can set your

Re: [HACKERS] ISN patch that applies cleanly with git apply

2010-10-20 Thread Peter Eisentraut
On ons, 2010-10-20 at 11:12 +0100, Peter Geoghegan wrote: Can I take it that there is no need for a formal review, where I answer various questions per http://wiki.postgresql.org/wiki/Reviewing_a_Patch? The short answer is no. But note that there is no such thing as a formal review. The page

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 1:06 AM, Greg Smith g...@2ndquadrant.com wrote: Josh Berkus wrote: Well, now that you mention it, I also think that hot standby should be the default.  Yes, I know about the overhead, but I also think that the number of our users who want easy replication *far*

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Robert Haas
On Tue, Oct 19, 2010 at 11:30 PM, daveg da...@sonic.net wrote: On Wed, Oct 20, 2010 at 04:08:37PM +1300, Mark Kirkwood wrote: On 20/10/10 16:05, Mark Kirkwood wrote: shmget and friends are hugetlbpage  aware, so it seems it should 'just work'. Heh - provided you specify SHM_HUGETLB

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: I would vote for overriding client_min_messages but not log_min_messages. Well it defaults to WARNING so I see your point. Then again, we're talking about hundreds of lines (3197 lines of isn, 531 lines for hstore) of output per message, containing a

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 9:33 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: I would vote for overriding client_min_messages but not log_min_messages. Well it defaults to WARNING so I see your point. Then again, we're talking about hundreds of

Re: [HACKERS] leaky views, yet again

2010-10-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I get the impression that you think that there's a problem not only with the approach but with any approach whatsoever to that underlying problem. Let's just say that the approaches proposed so far have performance and/or functionality and/or code

Re: [HACKERS] Domains versus arrays versus typmods

2010-10-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Oct 19, 2010 at 9:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: We've already accepted the cost of doing getBaseTypeAndTypmod() in a whole lot of performance-critical parsing paths, on the off chance that the target datatype might be a domain.  

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Oct 19, 2010 at 11:30 PM, daveg da...@sonic.net wrote: On Wed, Oct 20, 2010 at 04:08:37PM +1300, Mark Kirkwood wrote: Heh - provided you specify SHM_HUGETLB in the relevant call that is :-) I had a patch for this against 8.3 that I could

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Itagaki Takahiro itagaki.takah...@gmail.com writes: On Wed, Oct 20, 2010 at 12:58 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Lets rename the directory. Hmmm, but we call it 'xml2' in the doc. There is no 'pgxml' at all in it.

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: Josh Berkus wrote: Well, now that you mention it, I also think that hot standby should be the default. Yes, I know about the overhead, but I also think that the number of our users who want easy replication *far* outnumber the users who care about an

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: If the extensions manager is dependent on the assumption that a module's name matches the name of the directory it's built in It is not. There's some magic for simple cases so that contrib mostly works with no editing, but of course, that's only mostly. The

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Tom Lane
Stephen R. van den Berg s...@cuci.nl writes: It's just that matching table and file, and subsequently figuring out some missing columns which may have been added/removed later, can be rather timeconsuming and could be made a lot easier (not necessarily perfect) if that information would have

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Tatsuo Ishii
pg_is_in_recovery() returns a bool, are you proposing to change that? No. I just thought about adding more condition when it returns true. Here is the patch. Comments are welcome! *** a/src/backend/access/transam/xlog.c --- b/src/backend/access/transam/xlog.c *** *** 5604,5610

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Oct 20, 2010 at 6:22 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: In v6 patch, should client_min_messages or log_min_messages be lower than WARNING, they get set to WARNING for the script install context. We still dump the extension's

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Kenneth Marshall
On Wed, Oct 20, 2010 at 10:10:00AM -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Oct 19, 2010 at 11:30 PM, daveg da...@sonic.net wrote: On Wed, Oct 20, 2010 at 04:08:37PM +1300, Mark Kirkwood wrote: Heh - provided you specify SHM_HUGETLB in the relevant call

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 17:19, Tom Lane wrote: Greg Smithg...@2ndquadrant.com writes: Josh Berkus wrote: Well, now that you mention it, I also think that hot standby should be the default. Yes, I know about the overhead, but I also think that the number of our users who want easy replication *far*

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 17:32, Tatsuo Ishii wrote: pg_is_in_recovery() returns a bool, are you proposing to change that? No. I just thought about adding more condition when it returns true. Here is the patch. Comments are welcome! ... Datum pg_is_in_recovery(PG_FUNCTION_ARGS) { ! /* use

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Going from wal_level='minimal' to 'archivë́' incurs the penalty on WAL-logging COPY etc. That's a big penalty. However, the difference between wal_level='archive' and wal_level='hot_standby' should be tiny. I'm not sure I

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: Stephen R. van den Berg s...@cuci.nl writes: It's just that matching table and file, and subsequently figuring out some missing columns which may have been added/removed later, can be rather timeconsuming and could be made a lot easier (not necessarily perfect) if that

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Alvaro Herrera
Excerpts from Dimitri Fontaine's message of mié oct 20 07:22:53 -0300 2010: Itagaki Takahiro itagaki.takah...@gmail.com writes: CREATE EXTENSION command * Environment could be modified by the installer script. =# SHOW search_path; = $user,public =# CREATE EXTENSION dblink; =#

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: If the extensions manager is dependent on the assumption that a module's name matches the name of the directory it's built in It is not. There's some magic for simple cases so that contrib mostly works with

Re: [HACKERS] Domains versus arrays versus typmods

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:03 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Oct 19, 2010 at 9:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: We've already accepted the cost of doing getBaseTypeAndTypmod() in a whole lot of performance-critical parsing

Re: [HACKERS] leaky views, yet again

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:00 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I get the impression that you think that there's a problem not only with the approach but with any approach whatsoever to that underlying problem. Let's just say that the approaches

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 20.10.2010 17:32, Tatsuo Ishii wrote: pg_is_in_recovery() returns a bool, are you proposing to change that? No. I just thought about adding more condition when it returns true. Here is the

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié oct 20 10:29:04 -0300 2010: Actually, I think the best thing for default_statistics_target might be to scale the target based on the number of rows in the table, e.g. given N rows: 10 + (N / 1000), if N 40,000 46 + (N / 1), if 50,000 N

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Dimitri Fontaine's message of mié oct 20 07:22:53 -0300 2010: Using SPI to execute the extension's script already means that it can not contain explicit BEGIN and COMMIT commands. Now, is it possible to force a Reset of all GUCs

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:53 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mié oct 20 10:29:04 -0300 2010: Actually, I think the best thing for default_statistics_target might be to scale the target based on the number of rows in the table, e.g.

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Tatsuo Ishii
This returns 'false' if you're in hot standby mode running against an archive. That seems wrong, I don't think the walreceiver state should play any role in this. Apart this, I wonder why walsender/walreceiver do not transfer archive logs as well. -- Tatsuo Ishii SRA OSS, Inc. Japan English:

Re: [HACKERS] Serializable snapshot isolation patch

2010-10-20 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: On Tue, Oct 19, 2010 at 6:28 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: One thing that would work, but I really don't think I like it, is that a request for a snapshot for such a transaction would not only block until it could get a clean

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: I don't think that no changes to the makefiles is a requirement, or even a wish-list item, for this. I think it's perfectly reasonable for the makefile to have to specify the module name; far better that than that we get the name by some magic or other. It

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 18:06, Tatsuo Ishii wrote: This returns 'false' if you're in hot standby mode running against an archive. That seems wrong, I don't think the walreceiver state should play any role in this. Apart this, I wonder why walsender/walreceiver do not transfer archive logs as well.

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: and use the equivalent of SET LOCAL in the CREATE EXTENSION code? I had assumed that that was how he was doing it ... I'm currently doing: SetConfigOption(client_min_messages, warning, PGC_SUSET, PGC_S_SESSION); And then manually reverting to what

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: I don't think that no changes to the makefiles is a requirement, or even a wish-list item, for this. I think it's perfectly reasonable for the makefile to have to specify the module name; far better that than

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: That is simply a horrid idea. Just make it specify EXTENSION. Black magic it is, will remove in v7. Is there any sane use-case for the control file to not be named the same as the extension? It seems like that would accomplish little except to sow

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-20 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Tue, Oct 19, 2010 at 4:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think they're more alike than you think.  If we had the infrastructure to do local temp tables this way, it'd be pretty easy to use that to instantiate per-backend copies of global temp

[HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Marti Raudsepp
Hi pgsql-hackers, Currently contrib/pgcrypto/pgp-pubenc.c contains code like: uint8 algo = pk-algo; if (pk == NULL) ... However, if pk was NULL, then the if() condition would never be reached because the pk-algo dereference would segfault. This patch moves the dereference to below the

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Aidan Van Dyk
On Wed, Oct 20, 2010 at 10:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: Stephen R. van den Berg s...@cuci.nl writes: It's just that matching table and file, and subsequently figuring out some missing columns which may have been added/removed later, can be rather timeconsuming and could be made a

Re: [HACKERS] Simplifying replication

2010-10-20 Thread Robert Treat
On Tue, Oct 19, 2010 at 11:16 AM, Greg Smith g...@2ndquadrant.com wrote: Josh Berkus wrote: It is critical that we make replication easier to set up, administrate and monitor than it currently is. In my conversations with people, this is more important to our users and the adoption of

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Tom Lane
Aidan Van Dyk ai...@highrise.ca writes: On Wed, Oct 20, 2010 at 10:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: I can see the potential usefulness of a self-documenting table storage format, but this proposal isn't that; it's just an unreliable kluge. If we're looking to have any sort of out of

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Florian Weimer
* Terry Laurenzo: Agreed. BSON was born out of implementations that either lacked arbitrary precision numbers or had a strong affinity to an int/floating point way of thinking about numbers. I believe that if BSON had an arbitrary precision number type, it would be a proper superset of

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Andrew Dunstan
On 10/20/2010 01:15 PM, Florian Weimer wrote: * Terry Laurenzo: Agreed. BSON was born out of implementations that either lacked arbitrary precision numbers or had a strong affinity to an int/floating point way of thinking about numbers. I believe that if BSON had an arbitrary precision

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Aidan Van Dyk
On Wed, Oct 20, 2010 at 1:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: [ thinks for a bit... ]  Perhaps we could stick some sort of unique ID into tables, which could be correlated to the same unique ID appearing in a metadata fork. It would be awfully nice if the table name was in the first few

[HACKERS] pg_upgrade performance with 150k tables

2010-10-20 Thread Bruce Momjian
I received a private email report yesterday from someone using pg_upgrade with PG 9.0 who found it took five hours for pg_upgrade to upgrade a database with 150k tables. Yes, that is a lot of tables, but pg_upgrade should be able to do better than that. I have modified pg_upgrade in git master

[HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Bruce Momjian
Bruce Momjian wrote: I received a private email report yesterday from someone using pg_upgrade with PG 9.0 who found it took five hours for pg_upgrade to upgrade a database with 150k tables. Yes, that is a lot of tables, but pg_upgrade should be able to do better than that. I have modified

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread David Fetter
On Tue, Oct 19, 2010 at 08:51:16PM -0400, Robert Haas wrote: On Tue, Oct 19, 2010 at 5:42 PM, Andrew Dunstan and...@dunslane.net wrote: Well a bit more testing shows some benefit. I've sorted out a few kinks, so this seems to work. In particular, with the above tables, the version imported

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 7:10 AM, Tom Lane t...@sss.pgh.pa.us wrote: I believe that for the equivalent Solaris option, we just automatically enable it when available.  So there'd be no need for user documentation. However, I definitely *would* like to see some benchmarks proving that the change

Re: [HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:01 PM, Bruce Momjian br...@momjian.us wrote: One more question --- should I be sending pg_upgrade patches to the list for approval?  The restructuring patch was large and didn't seem necessary to post, and the speedups were tested by the bug reporter, so I figured

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark gsst...@mit.edu wrote: I don't think it's a big cost once all the processes have been forked if you're reusing them beyond perhaps slightly more efficient cache usage. Hm, this site claims to get a 13% win just from the reduced tlb misses using a

Re: [HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Magnus Hagander
On Wed, Oct 20, 2010 at 21:28, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 20, 2010 at 3:01 PM, Bruce Momjian br...@momjian.us wrote: One more question --- should I be sending pg_upgrade patches to the list for approval?  The restructuring patch was large and didn't seem necessary to

Re: [HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Bruce Momjian
Magnus Hagander wrote: On Wed, Oct 20, 2010 at 21:28, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 20, 2010 at 3:01 PM, Bruce Momjian br...@momjian.us wrote: One more question --- should I be sending pg_upgrade patches to the list for approval? ?The restructuring patch was large and

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

2010-10-20 Thread Alvaro Herrera
Excerpts from Tom Lane's message of lun oct 04 10:31:26 -0400 2010: In the particular case at hand here, I rather wonder why SQL functions are depending on postgres.c at all. It might be better to just duplicate a bit of code to make them independent. pg_parse_and_rewrite would then be dead

Re: [HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 18:44, Marti Raudsepp wrote: Hi pgsql-hackers, Currently contrib/pgcrypto/pgp-pubenc.c contains code like: uint8 algo = pk-algo; if (pk == NULL) ... However, if pk was NULL, then the if() condition would never be reached because the pk-algo dereference would segfault. This

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Alvaro Herrera
Excerpts from Greg Stark's message of mié oct 20 16:28:25 -0300 2010: On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark gsst...@mit.edu wrote: I don't think it's a big cost once all the processes have been forked if you're reusing them beyond perhaps slightly more efficient cache usage. Hm,

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 6:29 AM, Robert Haas robertmh...@gmail.com wrote: Exactly.  It doesn't take many 3-7% slowdowns to add up to being 50% or 100% slower, and that sucks.  In fact, I'm still not convinced that we were wise to boost default_statistics_target as much as we did.  I argued for

[HACKERS] pg_upgrade cleanup

2010-10-20 Thread Bruce Momjian
I have applied the attached patch to clean up pg_upgrade cache lookup code and remove useless NULL pointer tests. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:40 PM, Greg Stark gsst...@mit.edu wrote: On Wed, Oct 20, 2010 at 6:29 AM, Robert Haas robertmh...@gmail.com wrote: Exactly.  It doesn't take many 3-7% slowdowns to add up to being 50% or 100% slower, and that sucks.  In fact, I'm still not convinced that we were wise

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:47 PM, daveg da...@sonic.net wrote: On Wed, Oct 20, 2010 at 12:28:25PM -0700, Greg Stark wrote: On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark gsst...@mit.edu wrote: I don't think it's a big cost once all the processes have been forked if you're reusing them beyond

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 8:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think it's pointless to speculate about whether we might have divvied up the meta-information about tables differently if we'd foreseen wanting to do this.  It is what it is, and there is *way* too much code depending on it,

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:16 PM, David Fetter da...@fetter.org wrote: On Tue, Oct 19, 2010 at 08:51:16PM -0400, Robert Haas wrote: On Tue, Oct 19, 2010 at 5:42 PM, Andrew Dunstan and...@dunslane.net wrote: Well a bit more testing shows some benefit. I've sorted out a few kinks, so this seems

Re: [HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Marti Raudsepp
On Wed, Oct 20, 2010 at 22:34, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Did coccicheck find anything else interesting? There's a file descriptor leak in psql/command.c function process_file() -- on errors it just returns without closing the file. But since it's quitting

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-20 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Wed, Oct 20, 2010 at 8:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think it's pointless to speculate about whether we might have divvied up the meta-information about tables differently if we'd foreseen wanting to do this.  It is what it is, and there is

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 1:12 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 20, 2010 at 3:40 PM, Greg Stark gsst...@mit.edu wrote: On Wed, Oct 20, 2010 at 6:29 AM, Robert Haas robertmh...@gmail.com wrote: Actually, I think the best thing for default_statistics_target might be to

Re: [HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Tom Lane
Marti Raudsepp ma...@juffo.org writes: There's a file descriptor leak in psql/command.c function process_file() -- on errors it just returns without closing the file. But since it's quitting anyway, there's no practical impact. Should I submit a patch for this as well? Might as well. It's

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread daveg
On Wed, Oct 20, 2010 at 12:28:25PM -0700, Greg Stark wrote: On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark gsst...@mit.edu wrote: I don't think it's a big cost once all the processes have been forked if you're reusing them beyond perhaps slightly more efficient cache usage. Hm, this site

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: Aidan Van Dyk ai...@highrise.ca writes: If we're looking to have any sort of out of catalog documentation of table storage format, what about just having a new relation fork that just appends each and every change made to the table formats, including ones rolled back, etc. I

[HACKERS] psql: Don't close stdin, don't leak file descriptor with ON_ERROR_STOP

2010-10-20 Thread Marti Raudsepp
Hi list, Here's the second patch from my coccicheck run. Originally it flagged the fact that the opened file in psql's process_file() wasn't being closed in the ON_ERROR_STOP path, but there seem to be two more unintended behaviors here. (1) In the error path, the value of pset.inputfile wasn't

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: That is simply a horrid idea. Just make it specify EXTENSION. And VERSION too, finally. So any extension and guessing the CONTROL file name from the EXTENSION name only occurs when CONTROL has not been provided. Here, on the other hand, I'm wondering

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

2010-10-20 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mié oct 20 16:33:12 -0300 2010: The only quarrel I have with this code shuffling is that pg_rewrite_query is being called from exec_parse_message. Since it's now a static function, it would have to stop being static so that it can be called from both

Re: [HACKERS] Proposed Windows-specific change: Enable crash dumps (like core files)

2010-10-20 Thread Bruce Momjian
Craig Ringer wrote: On 4/10/2010 8:06 PM, Andrew Dunstan wrote: On 10/04/2010 07:50 AM, Craig Ringer wrote: - If the crash dump handler is enabled by setting the GUC, all backends register the handler during startup or (if it proves practical) when the GUC is changed. - When the

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Josh Berkus
Maybe what should be done about this is to have separate sizes for the MCV list and the histogram, where the MCV list is automatically sized during ANALYZE. It's been suggested multiple times that we should base our sample size on a % of the table, or at least offer that as an option. I've

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

2010-10-20 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: It strikes me that if we really want to restructure things to divide client interaction from other query-processing routines, we should create another file, say src/backend/tcop/queries.c; this would have stuff like pg_plan_query,

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Josh Berkus
Quite. Josh, have you got any evidence showing that the penalty is only 10%? There are cases, such as COPY and ALTER TABLE, where you'd be looking at 2X or worse penalties, because of the existing optimizations that avoid writing WAL at all for operations where a single final fsync can

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 5:30 PM, Stephen R. van den Berg s...@cuci.nl wrote: Ideal would be: put the table-oid inside the header of each page (either in the official header, or in the special area). This way even lost blocks can be correlated to the same table. I'd still vote for the latest

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 3:15 PM, Josh Berkus j...@agliodbs.com wrote: Maybe what should be done about this is to have separate sizes for the MCV list and the histogram, where the MCV list is automatically sized during ANALYZE. It's been suggested multiple times that we should base our sample

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread Merlin Moncure
On Tue, Oct 19, 2010 at 9:15 PM, Andrew Dunstan and...@dunslane.net wrote: Efficiency has  always been one of the major reasons for using enums, so it's important that we make them extensible without badly affecting performance. on that note is it worthwhile backpatching recent versions to

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:38 PM, Greg Stark gsst...@mit.edu wrote: On Wed, Oct 20, 2010 at 3:15 PM, Josh Berkus j...@agliodbs.com wrote: Maybe what should be done about this is to have separate sizes for the MCV list and the histogram, where the MCV list is automatically sized during ANALYZE.

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:54 PM, Merlin Moncure mmonc...@gmail.com wrote: On Tue, Oct 19, 2010 at 9:15 PM, Andrew Dunstan and...@dunslane.net wrote: Efficiency has  always been one of the major reasons for using enums, so it's important that we make them extensible without badly affecting

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 3:54 PM, Robert Haas robertmh...@gmail.com wrote: Yes, I think a percentage of the table is going to break down either at the high end or the low end.  Hand-waving (but based on experience), for a 1000 row table a statistics target of 10 is probably approximately right

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread David E. Wheeler
On Oct 20, 2010, at 3:12 PM, Dimitri Fontaine wrote: So, the idea is that $(EXTENSION) is a list of extensions you're providing from the Makefile (most often, a list of one extension, but contrib/spi is an exception here). Each extension in the list must have a corresponding

Re: [HACKERS] Issues with Quorum Commit

2010-10-20 Thread Bruce Momjian
Tom Lane wrote: Greg Smith g...@2ndquadrant.com writes: I don't see this as needing any implementation any more complicated than the usual way such timeouts are handled. Note how long you've been trying to reach the standby. Default to -1 for forever. And if you hit the timeout,

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Josh Berkus
Why? Afaict this has been suggested multiple times by people who don't justify it in any way except with handwavy -- larger samples are better. The sample size is picked based on what sample statistics tells us we need to achieve a given 95th percentile confidence interval for the bucket

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 6:03 PM, Josh Berkus j...@agliodbs.com wrote: I also just realized that I confused myself ... we don't really want more MCVs.  What we want it more *samples* to derive a small number of MCVs.  Right now # of samples and number of MCVs is inexorably bound, and they

[HACKERS] lazy snapshots?

2010-10-20 Thread Robert Haas
I had the following idea for an optimization. Feel free to tell me I'm nuts. Would it be possible to postpone the operation of taking a snapshot until we encounter an in-doubt tuple - that is, a tuple whose XMIN or XMAX is committed but not all-visible? It seems to me that there are many

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Josh Berkus
I don't see why the MCVs would need a particularly large sample size to calculate accurately. Have you done any tests on the accuracy of the MCV list? Yes, although I don't have them at my fingertips. In sum, though, you can't take 10,000 samples from a 1b row table and expect to get a

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 7:13 PM, Greg Stark gsst...@mit.edu wrote: On Wed, Oct 20, 2010 at 3:54 PM, Robert Haas robertmh...@gmail.com wrote: Yes, I think a percentage of the table is going to break down either at the high end or the low end.  Hand-waving (but based on experience), for a 1000

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Nathan Boley
Robert explained why having more MCVs might be useful because we use the frequency of the least common MCV as an upper bound on the frequency of any value in the MCV. Where is that being used? The only non-MCV frequency estimate that I recall seeing is ( nrows - n_ndistinct_rows )/ndistinct.

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:17 PM, Josh Berkus j...@agliodbs.com wrote: Quite.  Josh, have you got any evidence showing that the penalty is only 10%?  There are cases, such as COPY and ALTER TABLE, where you'd be looking at 2X or worse penalties, because of the existing optimizations that avoid

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Itagaki Takahiro
On Thu, Oct 21, 2010 at 7:12 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: This control file contains at minimum a single line for the name of the extension, but it's better already with a comment for users. I've been filling them for our extensions, pasting from the documentation:      

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 9:53 PM, Nathan Boley npbo...@gmail.com wrote: Robert explained why having more MCVs might be useful because we use the frequency of the least common MCV as an upper bound on the frequency of any value in the MCV. Where is that being used? var_eq_const The only

Re: [HACKERS] lazy snapshots?

2010-10-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: It's necessary to convince ourselves not only that this has some performance benefit but that it's actually correct. It's easy to see that, if we never take a snapshot, all the tuple visibility decisions we make will be exactly identical to the ones

  1   2   >