Re: [HACKERS] Domains versus arrays versus typmods

2010-10-20 Thread Richard Huxton
On 20/10/10 01:47, Robert Haas wrote: On Tue, Oct 19, 2010 at 6:14 PM, Tom Lane wrote: Comments? It might be reasonable to back-patch whatever we decide on into 9.0, because it is so new, but I would be reluctant to go back further unless we have some evidence that it's bothering people. It

Re: [HACKERS] UNION ALL has higher cost than inheritance

2010-10-20 Thread Itagaki Takahiro
On Thu, Oct 21, 2010 at 2:18 PM, Tom Lane wrote: > The plan for UNION initially involves a couple of SubqueryScan nodes, > which impose an extra cost of cpu_tuple_cost per tuple.  Those later > get optimized away, but we don't try to readjust the cost estimates > for that. Thanks. It also explain

Re: [HACKERS] Serializable snapshot isolation patch

2010-10-20 Thread Jeff Davis
On Sun, 2010-10-17 at 22:53 -0700, Jeff Davis wrote: > 2. I think there's a GiST bug (illustrating with PERIOD type): > > create table foo(p period); > create index foo_idx on foo using gist (p); > insert into foo select period( > '2009-01-01'::timestamptz + g * '1 microsecond'::interv

Re: [HACKERS] UNION ALL has higher cost than inheritance

2010-10-20 Thread Tom Lane
Itagaki Takahiro writes: > I found an explicit UNION ALL has higher cost than an automatic expansion > by inheritance (49 vs. 83 in the example below). Where does the difference > come from? The plan for UNION initially involves a couple of SubqueryScan nodes, which impose an extra cost of cpu_tu

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread David E. Wheeler
On Oct 20, 2010, at 9:58 PM, Alvaro Herrera wrote: > What's wrong with sticking to Makefile syntax? Are we intending to > build a JSON parser in GNU make perchance? That metadata isn't *for* make, is it? D -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

[HACKERS] UNION ALL has higher cost than inheritance

2010-10-20 Thread Itagaki Takahiro
I found an explicit UNION ALL has higher cost than an automatic expansion by inheritance (49 vs. 83 in the example below). Where does the difference come from? Since they have almost same plan trees, should it be the same cost? =# CREATE TABLE parent (i integer); =# CREATE TABLE child () INHERITS

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Alvaro Herrera
Excerpts from Itagaki Takahiro's message of jue oct 21 00:01:59 -0300 2010: > On Thu, Oct 21, 2010 at 8:14 AM, David E. Wheeler > wrote: > > Might I suggest instead a META.json file like PGXN requires? > > I think JSON is also reasonable, but one of the problem to use JSON format is > we cannot

Re: [HACKERS] pg_hba.conf host name wildcard support

2010-10-20 Thread Tom Lane
Peter Eisentraut writes: > So, as previously indicated, let's add some wildcard support to the > pg_hba.conf host name feature. After looking around a bit, two syntaxes > appear to be on offer: > 1. TCP Wrappers style, leading dot indicates suffix match. > So .example.com matches anything.exampl

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 6:41 PM, Josh Berkus wrote: > A while back I did a fair bit of reading on ndistinct and large tables > from the academic literature.  The consensus of many papers was that it > took a sample of at least 3% (or 5% for block-based) of the table in > order to have 95% confiden

[HACKERS] pg_hba.conf host name wildcard support

2010-10-20 Thread Peter Eisentraut
So, as previously indicated, let's add some wildcard support to the pg_hba.conf host name feature. After looking around a bit, two syntaxes appear to be on offer: 1. TCP Wrappers style, leading dot indicates suffix match. So .example.com matches anything.example.com. Not sure how useful that wou

Re: [HACKERS] lazy snapshots?

2010-10-20 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 20, 2010 at 10:07 PM, Tom Lane wrote: >> I'm less than convinced by the hypothesis that most transactions would >> avoid taking snapshots in this regime, anyway.  It would only hold up >> if there's little locality of reference in terms of which tuples are >> get

Re: [HACKERS] lazy snapshots?

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:07 PM, Tom Lane wrote: > Robert Haas writes: >> It's necessary to convince ourselves not only that this has some >> performance benefit but that it's actually correct.  It's easy to see >> that, if we never take a snapshot, all the tuple visibility decisions >> we make

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Itagaki Takahiro
On Thu, Oct 21, 2010 at 8:14 AM, David E. Wheeler wrote: > Might I suggest instead a META.json file like PGXN requires? I think JSON is also reasonable, but one of the problem to use JSON format is we cannot apply the extension patch until JSON patch has been applied ;-) BTW, does anyone needs J

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Greg Smith
Josh Berkus wrote: If we could agree on some workloads, I could run some benchmarks. I'm not sure what those would be though, given that COPY and ALTER TABLE aren't generally included in most benchmarks. You can usefully and easily benchmark this by timing a simple pgbench initialization at a

[HACKERS] psql autocompletion for \z and \dg

2010-10-20 Thread Josh Kupershmidt
Hi all, It looks like psql's tab completion for the \z and \dg commands in psql are missing. I couldn't see a reason for this, so attached patch fixes. Also, this patch proposes to change psql's "\?" help text to say that \dg and \du are the same, since AFAICT they do exactly the same thing. Jos

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Tom Lane
Josh Berkus writes: > A while back I did a fair bit of reading on ndistinct and large tables > from the academic literature. The consensus of many papers was that it > took a sample of at least 3% (or 5% for block-based) of the table in > order to have 95% confidence in ndistinct of 3X. I can't

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Joshua D. Drake
On Wed, 2010-10-20 at 15:15 -0700, Josh Berkus wrote: > >> Maybe what should be done about this is to have separate sizes for the > >> MCV list and the histogram, where the MCV list is automatically sized > >> during ANALYZE. > > It's been suggested multiple times that we should base our sample si

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Nathan Boley
> That one's used, too, but the other is used as an upper bound. > n_distinct tends to come out too small on large tables, so that > formula is prone to overestimation.  Actually, both formulas are prone > to overestimation. > Right - thanks. > When this happens depends on the values of a whole b

Re: [HACKERS] lazy snapshots?

2010-10-20 Thread Tom Lane
Robert Haas writes: > It's necessary to convince ourselves not only that this has some > performance benefit but that it's actually correct. It's easy to see > that, if we never take a snapshot, all the tuple visibility decisions > we make will be exactly identical to the ones that we would have

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 9:53 PM, Nathan Boley wrote: >> Robert explained why having more MCVs might be useful because we use >> the frequency of the least common MCV as an upper bound on the >> frequency of any value in the MCV. > > Where is that being used? var_eq_const > The only non-MCV frequ

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Itagaki Takahiro
On Thu, Oct 21, 2010 at 7:12 AM, Dimitri Fontaine wrote: > This control file contains at minimum a single line for the name of the > extension, but it's better already with a comment for users. I've been > filling them for our extensions, pasting from the documentation: > >        name        | v

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:17 PM, Josh Berkus wrote: >> Quite.  Josh, have you got any evidence showing that the penalty is >> only 10%?  There are cases, such as COPY and ALTER TABLE, where >> you'd be looking at 2X or worse penalties, because of the existing >> optimizations that avoid writing WA

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Nathan Boley
> Robert explained why having more MCVs might be useful because we use > the frequency of the least common MCV as an upper bound on the > frequency of any value in the MCV. Where is that being used? The only non-MCV frequency estimate that I recall seeing is ( nrows - n_ndistinct_rows )/ndistinct

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 7:13 PM, Greg Stark wrote: > On Wed, Oct 20, 2010 at 3:54 PM, Robert Haas wrote: >> Yes, I think a percentage of the table is going to break down either >> at the high end or the low end.  Hand-waving (but based on >> experience), for a 1000 row table a statistics target o

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Josh Berkus
> I don't see why the MCVs would need a particularly large sample size > to calculate accurately. Have you done any tests on the accuracy of > the MCV list? Yes, although I don't have them at my fingertips. In sum, though, you can't take 10,000 samples from a 1b row table and expect to get a rem

[HACKERS] lazy snapshots?

2010-10-20 Thread Robert Haas
I had the following idea for an optimization. Feel free to tell me I'm nuts. Would it be possible to postpone the operation of taking a snapshot until we encounter an in-doubt tuple - that is, a tuple whose XMIN or XMAX is committed but not all-visible? It seems to me that there are many transac

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 6:03 PM, Josh Berkus wrote: > I also just realized that I confused myself ... we don't really want > more MCVs.  What we want it more *samples* to derive a small number of > MCVs.  Right now # of samples and number of MCVs is inexorably bound, > and they shouldn't be.  On l

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Josh Berkus
> Why? Afaict this has been suggested multiple times by people who don't > justify it in any way except with handwavy -- larger samples are > better. The sample size is picked based on what sample statistics > tells us we need to achieve a given 95th percentile confidence > interval for the bucket

Re: [HACKERS] Issues with Quorum Commit

2010-10-20 Thread Bruce Momjian
Tom Lane wrote: > Greg Smith writes: > > I don't see this as needing any implementation any more complicated than > > the usual way such timeouts are handled. Note how long you've been > > trying to reach the standby. Default to -1 for forever. And if you hit > > the timeout, mark the standb

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread David E. Wheeler
On Oct 20, 2010, at 3:12 PM, Dimitri Fontaine wrote: > So, the idea is that $(EXTENSION) is a list of extensions you're > providing from the Makefile (most often, a list of one extension, but > contrib/spi is an exception here). Each extension in the list must have > a corresponding $EXTENSION.con

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 3:54 PM, Robert Haas wrote: > Yes, I think a percentage of the table is going to break down either > at the high end or the low end.  Hand-waving (but based on > experience), for a 1000 row table a statistics target of 10 is > probably approximately right and 100 is too muc

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:54 PM, Merlin Moncure wrote: > On Tue, Oct 19, 2010 at 9:15 PM, Andrew Dunstan wrote: >> Efficiency has  always been one of the major reasons for using enums, so >> it's important that we make them extensible without badly affecting >> performance. > > on that note is it

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 6:38 PM, Greg Stark wrote: > On Wed, Oct 20, 2010 at 3:15 PM, Josh Berkus wrote: Maybe what should be done about this is to have separate sizes for the MCV list and the histogram, where the MCV list is automatically sized during ANALYZE. >> >> It's been sugg

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread Merlin Moncure
On Tue, Oct 19, 2010 at 9:15 PM, Andrew Dunstan wrote: > Efficiency has  always been one of the major reasons for using enums, so > it's important that we make them extensible without badly affecting > performance. on that note is it worthwhile backpatching recent versions to allocate enums with

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 3:15 PM, Josh Berkus wrote: > >>> Maybe what should be done about this is to have separate sizes for the >>> MCV list and the histogram, where the MCV list is automatically sized >>> during ANALYZE. > > It's been suggested multiple times that we should base our sample size

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 5:30 PM, Stephen R. van den Berg wrote: > Ideal would be: put the table-oid inside the header of each page > (either in the official header, or in the special area). > This way even lost blocks can be correlated to the same table. > I'd still vote for the latest known table

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Josh Berkus
> Quite. Josh, have you got any evidence showing that the penalty is > only 10%? There are cases, such as COPY and ALTER TABLE, where > you'd be looking at 2X or worse penalties, because of the existing > optimizations that avoid writing WAL at all for operations where a > single final fsync can

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

2010-10-20 Thread Tom Lane
Alvaro Herrera writes: > It strikes me that if we really want to restructure things to divide > client interaction from other query-processing routines, we should > create another file, say src/backend/tcop/queries.c; this would have > stuff like pg_plan_query, pg_plan_queries, pg_rewrite_query, a

Re: [HACKERS] default_statistics_target WAS: max_wal_senders must die

2010-10-20 Thread Josh Berkus
>> Maybe what should be done about this is to have separate sizes for the >> MCV list and the histogram, where the MCV list is automatically sized >> during ANALYZE. It's been suggested multiple times that we should base our sample size on a % of the table, or at least offer that as an option. I

Re: [HACKERS] Proposed Windows-specific change: Enable crash dumps (like core files)

2010-10-20 Thread Bruce Momjian
Craig Ringer wrote: > On 4/10/2010 8:06 PM, Andrew Dunstan wrote: > > > > > > On 10/04/2010 07:50 AM, Craig Ringer wrote: > >> > >> - If the crash dump handler is enabled by setting the GUC, > >> all backends register the handler during startup or (if it > >> proves practical) when the GUC is chang

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

2010-10-20 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mié oct 20 16:33:12 -0300 2010: > The only quarrel I have with this code shuffling is that > pg_rewrite_query is being called from exec_parse_message. Since it's > now a static function, it would have to stop being static so that it can > be called from b

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane writes: > That is simply a horrid idea. Just make it specify EXTENSION. And VERSION too, finally. So any extension > >> and guessing >> the CONTROL file name from the EXTENSION name only occurs when CONTROL >> has not been provided. > > Here, on the other hand, I'm wondering why have

[HACKERS] psql: Don't close stdin, don't leak file descriptor with ON_ERROR_STOP

2010-10-20 Thread Marti Raudsepp
Hi list, Here's the second patch from my coccicheck run. Originally it flagged the fact that the opened file in psql's process_file() wasn't being closed in the ON_ERROR_STOP path, but there seem to be two more unintended behaviors here. (1) In the error path, the value of pset.inputfile wasn't b

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: >Aidan Van Dyk writes: >> If we're looking to have any sort of "out of catalog" documentation of >> table storage format, what about just having a new relation fork that >> just "appends" each and every change made to the table formats, >> including ones rolled back, etc. I presum

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread daveg
On Wed, Oct 20, 2010 at 12:28:25PM -0700, Greg Stark wrote: > On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark wrote: > > I don't think it's a big cost once all the processes > > have been forked if you're reusing them beyond perhaps slightly more > > efficient cache usage. > > Hm, this site claims t

Re: [HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Tom Lane
Marti Raudsepp writes: > There's a file descriptor leak in psql/command.c function > process_file() -- on errors it just returns without closing the file. > But since it's quitting anyway, there's no practical impact. Should I > submit a patch for this as well? Might as well. It's the kind of th

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 1:12 PM, Robert Haas wrote: > On Wed, Oct 20, 2010 at 3:40 PM, Greg Stark wrote: >> On Wed, Oct 20, 2010 at 6:29 AM, Robert Haas wrote: > >>> Actually, I think the best thing for default_statistics_target might >>> be to scale the target based on the number of rows in the

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-20 Thread Tom Lane
Greg Stark writes: > On Wed, Oct 20, 2010 at 8:37 AM, Tom Lane wrote: >> I think it's pointless to speculate about whether we might have divvied >> up the meta-information about tables differently if we'd foreseen >> wanting to do this.  It is what it is, and there is *way* too much code >> depen

Re: [HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Marti Raudsepp
On Wed, Oct 20, 2010 at 22:34, Heikki Linnakangas wrote: > Did coccicheck find anything else interesting? There's a file descriptor leak in psql/command.c function process_file() -- on errors it just returns without closing the file. But since it's quitting anyway, there's no practical impact. Sh

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:16 PM, David Fetter wrote: > On Tue, Oct 19, 2010 at 08:51:16PM -0400, Robert Haas wrote: >> On Tue, Oct 19, 2010 at 5:42 PM, Andrew Dunstan wrote: >> > Well a bit more testing shows some benefit. I've sorted out a few kinks, so >> > this seems to work. In particular, wi

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 8:37 AM, Tom Lane wrote: > I think it's pointless to speculate about whether we might have divvied > up the meta-information about tables differently if we'd foreseen > wanting to do this.  It is what it is, and there is *way* too much code > depending on it, both inside th

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:47 PM, daveg wrote: > On Wed, Oct 20, 2010 at 12:28:25PM -0700, Greg Stark wrote: >> On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark wrote: >> > I don't think it's a big cost once all the processes >> > have been forked if you're reusing them beyond perhaps slightly more >>

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:40 PM, Greg Stark wrote: > On Wed, Oct 20, 2010 at 6:29 AM, Robert Haas wrote: >> Exactly.  It doesn't take many 3-7% slowdowns to add up to being 50% >> or 100% slower, and that sucks.  In fact, I'm still not convinced that >> we were wise to boost default_statistics_ta

[HACKERS] pg_upgrade cleanup

2010-10-20 Thread Bruce Momjian
I have applied the attached patch to clean up pg_upgrade cache lookup code and remove useless NULL pointer tests. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + diff --git a/contrib

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 6:29 AM, Robert Haas wrote: > Exactly.  It doesn't take many 3-7% slowdowns to add up to being 50% > or 100% slower, and that sucks.  In fact, I'm still not convinced that > we were wise to boost default_statistics_target as much as we did.  I > argued for a smaller boost a

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Alvaro Herrera
Excerpts from Greg Stark's message of mié oct 20 16:28:25 -0300 2010: > On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark wrote: > > I don't think it's a big cost once all the processes > > have been forked if you're reusing them beyond perhaps slightly more > > efficient cache usage. > > Hm, this sit

Re: [HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 18:44, Marti Raudsepp wrote: Hi pgsql-hackers, Currently contrib/pgcrypto/pgp-pubenc.c contains code like: uint8 algo = pk->algo; if (pk == NULL) ... However, if pk was NULL, then the if() condition would never be reached because the pk->algo dereference would segfault. This pat

Re: [HACKERS] Review: Fix snapshot taking inconsistencies

2010-10-20 Thread Alvaro Herrera
Excerpts from Tom Lane's message of lun oct 04 10:31:26 -0400 2010: > In the particular case at hand here, I rather wonder why SQL functions > are depending on postgres.c at all. It might be better to just > duplicate a bit of code to make them independent. pg_parse_and_rewrite > would then be d

Re: [HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Bruce Momjian
Magnus Hagander wrote: > On Wed, Oct 20, 2010 at 21:28, Robert Haas wrote: > > On Wed, Oct 20, 2010 at 3:01 PM, Bruce Momjian wrote: > >> One more question --- should I be sending pg_upgrade patches to the list > >> for approval? ?The restructuring patch was large and didn't seem > >> necessary t

Re: [HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Magnus Hagander
On Wed, Oct 20, 2010 at 21:28, Robert Haas wrote: > On Wed, Oct 20, 2010 at 3:01 PM, Bruce Momjian wrote: >> One more question --- should I be sending pg_upgrade patches to the list >> for approval?  The restructuring patch was large and didn't seem >> necessary to post, and the speedups were tes

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 12:17 PM, Greg Stark wrote: > I don't think it's a big cost once all the processes > have been forked if you're reusing them beyond perhaps slightly more > efficient cache usage. Hm, this site claims to get a 13% win just from the reduced tlb misses using a preload hack wi

Re: [HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 3:01 PM, Bruce Momjian wrote: > One more question --- should I be sending pg_upgrade patches to the list > for approval?  The restructuring patch was large and didn't seem > necessary to post, and the speedups were tested by the bug reporter, so > I figured those were OK to

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Greg Stark
On Wed, Oct 20, 2010 at 7:10 AM, Tom Lane wrote: > I believe that for the equivalent Solaris option, we just automatically > enable it when available.  So there'd be no need for user documentation. > However, I definitely *would* like to see some benchmarks proving that > the change actually does

Re: [HACKERS] WIP: extensible enums

2010-10-20 Thread David Fetter
On Tue, Oct 19, 2010 at 08:51:16PM -0400, Robert Haas wrote: > On Tue, Oct 19, 2010 at 5:42 PM, Andrew Dunstan wrote: > > Well a bit more testing shows some benefit. I've sorted out a few kinks, so > > this seems to work. In particular, with the above tables, the version > > imported from 9.0 can

[HACKERS] pg_upgrade patch application process, and move to /bin?

2010-10-20 Thread Bruce Momjian
Bruce Momjian wrote: > I received a private email report yesterday from someone using > pg_upgrade with PG 9.0 who found it took five hours for pg_upgrade to > upgrade a database with 150k tables. Yes, that is a lot of tables, but > pg_upgrade should be able to do better than that. > > I have mod

[HACKERS] pg_upgrade performance with 150k tables

2010-10-20 Thread Bruce Momjian
I received a private email report yesterday from someone using pg_upgrade with PG 9.0 who found it took five hours for pg_upgrade to upgrade a database with 150k tables. Yes, that is a lot of tables, but pg_upgrade should be able to do better than that. I have modified pg_upgrade in git master to

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Aidan Van Dyk
On Wed, Oct 20, 2010 at 1:09 PM, Tom Lane wrote: > [ thinks for a bit... ]  Perhaps we could stick some sort of unique > ID into tables, which could be correlated to the same unique ID > appearing in a metadata fork. It would be awfully nice if the table name was in the first few bytes of the se

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Andrew Dunstan
On 10/20/2010 01:15 PM, Florian Weimer wrote: * Terry Laurenzo: Agreed. BSON was born out of implementations that either lacked arbitrary precision numbers or had a strong affinity to an int/floating point way of thinking about numbers. I believe that if BSON had an arbitrary precision numb

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-20 Thread Florian Weimer
* Terry Laurenzo: > Agreed. BSON was born out of implementations that either lacked > arbitrary precision numbers or had a strong affinity to an > int/floating point way of thinking about numbers. I believe that if > BSON had an arbitrary precision number type, it would be a proper > superset of

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Tom Lane
Aidan Van Dyk writes: > On Wed, Oct 20, 2010 at 10:28 AM, Tom Lane wrote: >> I can see the potential usefulness of a self-documenting table storage >> format, but this proposal isn't that; it's just an unreliable kluge. > If we're looking to have any sort of "out of catalog" documentation of > t

Re: [HACKERS] Simplifying replication

2010-10-20 Thread Robert Treat
On Tue, Oct 19, 2010 at 11:16 AM, Greg Smith wrote: > Josh Berkus wrote: > >> It is critical that we make replication easier to set up, administrate and >> monitor than it currently is. In my conversations with people, this is more >> important to our users and the adoption of PostgreSQL than sy

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Aidan Van Dyk
On Wed, Oct 20, 2010 at 10:28 AM, Tom Lane wrote: > "Stephen R. van den Berg" writes: >> It's just that matching table and file, and subsequently figuring out >> some missing columns which may have been added/removed later, >> can be rather timeconsuming and could be made a lot easier (not necess

[HACKERS] [PATCH] pgcrypto: Test for NULL before dereferencing pointer

2010-10-20 Thread Marti Raudsepp
Hi pgsql-hackers, Currently contrib/pgcrypto/pgp-pubenc.c contains code like: uint8 algo = pk->algo; if (pk == NULL) ... However, if pk was NULL, then the if() condition would never be reached because the pk->algo dereference would segfault. This patch moves the dereference to below the conditi

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-20 Thread Tom Lane
Greg Stark writes: > On Tue, Oct 19, 2010 at 4:12 PM, Tom Lane wrote: >> I think they're more alike than you think.  If we had the infrastructure >> to do local temp tables this way, it'd be pretty easy to use that to >> instantiate per-backend copies of global temp tables.  (The global >> entiti

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane writes: > That is simply a horrid idea. Just make it specify EXTENSION. Black magic it is, will remove in v7. > Is there any sane use-case for the control file to not be named the same > as the extension? It seems like that would accomplish little except to > sow confusion. The goal

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> I don't think that "no changes to the makefiles" is a requirement, >> or even a wish-list item, for this. I think it's perfectly reasonable >> for the makefile to have to specify the module name; far better that >> than that we get the name by some

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane writes: >> and use the equivalent of SET LOCAL in the CREATE EXTENSION code? > > I had assumed that that was how he was doing it ... I'm currently doing: SetConfigOption("client_min_messages", "warning", PGC_SUSET, PGC_S_SESSION); And then manually reverting to what was there b

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 18:06, Tatsuo Ishii wrote: This returns 'false' if you're in hot standby mode running against an archive. That seems wrong, I don't think the walreceiver state should play any role in this. Apart this, I wonder why walsender/walreceiver do not transfer archive logs as well. What

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane writes: > I don't think that "no changes to the makefiles" is a requirement, > or even a wish-list item, for this. I think it's perfectly reasonable > for the makefile to have to specify the module name; far better that > than that we get the name by some "magic" or other. It seemed eas

Re: [HACKERS] Serializable snapshot isolation patch

2010-10-20 Thread Kevin Grittner
Robert Haas wrote: > On Tue, Oct 19, 2010 at 6:28 PM, Kevin Grittner > wrote: >> One thing that would work, but I really don't think I like it, is >> that a request for a snapshot for such a transaction would not >> only block until it could get a "clean" snapshot (no overlapping >> serializable

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Tatsuo Ishii
> This returns 'false' if you're in hot standby mode running against an > archive. That seems wrong, I don't think the walreceiver state should > play any role in this. Apart this, I wonder why walsender/walreceiver do not transfer archive logs as well. -- Tatsuo Ishii SRA OSS, Inc. Japan English:

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:53 AM, Alvaro Herrera wrote: > Excerpts from Robert Haas's message of mié oct 20 10:29:04 -0300 2010: > >> Actually, I think the best thing for default_statistics_target might >> be to scale the target based on the number of rows in the table, e.g. >> given N rows: >> >>

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Alvaro Herrera writes: > Excerpts from Dimitri Fontaine's message of mié oct 20 07:22:53 -0300 2010: >> Using SPI to execute the extension's script already means that it can >> not contain explicit BEGIN and COMMIT commands. Now, is it possible to >> force a Reset of all GUCs after script's execu

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié oct 20 10:29:04 -0300 2010: > Actually, I think the best thing for default_statistics_target might > be to scale the target based on the number of rows in the table, e.g. > given N rows: > > 10 + (N / 1000), if N < 40,000 > 46 + (N / 1), if 50,000 <

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:35 AM, Heikki Linnakangas wrote: > On 20.10.2010 17:32, Tatsuo Ishii wrote: pg_is_in_recovery() returns a bool, are you proposing to change that? >>> >>> No. I just thought about adding more condition when it returns true. >> >> Here is the patch. Comments are

Re: [HACKERS] leaky views, yet again

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:00 AM, Tom Lane wrote: > Robert Haas writes: >> I get the impression that you think that there's a problem not only >> with the approach but with any approach whatsoever to that underlying >> problem. > > Let's just say that the approaches proposed so far have performan

Re: [HACKERS] Domains versus arrays versus typmods

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 10:03 AM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Oct 19, 2010 at 9:17 PM, Tom Lane wrote: >>> We've already accepted the cost of doing getBaseTypeAndTypmod() in a >>> whole lot of performance-critical parsing paths, on the off chance that >>> the target datatype

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> If the extensions manager is dependent on the assumption that a module's >> name matches the name of the directory it's built in > It is not. There's some magic for simple cases so that contrib mostly > "works" with no editing, but of course, that's

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Alvaro Herrera
Excerpts from Dimitri Fontaine's message of mié oct 20 07:22:53 -0300 2010: > Itagaki Takahiro writes: > > CREATE EXTENSION command > > * Environment could be modified by the installer script. > > =# SHOW search_path; => "$user",public > > =# CREATE EXTENSION dblink; > > =# SHOW search_

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: >"Stephen R. van den Berg" writes: >> It's just that matching table and file, and subsequently figuring out >> some missing columns which may have been added/removed later, >> can be rather timeconsuming and could be made a lot easier (not necessarily >> perfect) if that informatio

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Tom Lane
Heikki Linnakangas writes: > Going from wal_level='minimal' to 'archivë́' incurs the penalty on > WAL-logging COPY etc. That's a big penalty. However, the difference > between wal_level='archive' and wal_level='hot_standby' should be tiny. I'm not sure I believe that either, because of the cos

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 17:32, Tatsuo Ishii wrote: pg_is_in_recovery() returns a bool, are you proposing to change that? No. I just thought about adding more condition when it returns true. Here is the patch. Comments are welcome! ... Datum pg_is_in_recovery(PG_FUNCTION_ARGS) { ! /* use

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Heikki Linnakangas
On 20.10.2010 17:19, Tom Lane wrote: Greg Smith writes: Josh Berkus wrote: Well, now that you mention it, I also think that "hot standby" should be the default. Yes, I know about the overhead, but I also think that the number of our users who want easy replication *far* outnumber the users wh

Re: [HACKERS] PostgreSQL and HugePage

2010-10-20 Thread Kenneth Marshall
On Wed, Oct 20, 2010 at 10:10:00AM -0400, Tom Lane wrote: > Robert Haas writes: > > On Tue, Oct 19, 2010 at 11:30 PM, daveg wrote: > >> On Wed, Oct 20, 2010 at 04:08:37PM +1300, Mark Kirkwood wrote: > >>> Heh - provided you specify > >>> SHM_HUGETLB > >>> in the relevant call that is :-) > > >>

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 20, 2010 at 6:22 AM, Dimitri Fontaine > wrote: >> In v6 patch, should client_min_messages or log_min_messages be lower >> than WARNING, they get set to WARNING for the script install context. We >> still dump the extension's script at each WARNING, but you can se

Re: [HACKERS] How to reliably detect if it's a promoting standby

2010-10-20 Thread Tatsuo Ishii
>> pg_is_in_recovery() returns a bool, are you proposing to change that? > > No. I just thought about adding more condition when it returns true. Here is the patch. Comments are welcome! *** a/src/backend/access/transam/xlog.c --- b/src/backend/access/transam/xlog.c *** *** 5604,5610

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Tom Lane
"Stephen R. van den Berg" writes: > It's just that matching table and file, and subsequently figuring out > some missing columns which may have been added/removed later, > can be rather timeconsuming and could be made a lot easier (not necessarily > perfect) if that information would have been pre

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Dimitri Fontaine
Tom Lane writes: > If the extensions manager is dependent on the assumption that a module's > name matches the name of the directory it's built in It is not. There's some magic for simple cases so that contrib mostly "works" with no editing, but of course, that's only mostly. The version Itakagi

Re: [HACKERS] max_wal_senders must die

2010-10-20 Thread Tom Lane
Greg Smith writes: > Josh Berkus wrote: >> Well, now that you mention it, I also think that "hot standby" should be >> the default. Yes, I know about the overhead, but I also think that the >> number of our users who want easy replication *far* outnumber the users >> who care about an extra 10% W

Re: [HACKERS] Extensions, this time with a patch

2010-10-20 Thread Tom Lane
Itagaki Takahiro writes: > On Wed, Oct 20, 2010 at 12:58 PM, Alvaro Herrera > wrote: >> Lets rename the directory. > Hmmm, but we call it 'xml2' in the doc. There is no 'pgxml' at all in it. > http://developer.postgresql.org/pgdocs/postgres/xml2.html > However, I don't think we can change the m

  1   2   >