[HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-02-09 Thread Christoph Berg
Hi, we have a database that is storing strings in various encodings (and non-encodings, namely the arbitrary byte soup that you might see in email headers from the internet). For this reason, the database uses sql_ascii encoding. The columns are text, as most characters are ascii, so bytea didn't

Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work outside WALInsertLock)

2012-02-09 Thread Fujii Masao
On Thu, Feb 9, 2012 at 3:32 AM, Jeff Janes jeff.ja...@gmail.com wrote: On Wed, Feb 1, 2012 at 11:46 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 31.01.2012 17:35, Fujii Masao wrote: On Fri, Jan 20, 2012 at 11:11 PM, Heikki Linnakangas

Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work outside WALInsertLock)

2012-02-09 Thread Fujii Masao
On Thu, Feb 9, 2012 at 7:25 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Feb 9, 2012 at 3:32 AM, Jeff Janes jeff.ja...@gmail.com wrote: On Wed, Feb 1, 2012 at 11:46 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 31.01.2012 17:35, Fujii Masao wrote: On Fri, Jan

Re: [HACKERS] Add protransform for numeric, varbit, and temporal types

2012-02-09 Thread Noah Misch
On Wed, Feb 08, 2012 at 09:37:01AM -0500, Robert Haas wrote: On Tue, Feb 7, 2012 at 12:43 PM, Robert Haas robertmh...@gmail.com wrote: I've committed the numeric and varbit patches and will look at the temporal one next. Committed, after changing the OIDs so they don't conflict. Here's

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Noah Misch
On Tue, Feb 07, 2012 at 09:38:39PM -0500, Robert Haas wrote: Second, there's a concern about binary bloat: duplicating lots of code with different comparators inlined generates, well, a lot of machine code. Of course, an 0.8% increase in the size of the resulting binary is very unlikely to

Re: pg_receivexlog and sync rep Re: [HACKERS] Updated version of pg_receivexlog

2012-02-09 Thread Magnus Hagander
On Wed, Feb 8, 2012 at 19:39, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Oct 25, 2011 at 7:37 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Oct 24, 2011 at 14:40, Magnus Hagander mag...@hagander.net wrote: On Mon, Oct 24, 2011 at 13:46, Heikki Linnakangas

Re: [HACKERS] [COMMITTERS] pgsql: Add new keywords SNAPSHOT and TYPES to the keyword list in gram.

2012-02-09 Thread Alvaro Herrera
Excerpts from Heikki Linnakangas's message of jue feb 09 06:42:12 -0300 2012: Add new keywords SNAPSHOT and TYPES to the keyword list in gram.y These were added to kwlist.h as unreserved keywords in separate patches, but authors forgot to add them to the corresponding list in gram.y.

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Robert Haas
On Thu, Feb 9, 2012 at 7:24 AM, Noah Misch n...@leadboat.com wrote: On Tue, Feb 07, 2012 at 09:38:39PM -0500, Robert Haas wrote: Second, there's a concern about binary bloat: duplicating lots of code with different comparators inlined generates, well, a lot of machine code.  Of course, an 0.8%

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Peter Geoghegan
On 9 February 2012 13:50, Robert Haas robertmh...@gmail.com wrote: I'm also more than slightly concerned that we are losing sight of the forest for the trees.  I have heard reports that sorting large amounts of data is many TIMES slower for us than for a certain other database product.  I

Re: [HACKERS] Vacuum rate limit in KBps

2012-02-09 Thread Bruce Momjian
On Wed, Feb 08, 2012 at 10:41:43PM -0500, Greg Smith wrote: Just trying to set the expectations bar realistically here. I don't consider the easier problem of checkpoint smoothing a solved one, either. Saying autovacuum needs to reach even that level of automation to be a useful improvement

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Robert Haas
On Thu, Feb 9, 2012 at 9:37 AM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 9 February 2012 13:50, Robert Haas robertmh...@gmail.com wrote: I'm also more than slightly concerned that we are losing sight of the forest for the trees.  I have heard reports that sorting large amounts of data

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Bruce Momjian
On Thu, Feb 09, 2012 at 07:24:49AM -0500, Noah Misch wrote: This patch has gotten more than its fair share of attention for bloat, and I think that's mostly because there's a dial-a-bloat-level knob sticking out and begging to be frobbed. I already emailed Peter privately stating that he

Re: [HACKERS] 16-bit page checksums for 9.2

2012-02-09 Thread Simon Riggs
On Wed, Feb 8, 2012 at 2:05 PM, Noah Misch n...@leadboat.com wrote: On Wed, Feb 08, 2012 at 11:40:34AM +, Simon Riggs wrote: On Wed, Feb 8, 2012 at 3:24 AM, Noah Misch n...@leadboat.com wrote: On Tue, Feb 07, 2012 at 08:58:59PM +, Simon Riggs wrote: On Thu, Jan 26, 2012 at 8:20 PM,

Re: [HACKERS] [COMMITTERS] pgsql: Add new keywords SNAPSHOT and TYPES to the keyword list in gram.

2012-02-09 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Heikki Linnakangas's message of jue feb 09 06:42:12 -0300 2012: src/tools/check_keywords.pl is your friend. I wondered a couple of weeks ago if we could, instead, generate the lists in gram.y from kwlist.h. We've looked into

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Peter Geoghegan
On 9 February 2012 14:51, Robert Haas robertmh...@gmail.com wrote: I'm not sure I entirely follow all this, but I'll look at the code once you have it.  Are you saying that all the comparetup_yadda functions are redundant to each other in the single-key case? Yes, I am. The main reason that

[HACKERS] Notify system doesn't recover from No space error

2012-02-09 Thread Kevin Grittner
On a development system which wasn't being monitored very closely I think we've uncovered a bug in the listen/notify behavior. The system was struggling with lack of disk space for about a week before this surfaced, so it's probably not the most critical sort of bug we could have, but it probably

[HACKERS] index-only quals vs. security_barrier views

2012-02-09 Thread Robert Haas
When Heikki worked up his original index-only scan patches (which didn't end up looking much like what eventually got committed), he had the notion of an index-only qual. That is, given a query like this: select sum(1) from foo where substring(a,1,3) = 'abc'; We could evaluate the substring

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Bruce Momjian
On Thu, Feb 09, 2012 at 03:36:23PM +, Peter Geoghegan wrote: On 9 February 2012 14:51, Robert Haas robertmh...@gmail.com wrote: I'm not sure I entirely follow all this, but I'll look at the code once you have it.  Are you saying that all the comparetup_yadda functions are redundant to

Re: [HACKERS] Add protransform for numeric, varbit, and temporal types

2012-02-09 Thread Robert Haas
On Thu, Feb 9, 2012 at 7:18 AM, Noah Misch n...@leadboat.com wrote: The code has a thinko; a given interval typmod ultimately implies a single point from which we truncate rightward.  The precision only matters if the range covers SECOND.  Thanks; the attached patch improves this. Thanks,

Re: [HACKERS] Progress on fast path sorting, btree index creation time

2012-02-09 Thread Peter Geoghegan
On 9 February 2012 17:16, Bruce Momjian br...@momjian.us wrote: Yes, I am. The main reason that the loops exist in those functions (which is the only way that they substantially differ) is because they each have to get the other keys through various ways that characterise the tuple class that

Re: [HACKERS] Notify system doesn't recover from No space error

2012-02-09 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: We got three errors in the log with exactly this DETAIL (file, offset and message): [2012-02-05 01:27:36.878 CST] 14892 cc cc 127.0.0.1(35320) ERROR: could not access status of transaction 0 [2012-02-05 01:27:36.878 CST] 14892 cc cc

Re: [HACKERS] Notify system doesn't recover from No space error

2012-02-09 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: [2012-02-05 01:30:36.952 CST] 16383 cc cc 127.0.0.1(38931) ERROR: could not access status of transaction 0 [2012-02-05 01:30:36.952 CST] 16383 cc cc 127.0.0.1(38931) DETAIL: Could not read from file

Re: [HACKERS] Notify system doesn't recover from No space error

2012-02-09 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: The application LISTENs on channel tcn and a trigger function is attached to most permanent tables to NOTIFY for DML on that channel. ... The report to us was that testers were unable to start the application. I believe that the above error

Re: [HACKERS] [COMMITTERS] pgsql: Add new keywords SNAPSHOT and TYPES to the keyword list in gram.

2012-02-09 Thread Alvaro Herrera
Excerpts from Tom Lane's message of jue feb 09 12:17:59 -0300 2012: Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Heikki Linnakangas's message of jue feb 09 06:42:12 -0300 2012: src/tools/check_keywords.pl is your friend. I wondered a couple of weeks ago if we

Re: [HACKERS] [COMMITTERS] pgsql: Add new keywords SNAPSHOT and TYPES to the keyword list in gram.

2012-02-09 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Tom Lane's message of jue feb 09 12:17:59 -0300 2012: If people are sufficiently worried about this, a better answer would be to teach the makefiles to run check_keywords.pl during every build. FWIW that script is throwing a

Re: [HACKERS] index-only quals vs. security_barrier views

2012-02-09 Thread Jesper Krogh
On 2012-02-09 18:02, Robert Haas wrote: I don't have any appetite for trying to do anything more with index-only scans for 9.2, though maybe someone else will think otherwise. But I would like very much to get KaiGai's leakproof stuff committed, and so it seems like a good idea to reconcile the

Re: [HACKERS] Notify system doesn't recover from No space error

2012-02-09 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: The application LISTENs on channel tcn and a trigger function is attached to most permanent tables to NOTIFY for DML on that channel. ... The report to us was that testers were unable to start the

Re: [HACKERS] Notify system doesn't recover from No space error

2012-02-09 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: In continued digging through logs I found something to indicate the transaction on which COMMIT failed for 2528 of the failures. In all cases the transaction made a change which would have fired a NOTIFY at commit (in a deferred trigger).

Re: [HACKERS] When do we lose column names?

2012-02-09 Thread Andrew Dunstan
On 02/07/2012 03:03 PM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: On 02/07/2012 12:47 PM, Tom Lane wrote: In general I think we'd have to require that colnames be supplied in all RowExprs if we go this way. Anyplace that's trying to slide by without will have to be fixed.

Re: [HACKERS] When do we lose column names?

2012-02-09 Thread Alvaro Herrera
Excerpts from Andrew Dunstan's message of jue feb 09 16:38:27 -0300 2012: OK, the one place that needs to be fixed to avoid the crash caused by the json regression tests with the original patch is in src/backend/parser/parse_expr.c:transformRowExpr(). Other candidates I have found

Re: [HACKERS] When do we lose column names?

2012-02-09 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: OK, the one place that needs to be fixed to avoid the crash caused by the json regression tests with the original patch is in src/backend/parser/parse_expr.c:transformRowExpr(). Other candidates I have found that don't set colnames and should

Re: [HACKERS] When do we lose column names?

2012-02-09 Thread Andrew Dunstan
On 02/09/2012 02:54 PM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: OK, the one place that needs to be fixed to avoid the crash caused by the json regression tests with the original patch is in src/backend/parser/parse_expr.c:transformRowExpr(). Other candidates I have

Re: [HACKERS] index-only quals vs. security_barrier views

2012-02-09 Thread Robert Haas
On Thu, Feb 9, 2012 at 1:33 PM, Jesper Krogh jes...@krogh.cc wrote: On 2012-02-09 18:02, Robert Haas wrote: I don't have any appetite for trying to do anything more with index-only scans for 9.2, though maybe someone else will think otherwise.  But I would like very much to get KaiGai's

Re: [HACKERS] Bugs/slowness inserting and indexing cubes

2012-02-09 Thread Jay Levitt
Tom Lane wrote: Jay Levittjay.lev...@gmail.com writes: [Posted at Andres's request] TL;DR: Inserting and indexing cubes is slow and/or broken in various ways in various builds. 1. In 9.1.2, inserting 10x rows takes 19x the time. - 9.1-HEAD and 9.2 fix this; it now slows down linearly

[HACKERS] psql tab completion for SELECT

2012-02-09 Thread Peter Eisentraut
In his blog entry http://www.depesz.com/2011/07/08/wish-list-for-psql/ depesz described a simple way to do tab completion for SELECT in psql: Make tab-completion complete also function names – like: SELECT pg_gettabtab to see all functions that start with pg_get. Make tab-completion work for

[HACKERS] RFC: Making TRUNCATE more MVCC-safe

2012-02-09 Thread Marti Raudsepp
Hi! I've always been a little wary of using the TRUNCATE command due to the warning in the documentation about it not being MVCC-safe: queries may silently give wrong results and it's hard to tell when they are affected. That got me thinking, why can't we handle this like a standby server does

Re: [HACKERS] index-only quals vs. security_barrier views

2012-02-09 Thread Jesper Krogh
On 2012-02-09 21:09, Robert Haas wrote: That doesn't make sense to me. If you probe index A for rows where a = 1 and find that CTID (100,1) is such a row, and now want to return a column value b that is not present in that index, the fastest way to get the row is going to be to fetch block 100

Re: [HACKERS] psql tab completion for SELECT

2012-02-09 Thread Dimitri Fontaine
Peter Eisentraut pete...@gmx.net writes: Make tab-completion complete also function names – like: SELECT pg_gettabtab to see all functions that start with pg_get. Make tab-completion work for columns in SELECT. I know that when writing SELECT clause, psql doesn’t know which table it will deal

Re: [HACKERS] xlog location arithmetic

2012-02-09 Thread Euler Taveira de Oliveira
On 08-02-2012 09:35, Fujii Masao wrote: Fujii, new patch attached. Thanks for your tests. But another problem happened. When I changed pg_proc.h so that the unused OID was assigned to pg_xlog_location_diff(), and executed the above again, I encountered the segmentation fault: I reproduced

[HACKERS] Patch: fix pg_dump for inherited defaults not-null flags

2012-02-09 Thread Tom Lane
Attached is a proposed patch to deal with the issue described here: http://archives.postgresql.org/pgsql-bugs/2012-02/msg0.php Even though we'd previously realized that comparing the text of inherited CHECK expressions is an entirely unsafe way to detect expression equivalence (cf comments

Re: [HACKERS] psql tab completion for SELECT

2012-02-09 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: That seems pretty useful, and it's more or less a one-line change, as in the attached patch. That seems pretty nearly entirely bogus. What is the argument for supposing that the word right after SELECT is a function name? I would think it would be a