Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Tomas Vondra
On 06/19/2015 09:48 PM, Jeff Janes wrote: On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: But I think you might be on to something, because I manually collected a random sample with 30k rows (by explicitly generating

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Sat, Jun 20, 2015 at 12:44 AM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@anarazel.de writes: On 2015-06-19 11:16:18 -0400, Robert Haas wrote: On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wonder whether it's such a good idea for the postmaster to give

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-19 Thread Brendan Jurd
On Fri, 19 Jun 2015 at 21:05 Petr Jelinek p...@2ndquadrant.com wrote: On 2015-06-19 09:08, Brendan Jurd wrote: I think it would be convenient and user-friendly to complete the opening bracket -- it would make it perfectly clear that an argument is required for the syntax to be valid.

[HACKERS] Insufficient locking for ALTER DEFAULT PRIVILEGES

2015-06-19 Thread Vik Fearing
I came across the following bug this week: Session 0: begin; create schema bug; alter default privileges in schema bug grant all on tables to postgres; commit; Session 1: begin; alter default privileges in schema bug grant all on tables to postgres; Session 2: alter default privileges in schema

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Fri, Jun 19, 2015 at 11:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Michael Paquier michael.paqu...@gmail.com writes: Attached is a patch fixing those problems and improving the log facility as it really helped me out with those issues. The simplest fix would be to include the -w switch

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Sat, Jun 20, 2015 at 12:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: Michael Paquier michael.paqu...@gmail.com writes: Now if we look at RewindTest.pm, there is the following code: if ($test_master_datadir) { system pg_ctl -D

Re: [HACKERS] Need Multixact Freezing Docs

2015-06-19 Thread Alvaro Herrera
Jim Nasby wrote: On 6/14/15 9:50 AM, Alvaro Herrera wrote: +values[0] = MultiXactState-oldestMultiXactId; What about oldestOffset and offsetStopLimit? Seems those would be useful too. Looks good other than that. Yeah, that's what I was trying to say. How about this? I realized that

[HACKERS] castoroides spinlock failure on test_shm_mq

2015-06-19 Thread Alvaro Herrera
Has anybody noticed the way castoroides is randomly failing? SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,27)), 200, 3); ! PANIC: stuck spinlock (100cb92f4) detected at atomics.c:30 ! server closed the connection

Re: [HACKERS] anole: assorted stability problems

2015-06-19 Thread Alvaro Herrera
Alvaro Herrera wrote: Tom Lane wrote: Andres Freund and...@anarazel.de writes: Uh. I'm pretty sure there were some back when that patch went in. And there definitely used to be a couple earlier. I guess itanium really is dying (mixed bad: It's a horrible architecture, but more

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Jeff Janes
On Fri, Jun 19, 2015 at 1:39 PM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: On 06/19/2015 09:48 PM, Jeff Janes wrote: On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: But I think you might be on to something,

[HACKERS] pg_regress not waiting for postmaster to stop

2015-06-19 Thread Michael Paquier
Hi all, In pg_regress.c, there is the following code: static void stop_postmaster(void) { if (postmaster_running) { /* We use pg_ctl to issue the kill and wait for stop */ charbuf[MAXPGPATH * 2]; int

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-19 Thread Brendan Jurd
On Sun, 14 Jun 2015 at 20:44 Petr Jelinek p...@2ndquadrant.com wrote: looks like I omitted psql tab completion from the TABLESAMPLE patch. The attached patch adds it. Hi Petr, I'm doing an initial review of this patch. It applies and compiles cleanly. Code style is consistent with its

Re: [HACKERS] Is it possible to have a fast-write Index?

2015-06-19 Thread Simon Riggs
On 19 June 2015 at 14:30, Robert Haas robertmh...@gmail.com wrote: So I really doubt that anyone would have any enthusiasm for saddling btree with a similar mechanism. It's complicated (and has been the cause of multiple bugs); it's hard to figure out when is the optimal time to flush

Re: [HACKERS] Time to get rid of PQnoPasswordSupplied?

2015-06-19 Thread Robert Haas
On Sun, Jun 14, 2015 at 11:34 PM, Craig Ringer cr...@2ndquadrant.com wrote: I frequently see users confused by one of our more common and less clear error messages: fe_sendauth: no password supplied I've never seen this error message, but I'm not opposed to improving it in some way. --

Re: [HACKERS] Is it possible to have a fast-write Index?

2015-06-19 Thread Merlin Moncure
On Thu, Jun 11, 2015 at 9:32 PM, Qingqing Zhou zhouqq.postg...@gmail.com wrote: On Fri, Jun 5, 2015 at 10:59 AM, Tom Lane t...@sss.pgh.pa.us wrote: So I really doubt that anyone would have any enthusiasm for saddling btree with a similar mechanism. It's complicated (and has been the cause of

Re: [HACKERS] GIN function of pageinspect has inconsistency data type.

2015-06-19 Thread Sawada Masahiko
On Wed, Jun 17, 2015 at 4:11 PM, Jim Nasby jim.na...@bluetreble.com wrote: On 6/16/15 8:26 AM, Sawada Masahiko wrote: I noticed while using gin function of pageinspect that there are some inconsistency data types. For example, data type of GinMetaPageData.head, and tail is BlockNumber, i.g,

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier michael.paqu...@gmail.com wrote: Listing the directories with pg_ls_dir() has the same problem. (After some discussion on IM with Heikki on this one). This is actually

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier michael.paqu...@gmail.com wrote: Listing the directories with pg_ls_dir() has the same problem. (After some discussion on IM with Heikki on this one). This is actually more tricky because pg_ls_dir() does not return '.' or '..' that we could

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-19 Thread Petr Jelinek
On 2015-06-19 09:08, Brendan Jurd wrote: On Sun, 14 Jun 2015 at 20:44 Petr Jelinek p...@2ndquadrant.com mailto:p...@2ndquadrant.com wrote: looks like I omitted psql tab completion from the TABLESAMPLE patch. The attached patch adds it. Hi Petr, I'm doing an initial review of this

Re: [HACKERS] [Proposal] Progress bar for pg_dump/pg_restore

2015-06-19 Thread Taiki Kondo
Hi, andres Thank you for your comment, and sorry for late response. The question is how to actually get useful estimates. As there's no progress report for indvidiual COPY and CREATE INDEX commands you'll, in many cases, have very irregular progress updates. In many many cases most of the

Re: [HACKERS] Is it possible to have a fast-write Index?

2015-06-19 Thread Robert Haas
So I really doubt that anyone would have any enthusiasm for saddling btree with a similar mechanism. It's complicated (and has been the cause of multiple bugs); it's hard to figure out when is the optimal time to flush the pending insertions; and it slows down searches in favor of making

Re: [HACKERS] Tab completion for CREATE SEQUENCE

2015-06-19 Thread Brendan Jurd
On Tue, 16 Jun 2015 at 00:52 Vik Fearing v...@2ndquadrant.fr wrote: While reviewing the seqam patches, I noticed that psql has tab completion for ALTER SEQUENCE, but not for CREATE SEQUENCE. The attached trivial patch fixes that. Hi Vik, I'm doing an initial review of this patch. It

[HACKERS] Missing tab-complete for PASSWORD word in CREATE ROLE syntax

2015-06-19 Thread Jeevan Chalke
Hi, I have observed that we are not tab-completing word PASSWORD in the following syntaxes: 1. CREATE|ALTER ROLE|USER rolname 2. CREATE|ALTER ROLE|USER rolname WITH PASSWORD is used many times and should be in the tab-complete list. Was there any reason we have deliberately kept this out? If

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Thu, Jun 18, 2015 at 3:52 PM, Michael Paquier wrote: I think that it would be useful as well to improve the buildfarm output. Thoughts? And after running the tests more or less 6~7 times in a row on a PI, I have been able to trigger the problem and I think that I have found its origin.

[HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Andres Freund
Hi, To make slot usage in pg_receivexlog easier, should we add --create-slot-if-not-exists? That'd mean you could run the same command the first and later invocation. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] checkpointer continuous flushing

2015-06-19 Thread Andres Freund
Hi, On 2015-06-17 08:24:38 +0200, Fabien COELHO wrote: Here is version 3, including many performance tests with various settings, representing about 100 hours of pgbench run. This patch aims at improving checkpoint I/O behavior so that tps throughput is improved, late transactions are less

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Michael Paquier michael.paqu...@gmail.com writes: Attached is a patch fixing those problems and improving the log facility as it really helped me out with those issues. The simplest fix would be to include the -w switch missing in the tests of pg_rewind and pg_ctl though. I agree with adding

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Stephen Frost
* Andres Freund (and...@anarazel.de) wrote: To make slot usage in pg_receivexlog easier, should we add --create-slot-if-not-exists? That'd mean you could run the same command the first and later invocation. Yes, please. Thanks! Stephen signature.asc Description:

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Andres Freund
On 2015-06-19 11:19:23 -0400, Magnus Hagander wrote: On Fri, Jun 19, 2015 at 11:17 AM, Stephen Frost sfr...@snowman.net wrote: * Andres Freund (and...@anarazel.de) wrote: To make slot usage in pg_receivexlog easier, should we add --create-slot-if-not-exists? That'd mean you could run the

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Cédric Villemain
To make slot usage in pg_receivexlog easier, should we add --create-slot-if-not-exists? That'd mean you could run the same command the first and later invocation. +1 (with a shorter name please, if you can find one... ) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/

Re: [HACKERS] Inheritance planner CPU and memory usage change since 9.3.2

2015-06-19 Thread Tom Lane
Petr Jelinek p...@2ndquadrant.com writes: On 2015-06-19 01:04, Petr Jelinek wrote: On 2015-06-19 00:38, Petr Jelinek wrote: On 2015-06-18 22:04, Tom Lane wrote: By the by, the tablesample additions to range_table_mutator are obviously broken. Apparently it's not a good idea to do this at

Re: [HACKERS] pg_regress not waiting for postmaster to stop

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 2:33 AM, Michael Paquier michael.paqu...@gmail.com wrote: In pg_regress.c, there is the following code: static void stop_postmaster(void) { if (postmaster_running) { /* We use pg_ctl to issue the kill and wait for stop */

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wonder whether it's such a good idea for the postmaster to give up waiting before all children are gone (postmaster.c:1722 in HEAD). I doubt it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Magnus Hagander
On Fri, Jun 19, 2015 at 11:17 AM, Stephen Frost sfr...@snowman.net wrote: * Andres Freund (and...@anarazel.de) wrote: To make slot usage in pg_receivexlog easier, should we add --create-slot-if-not-exists? That'd mean you could run the same command the first and later invocation. Yes,

Re: [HACKERS] Time to get rid of PQnoPasswordSupplied?

2015-06-19 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Jun 14, 2015 at 11:34 PM, Craig Ringer cr...@2ndquadrant.com wrote: I frequently see users confused by one of our more common and less clear error messages: fe_sendauth: no password supplied I've never seen this error message, but I'm not

Re: [HACKERS] Missing tab-complete for PASSWORD word in CREATE ROLE syntax

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 5:52 AM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: I have observed that we are not tab-completing word PASSWORD in the following syntaxes: 1. CREATE|ALTER ROLE|USER rolname 2. CREATE|ALTER ROLE|USER rolname WITH PASSWORD is used many times and should be

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Andres Freund
On 2015-06-19 11:16:18 -0400, Robert Haas wrote: On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wonder whether it's such a good idea for the postmaster to give up waiting before all children are gone (postmaster.c:1722 in HEAD). I doubt it. Seconded. It's pretty

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Andres Freund and...@anarazel.de writes: On 2015-06-19 11:16:18 -0400, Robert Haas wrote: On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wonder whether it's such a good idea for the postmaster to give up waiting before all children are gone (postmaster.c:1722 in HEAD).

[HACKERS] outstanding multixact bugs

2015-06-19 Thread Robert Haas
During the Developer Unconference at PGCon, we made this wiki page to track the MultiXact bugs that are outstanding: https://wiki.postgresql.org/wiki/MultiXact_Bugs Let's update this page as we find or fix things, and let's release when we've fixed enough stuff. Even if you are not involved in

Re: [HACKERS] Auto-vacuum is not running in 9.1.12

2015-06-19 Thread Alvaro Herrera
Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Yeah, the case is pretty weird and I'm not really sure that the server ought to be expected to behave. But if this is actually the only part of the server that misbehaves because of sudden gigantic time jumps, I think it's

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Cédric Villemain
Le 19/06/2015 17:21, Andres Freund a écrit : On 2015-06-19 11:19:23 -0400, Magnus Hagander wrote: On Fri, Jun 19, 2015 at 11:17 AM, Stephen Frost sfr...@snowman.net wrote: * Andres Freund (and...@anarazel.de) wrote: To make slot usage in pg_receivexlog easier, should we add

[HACKERS] backend depends on libpgport but Make doesn't know

2015-06-19 Thread Alvaro Herrera
I think we're missing a dependency declaration from the backend to libpgport in the makefiles somewhere. I just git updated on 9.1 on a tree that was already built, ran make, and got this: ../../src/port/libpgport_srv.a(dirmod_srv.o): In function `rmtree':

[HACKERS] [PATCH] pg_upgrade fails when postgres/template1 isn't in default tablespace

2015-06-19 Thread Marti Raudsepp
Hi list One of my databases failed to upgrade successfully and produced this error in the copying phase: error while copying relation pg_catalog.pg_largeobject (/srv/ssd/PG_9.3_201306121/1/12023 to /PG_9.4_201409291/1/12130): No such file or directory Turns out this happens when either the

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Alvaro Herrera
Tom Lane wrote: Andres Freund and...@anarazel.de writes: On 2015-06-19 11:16:18 -0400, Robert Haas wrote: On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wonder whether it's such a good idea for the postmaster to give up waiting before all children are gone

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Andres Freund
On 2015-06-19 13:56:21 -0300, Alvaro Herrera wrote: We discussed this when that patch got in (82233ce7ea42d6b). The reason for not waiting, it was argued, is that the most likely reason for those processes not to have already gone away by the time we send SIGKILL was that they are stuck

Re: [HACKERS] Auto-vacuum is not running in 9.1.12

2015-06-19 Thread Andres Freund
On 2015-06-17 18:10:42 -0300, Alvaro Herrera wrote: Yeah, the case is pretty weird and I'm not really sure that the server ought to be expected to behave. But if this is actually the only part of the server that misbehaves because of sudden gigantic time jumps, I think it's fair to patch it.

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Andres Freund and...@anarazel.de writes: On 2015-06-19 13:56:21 -0300, Alvaro Herrera wrote: We discussed this when that patch got in (82233ce7ea42d6b). The reason for not waiting, it was argued, is that the most likely reason for those processes not to have already gone away by the time we

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Alvaro Herrera
Alvaro Herrera wrote: We discussed this when that patch got in (82233ce7ea42d6b). The reason for not waiting, it was argued, is that the most likely reason for those processes not to have already gone away by the time we send SIGKILL was that they are stuck somewhere in the kernel, and so we

Re: [HACKERS] 9.5 release notes

2015-06-19 Thread Andres Freund
Hi, On 2015-06-11 00:15:21 -0400, Bruce Momjian wrote: I have committed the first draft of the 9.5 release notes. You can view the output here: So, I did a pass through master's state: listitem para Add link linkend=BRINBlock Range Indexes/ (acronymBRIN/)

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Tomas Vondra
On 06/19/2015 08:32 PM, Jeff Janes wrote: On Wed, Jun 17, 2015 at 10:52 AM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: Hi, I'm currently running some tests on a 3TB TPC-H data set, and I tripped over a pretty bad n_distinct underestimate,

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Jeff Janes
On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: But I think you might be on to something, because I manually collected a random sample with 30k rows (by explicitly generating 30k random TIDs), and I get this: tpch=# select cnt, count(*) from (select

Re: [HACKERS] 9.5 release notes

2015-06-19 Thread Andres Freund
Hi, On 2015-06-11 00:15:21 -0400, Bruce Momjian wrote: I have committed the first draft of the 9.5 release notes. You can view the output here: I'm looking through all the commits, checking which I think should possibly be mentioned additionally: - 9f03ca91 - Speed up CREATE INDEX by avoiding

Re: [HACKERS] pg_regress not waiting for postmaster to stop

2015-06-19 Thread Andrew Dunstan
On 06/19/2015 02:33 AM, Michael Paquier wrote: Hi all, In pg_regress.c, there is the following code: static void stop_postmaster(void) { if (postmaster_running) { /* We use pg_ctl to issue the kill and wait for stop */ char

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-06-19 Thread Josh Berkus
On 06/19/2015 02:51 PM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: On 05/14/2015 12:10 PM, Fabien COELHO wrote: Add backslash continuations to pgbench custom scripts. I don't personally agree. I believe that it it worth breaking backwards compatibility to support line breaks in

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-06-19 Thread Josh Berkus
On 05/14/2015 12:10 PM, Fabien COELHO wrote: Add backslash continuations to pgbench custom scripts. The benefit of this approach is that it is upward compatible, and it is also pretty simple to implement. The downside is that backslash continuation is not the best syntax ever invented, but

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Jeff Janes
On Wed, Jun 17, 2015 at 10:52 AM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: Hi, I'm currently running some tests on a 3TB TPC-H data set, and I tripped over a pretty bad n_distinct underestimate, causing OOM in HashAgg (which somehow illustrates the importance of the memory-bounded

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-06-19 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: On 05/14/2015 12:10 PM, Fabien COELHO wrote: Add backslash continuations to pgbench custom scripts. I don't personally agree. I believe that it it worth breaking backwards compatibility to support line breaks in pgbench statements, and that if we're not

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Alvaro Herrera
Tom Lane wrote: Andres Freund and...@anarazel.de writes: On 2015-06-19 13:56:21 -0300, Alvaro Herrera wrote: We discussed this when that patch got in (82233ce7ea42d6b). The reason for not waiting, it was argued, is that the most likely reason for those processes not to have already gone