Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-22 Thread Tomas Vondra
On 06/22/2015 07:21 AM, Jeff Janes wrote: On Sat, Jun 20, 2015 at 9:55 AM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: Hi, On 06/20/2015 03:01 AM, Jeff Janes wrote: I don't think we need to really assume the density to be

[HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread Robert Haas
When a PostgreSQL system wedges, or when it becomes dreadfully slow for some reason, I often find myself relying on tools like strace, gdb, or perf to figure out what is happening. This doesn't tend to instill customers with confidence; they would like (quite understandably) a process that

Re: [HACKERS] Auto-vacuum is not running in 9.1.12

2015-06-22 Thread Prakash Itnal
Hi Tom/Alvaro, Kindly let us know if the correction provided in previous mail is fine or not! Current code any way handle scenario-1 whereas it is still vulnerable to scenario-2. From previous mail: *Scenario-1:* current_time (2015) - changed_to_past (1995) - stays-here-for-half-day - corrected

[HACKERS] PGXS check target forcing an install ?

2015-06-22 Thread Sandro Santilli
I've noted that upgrading from PostgreSQL 9.3 to 9.5 I'm suddenly unable to specify a check rule in the Makefile that includes the PGXS one. The error is: $ make check rm -rf ''/tmp_install make -C '/home/postgresql-9.5/lib/pgxs/src/makefiles/../..' DESTDIR=''/tmp_install install make[1]:

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread Joshua D. Drake
On 06/22/2015 10:37 AM, Robert Haas wrote: I'm less sure about this next part, but I think we might also want to report ourselves as waiting when we are doing an OS read or an OS write, because it's pretty common for people to think that a PostgreSQL bug is to blame when in fact it's the

Re: [HACKERS] [PATCH] pg_upgrade fails when postgres/template1 isn't in default tablespace

2015-06-22 Thread Robert Haas
On Fri, Jun 19, 2015 at 12:10 PM, Marti Raudsepp ma...@juffo.org wrote: One of my databases failed to upgrade successfully and produced this error in the copying phase: error while copying relation pg_catalog.pg_largeobject (/srv/ssd/PG_9.3_201306121/1/12023 to /PG_9.4_201409291/1/12130): No

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread Merlin Moncure
On Mon, Jun 22, 2015 at 12:37 PM, Robert Haas robertmh...@gmail.com wrote: When a PostgreSQL system wedges, or when it becomes dreadfully slow for some reason, I often find myself relying on tools like strace, gdb, or perf to figure out what is happening. This doesn't tend to instill

Re: [HACKERS] NULL passed as an argument to memcmp() in parse_func.c

2015-06-22 Thread Robert Haas
On Mon, Jun 22, 2015 at 2:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Piotr Stefaniak postg...@piotr-stefaniak.me writes: There are two places in parse_func.c where memcmp() conditionally gets a NULL as its first argument, which invokes undefined behavior. I guess gcc -O2 will make some

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread Robert Haas
On Mon, Jun 22, 2015 at 4:40 PM, Merlin Moncure mmonc...@gmail.com wrote: Instead of changing the column, can't we add a new one? Adjusting columns in PSA requires the innumerable queries written against it to be adjusted along with all the wiki instructions to dev ops for emergency stuck

Re: [HACKERS] 9.5 release notes

2015-06-22 Thread Rajeev rastogi
On 11 June 2015 09:45, Bruce Momjian Wrote: I have committed the first draft of the 9.5 release notes. You can view the output here: http://momjian.us/pgsql_docs/release-9-5.html and it will eventually appear here: http://www.postgresql.org/docs/devel/static/release.html

Re: [HACKERS] NULL passed as an argument to memcmp() in parse_func.c

2015-06-22 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jun 22, 2015 at 2:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: If I recall that code correctly, the assumption was that if the third argument is zero then memcmp() must not fetch any bytes (not should not, but MUST not) and therefore it doesn't

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread David G. Johnston
On Mon, Jun 22, 2015 at 4:09 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jun 22, 2015 at 1:59 PM, David G. Johnston david.g.johns...@gmail.com wrote: In addition to the codes themselves I think it would aid less-experienced operators if we would provide a meta-data categorization

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread Jim Nasby
On 6/22/15 12:37 PM, Robert Haas wrote: It's not my goal here to create some kind of a performance counter system, even though that would be valuable and could possibly be based on the same infrastructure, but rather just to create a very simple system that lets people know, without any

Re: [HACKERS] proposal: row_to_array function

2015-06-22 Thread Jim Nasby
On 6/22/15 2:46 AM, Pavel Stehule wrote: FOREACH key, val IN RECORD myrow LOOP IF pg_typeof(val) IN ('int4', 'double precision', 'numeric') THEN val := val + 1; -- these variables can be mutable -- or maybe in futore myrow[key] := val + 1; END IF; END LOOP; What is

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-22 Thread Peter Geoghegan
On Wed, Jun 10, 2015 at 11:48 AM, Andrew Dunstan and...@dunslane.net wrote: Please submit a patch to adjust the treatment of negative integers in the old functions to be consistent with their treatment in the new functions. i.e. in the range [-n,-1] they should refer to the corresponding

Re: [HACKERS] Hash index creation warning

2015-06-22 Thread Michael Paquier
On Tue, Jun 23, 2015 at 9:06 AM, Jim Nasby jim.na...@bluetreble.com wrote: On 6/12/15 5:00 PM, Thom Brown wrote: On 18 October 2014 at 15:36, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 17, 2014 at 02:36:55PM -0400, Bruce Momjian wrote: On Fri, Oct 17, 2014 at 12:56:52PM -0400, Tom

Re: [HACKERS] pg_stat_*_columns?

2015-06-22 Thread Robert Haas
On Sun, Jun 21, 2015 at 12:52 PM, Andres Freund and...@anarazel.de wrote: On 2015-06-21 12:40:50 -0400, Tom Lane wrote: Andres Freund and...@anarazel.de writes: We could also just mmap() the stats file into memory in various processes. With a bit care it should be quite possible to only mmap

Re: [HACKERS] less log level for success dynamic background workers for 9.5

2015-06-22 Thread Michael Paquier
On Tue, Jun 23, 2015 at 10:07 AM, Robert Haas wrote: On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby wrote: Anything ever happen with this? I agree that LOG is to high for reporting most (if not all) of these things. I think we should consider having a flag for this behavior rather than changing

Re: [HACKERS] Hash index creation warning

2015-06-22 Thread Jim Nasby
On 6/12/15 5:00 PM, Thom Brown wrote: On 18 October 2014 at 15:36, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 17, 2014 at 02:36:55PM -0400, Bruce Momjian wrote: On Fri, Oct 17, 2014 at 12:56:52PM -0400, Tom Lane wrote: David G Johnston david.g.johns...@gmail.com writes: The question

Re: [HACKERS] [Proposal] Progress bar for pg_dump/pg_restore

2015-06-22 Thread Jim Nasby
On 6/21/15 9:45 PM, Craig Ringer wrote: And, I also understood your concern about CREATE INDEX, but we have no way to get progress information of CREATE INDEX. At present, I think it may be good to refer to the same time as estimated time to execute COPY TO. You could probably get a

Re: [HACKERS] less log level for success dynamic background workers for 9.5

2015-06-22 Thread Jim Nasby
On 6/14/15 12:25 AM, Pavel Stehule wrote: Hi I am working on scheduler extension for 9.5. It use bgworkers intensively for any task. This is reason, why I need to decrease a log level - and I am thinking so parallel computing needs it due high number of created and finished workers. It should

Re: [HACKERS] pg_stat_*_columns?

2015-06-22 Thread Robert Haas
On Sun, Jun 21, 2015 at 11:43 AM, Magnus Hagander mag...@hagander.net wrote: On Sat, Jun 20, 2015 at 11:55 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jun 20, 2015 at 7:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: But if the structure got too big to map (on a 32-bit system), then you'd

Re: [HACKERS] less log level for success dynamic background workers for 9.5

2015-06-22 Thread Robert Haas
On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby jim.na...@bluetreble.com wrote: On 6/14/15 12:25 AM, Pavel Stehule wrote: I am working on scheduler extension for 9.5. It use bgworkers intensively for any task. This is reason, why I need to decrease a log level - and I am thinking so parallel

[HACKERS] user space function is_power_user

2015-06-22 Thread Pavel Stehule
Hi I often need a function for identification if current user is database owner or is superuser. It can be pretty simply implemented in C level. Do you think it should be available in core? Regards Pavel

Re: [HACKERS] checkpointer continuous flushing

2015-06-22 Thread Fabien COELHO
sorry, resent stalled post, wrong from It'd be interesting to see numbers for tiny, without the overly small checkpoint timeout value. 30s is below the OS's writeback time. Here are some tests with longer timeout: tiny2: scale=10 shared_buffers=1GB checkpoint_timeout=5min

Re: [HACKERS] checkpointer continuous flushing

2015-06-22 Thread Fabien COELHO
Hello Jim, The small problem I see is that for a very large setting there could be several seconds or even minutes of sorting, which may or may not be desirable, so having some control on that seems a good idea. ISTM a more elegant way to handle that would be to start off with a very small

Re: [HACKERS] proposal: row_to_array function

2015-06-22 Thread Pavel Stehule
Hi 2015-06-22 5:18 GMT+02:00 Craig Ringer cr...@2ndquadrant.com: On 2 April 2015 at 01:59, Merlin Moncure mmonc...@gmail.com wrote: On Sun, Mar 29, 2015 at 1:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: Pavel Stehule pavel.steh...@gmail.com writes: here is rebased patch. It contains both

Re: [HACKERS] user space function is_power_user

2015-06-22 Thread Pavel Stehule
2015-06-22 11:20 GMT+02:00 Heikki Linnakangas hlinn...@iki.fi: On 06/22/2015 09:51 AM, Pavel Stehule wrote: Hi I often need a function for identification if current user is database owner or is superuser. It can be pretty simply implemented in C level. Do you think it should be

Re: [HACKERS] user space function is_power_user

2015-06-22 Thread Heikki Linnakangas
On 06/22/2015 09:51 AM, Pavel Stehule wrote: Hi I often need a function for identification if current user is database owner or is superuser. It can be pretty simply implemented in C level. Do you think it should be available in core? current_setting('is_superuser'); - Heikki -- Sent

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: On Mon, Jun 22, 2015 at 12:37 PM, Robert Haas robertmh...@gmail.com wrote: ... The basic idea is that pg_stat_activity.waiting would be replaced by a new column pg_stat_activity.wait_event, which would display the reason why that backend is waiting.

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-22 Thread Robert Haas
On Fri, Jun 19, 2015 at 4:01 PM, Brendan Jurd dire...@gmail.com wrote: On Fri, 19 Jun 2015 at 21:05 Petr Jelinek p...@2ndquadrant.com wrote: On 2015-06-19 09:08, Brendan Jurd wrote: I think it would be convenient and user-friendly to complete the opening bracket -- it would make it

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-22 Thread David G. Johnston
On Mon, Jun 22, 2015 at 1:37 PM, Robert Haas robertmh...@gmail.com wrote: and doesn't require a developer to interpret the results, ​[...]​ We could also invent codes for things like I'm doing a pg_usleep because I've exceeded max_spins_per_delay and I'm waiting for a cleanup lock on a

Re: [HACKERS] NULL passed as an argument to memcmp() in parse_func.c

2015-06-22 Thread Tom Lane
Piotr Stefaniak postg...@piotr-stefaniak.me writes: There are two places in parse_func.c where memcmp() conditionally gets a NULL as its first argument, which invokes undefined behavior. I guess gcc -O2 will make some assumptions based on memcpy's __nonnull attribute. If I recall that code

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-22 Thread Robert Haas
On Fri, Jun 19, 2015 at 12:00 PM, Cédric Villemain ced...@2ndquadrant.com wrote: +1 for it in 9.5 If we can do it soon, sure. But not in September. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-22 Thread Jim Nasby
On 6/5/15 3:51 PM, Alvaro Herrera wrote: Jim Nasby wrote: On 6/5/15 2:08 PM, Petr Jelinek wrote: That's a good point, and it won't get any better if/when we add the json point support in 9.6 since the syntax would be something like select jsonb '{a:1, b:2, c: {a: 2}}' - '/c/a'; and we will

Re: [HACKERS] NULL passed as an argument to memcmp() in parse_func.c

2015-06-22 Thread Tom Lane
Piotr Stefaniak postg...@piotr-stefaniak.me writes: On 06/22/2015 08:55 PM, Tom Lane wrote: If I recall that code correctly, the assumption was that if the third argument is zero then memcmp() must not fetch any bytes (not should not, but MUST not) and therefore it doesn't matter if we pass a

Re: [HACKERS] upper planner path-ification

2015-06-22 Thread Kouhei Kaigai
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas Sent: Thursday, May 14, 2015 10:39 AM To: pgsql-hackers@postgresql.org; Tom Lane Subject: [HACKERS] upper planner path-ification Hi, I've been

Re: [HACKERS] less log level for success dynamic background workers for 9.5

2015-06-22 Thread Tom Lane
Michael Paquier michael.paqu...@gmail.com writes: On Tue, Jun 23, 2015 at 10:07 AM, Robert Haas wrote: On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby wrote: Anything ever happen with this? I agree that LOG is to high for reporting most (if not all) of these things. I think we should consider

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-22 Thread Tomas Vondra
Hi, On 06/22/2015 07:47 AM, Jeff Janes wrote: On Sat, Jun 20, 2015 at 8:28 AM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: Hi Tomas, I've lobotomized the sampling a bit to really produce a random set of blocks first, and that produces way

Re: [HACKERS] checkpointer continuous flushing

2015-06-22 Thread Amit Kapila
On Mon, Jun 22, 2015 at 1:41 PM, Fabien COELHO coe...@cri.ensmp.fr wrote: sorry, resent stalled post, wrong from It'd be interesting to see numbers for tiny, without the overly small checkpoint timeout value. 30s is below the OS's writeback time. Here are some tests with longer timeout:

[HACKERS] A couple of newlines missing in pg_rewind log entries

2015-06-22 Thread Michael Paquier
Hi all, Some grepping is showing up that a couple of newlines are missing in pg_rewind, leading to unreadable log entries: libpq_fetch.c:pg_log(PG_DEBUG, getting file chunks); logging.c:pg_log(PG_PROGRESS, %*s/%s kB (%d%%) copied, filemap.c:pg_fatal(could not stat file \%s\:

Re: [HACKERS] checkpointer continuous flushing

2015-06-22 Thread Fabien COELHO
Hello Amit, medium2: scale=300 shared_buffers=5GB checkpoint_timeout=30min max_wal_size=4GB warmup=1200 time=7500 flsh | full speed tps | percent of late tx, 4 clients /srt | 1 client | 4 clients | 100 | 200 | 400 | N/N | 173 +- 289* | 198 +- 531* |

Re: [HACKERS] A couple of newlines missing in pg_rewind log entries

2015-06-22 Thread Michael Paquier
On Tue, Jun 23, 2015 at 1:39 PM, Michael Paquier michael.paqu...@gmail.com wrote: Hi all, Some grepping is showing up that a couple of newlines are missing in pg_rewind, leading to unreadable log entries: libpq_fetch.c:pg_log(PG_DEBUG, getting file chunks); logging.c:

Re: [HACKERS] 9.5 make world failing due to sgml tools missing

2015-06-22 Thread Keith Fiske
http://www.keithf4.com On Sun, Jun 21, 2015 at 10:56 AM, Peter Eisentraut pete...@gmx.net wrote: On 6/18/15 8:54 AM, Tom Lane wrote: Sure; the point is that libxml2 has suddenly been reclassified as a documentation build tool, which is at least a surprising categorization. libxml2 has

Re: [HACKERS] PGXS check target forcing an install ?

2015-06-22 Thread Michael Paquier
On Tue, Jun 23, 2015 at 12:11 AM, Sandro Santilli s...@keybit.net wrote: I've noted that upgrading from PostgreSQL 9.3 to 9.5 I'm suddenly unable to specify a check rule in the Makefile that includes the PGXS one. The error is: $ make check rm -rf ''/tmp_install make -C

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-22 Thread Michael Paquier
On Fri, Jun 19, 2015 at 9:22 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier michael.paqu...@gmail.com wrote: Listing the directories with pg_ls_dir() has the same problem.

Re: [HACKERS] upper planner path-ification

2015-06-22 Thread David Rowley
On 23 June 2015 at 13:55, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Once we support to add aggregation path during path consideration, we need to pay attention morphing of the final target-list according to the intermediate path combination, tentatively chosen. For example, if

Re: [HACKERS] 9.5 make world failing due to sgml tools missing

2015-06-22 Thread Fabien COELHO
libxml2 has been a required documentation build tool since PostgreSQL 9.0. The only thing that's new is that xmllint is in a different subpackage on some systems. So just install that and you're all set for the foreseeable future. Well, something is different in 9.5. On this same system

Re: [HACKERS] Extension support for postgres_fdw

2015-06-22 Thread Tom Lane
Jim Nasby jim.na...@bluetreble.com writes: On 6/20/15 12:19 PM, Tom Lane wrote: Note that no matter what the details are, something like this is putting the onus on the DBA to mark as transmittable only functions that actually are safe to transmit, ie they exist*and have identical semantics*

Re: [HACKERS] Time to get rid of PQnoPasswordSupplied?

2015-06-22 Thread Tom Lane
Jim Nasby jim.na...@bluetreble.com writes: On 6/19/15 10:35 AM, Tom Lane wrote: On the other hand, you could argue that improving the string is going to break clients that do the right thing (even if klugily) in order to help clients that are doing the wrong thing (ie, failing without