[HACKERS] COPY and heap_sync

2014-08-30 Thread Jeff Janes
If you insert tuples with COPY into a table created or truncated in the same transaction, at the end of the COPY it calls heap_sync. But there cases were people use COPY in a loop with a small amount of data in each statement. Now it is calling heap_sync many times, and if NBuffers is large

Re: [HACKERS] Misleading error message in logical decoding for binary plugins

2014-08-30 Thread Michael Paquier
On Fri, Aug 29, 2014 at 11:15 PM, Andres Freund and...@2ndquadrant.com wrote: On top of that, a logical receiver receives data in textual form even if the decoder is marked as binary when getting a message with PQgetCopyData. I have no idea what you mean by that. The data is sent in the

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-08-30 Thread Amit Kapila
On Tue, Aug 26, 2014 at 9:49 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: So my proposal is a bit more complicated. First we introduce the notion of a single number, to enable sorting and computations: the delay equivalent, which is the cost_limit divided by cost_delay. The highest

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-08-30 Thread Amit Kapila
On Thu, Aug 28, 2014 at 11:06 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Robert Haas wrote: Now, in the case where you are setting an overall limit, there is at least an argument to be made that you can determine the overall rate of autovacuum-induced I/O activity that the system

Re: [HACKERS] COPY and heap_sync

2014-08-30 Thread Amit Kapila
On Sat, Aug 30, 2014 at 11:56 AM, Jeff Janes jeff.ja...@gmail.com wrote: If you insert tuples with COPY into a table created or truncated in the same transaction, at the end of the COPY it calls heap_sync. But there cases were people use COPY in a loop with a small amount of data in each

Re: [HACKERS] COPY and heap_sync

2014-08-30 Thread Atri Sharma
On Saturday, August 30, 2014, Amit Kapila amit.kapil...@gmail.com wrote: On Sat, Aug 30, 2014 at 11:56 AM, Jeff Janes jeff.ja...@gmail.com javascript:_e(%7B%7D,'cvml','jeff.ja...@gmail.com'); wrote: If you insert tuples with COPY into a table created or truncated in the same transaction,

[HACKERS] Make LWLockAcquireCommon() inline?

2014-08-30 Thread Andres Freund
Hi, when profiling optimized builds (linux, gcc 4.9) it's currently LWLockAcquireCommon() showing up, not it's callers. Instruction level profiles show that the tests for valptr show up in profiles to some extent. Since most callers don't need the valptr logic it seems prudent to mark the

Re: [HACKERS] delta relations in AFTER triggers

2014-08-30 Thread Kevin Grittner
Kevin Grittner kgri...@ymail.com wrote: Kevin Grittner kgri...@ymail.com wrote: executor-tuplestore-relations covers parse analysis, planner/optimizer, and executor layers. 30 files changed, 786 insertions(+), 9 deletions(-) Testing and further review found a few places that needed to add

Re: [HACKERS] pgbench throttling latency limit

2014-08-30 Thread Andres Freund
Hi, I generally want to say that having a feature like this feels *very* helpful to me. Lots of pg development hasn't really paid attention to anything but the final pgbench results... On 2014-08-29 19:48:43 +0200, Fabien COELHO wrote: + if (latency_limit) + printf(number of

Re: [HACKERS] pgbench throttling latency limit

2014-08-30 Thread Fabien COELHO
+ if (latency_limit) + printf(number of transactions above the %.1f ms latency limit: INT64_FORMAT \n, + latency_limit / 1000.0, latency_late); + Any reason not to report a percentage here? Yes: I did not thought of it. Here is a v7, with a

Re: [HACKERS] [PATCH] empty xml values

2014-08-30 Thread Ali Akbar
While looking into this report http://www.postgresql.org/message-id/cf48ccfb.65a9d%tim.k...@gmail.com I noticed that we don't accept empty values as xml content values, even though this should apparently be allowed by the spec. Attached is a patch to fix it (which needs updates to

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Andres Freund
On 2014-08-27 19:23:04 +0300, Heikki Linnakangas wrote: A long time ago, Itagaki Takahiro wrote a patch sort the buffers and write them out in order (http://www.postgresql.org/message-id/flat/20070614153758.6a62.itagaki.takah...@oss.ntt.co.jp). The performance impact of that was inconclusive,

Re: [HACKERS] Built-in binning functions

2014-08-30 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes: 1. I am thinking so reduction to only numeric types is not necessary - although we can live without it - but there are lot of non numeric categories: chars, date, ... I wasn't terribly happy about that either. I still think we should reduce this

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-27 19:23:04 +0300, Heikki Linnakangas wrote: A long time ago, Itagaki Takahiro wrote a patch sort the buffers and write them out in order (http://www.postgresql.org/message-id/flat/20070614153758.6a62.itagaki.takah...@oss.ntt.co.jp).

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Andres Freund
On 2014-08-30 13:50:40 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-27 19:23:04 +0300, Heikki Linnakangas wrote: A long time ago, Itagaki Takahiro wrote a patch sort the buffers and write them out in order

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-30 13:50:40 -0400, Tom Lane wrote: A possible compromise is to sort a limited number of buffers say, collect a few thousand dirty buffers then sort, dump and fsync them, repeat as needed. Yea, that's what I suggested nearby. But I

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Andres Freund
On 2014-08-30 14:16:10 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-30 13:50:40 -0400, Tom Lane wrote: A possible compromise is to sort a limited number of buffers say, collect a few thousand dirty buffers then sort, dump and fsync them, repeat as

Re: [HACKERS] Selectivity estimation for inet operators

2014-08-30 Thread Tom Lane
Emre Hasegeli e...@hasegeli.com writes: I updated the patch to cover semi and anti joins with eqjoinsel_semi(). I think it is better than returning a constant. What you did there is utterly unacceptable from a modularity standpoint; and considering that the values will be nowhere near right,

Re: [HACKERS] Selectivity estimation for inet operators

2014-08-30 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: * inet_mcv_join_selec() is O(n^2) where n is the number of entries in the MCV lists. With the max statistics target of 1, a worst case query on my laptop took about 15 seconds to plan. Maybe that's acceptable, but you went through some

Re: [HACKERS] What in the world is happening with castoroides and protosciurus?

2014-08-30 Thread Noah Misch
On Tue, Aug 26, 2014 at 10:17:05AM +0100, Dave Page wrote: On Tue, Aug 26, 2014 at 1:46 AM, Tom Lane t...@sss.pgh.pa.us wrote: For the last month or so, these two buildfarm animals (which I believe are the same physical machine) have been erratically failing with errors that reflect

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Heikki Linnakangas
On 08/30/2014 09:45 PM, Andres Freund wrote: On 2014-08-30 14:16:10 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-30 13:50:40 -0400, Tom Lane wrote: A possible compromise is to sort a limited number of buffers say, collect a few thousand dirty buffers

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Andres Freund
On 2014-08-31 01:50:48 +0300, Heikki Linnakangas wrote: On 08/30/2014 09:45 PM, Andres Freund wrote: On 2014-08-30 14:16:10 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-30 13:50:40 -0400, Tom Lane wrote: A possible compromise is to sort a limited number of

[HACKERS] Re: [BUGS] Re: BUG #9555: pg_dump for tables with inheritance recreates the table with the wrong order of columns

2014-08-30 Thread Bruce Momjian
On Wed, Aug 27, 2014 at 09:40:30PM -0400, Noah Misch wrote: 3. use the pg_dump binary-upgrade code when such cases happen +1. We have the convention that, while --binary-upgrade can inject catalog hacks, regular pg_dump uses standard, documented DDL. I like that convention on general

Re: [HACKERS] COPY and heap_sync

2014-08-30 Thread Fabrízio de Royes Mello
On Sat, Aug 30, 2014 at 5:05 AM, Atri Sharma atri.j...@gmail.com wrote: On Saturday, August 30, 2014, Amit Kapila amit.kapil...@gmail.com wrote: On Sat, Aug 30, 2014 at 11:56 AM, Jeff Janes jeff.ja...@gmail.com wrote: If you insert tuples with COPY into a table created or truncated in the

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-30 Thread Thomas Munro
On 28 August 2014 00:25, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Thomas Munro wrote: I haven't yet figured out how to get get into a situation where heap_lock_updated_tuple_rec waits. Well, as I think I said in the first post I mentioned this, maybe there is no such situation. In any

Re: [HACKERS] PATCH: Allow distdir to be overridden on make command line

2014-08-30 Thread Peter Eisentraut
On Fri, 2014-08-29 at 10:04 +0800, Craig Ringer wrote: Not just a one line patch, a one character patch. Use ?= instead of = in distdir assignment, so it can be overridden on the command line when building dist tarballs with patches. This is already possible without this patch. You can also

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-08-30 Thread Mitsumasa KONDO
Hi, 2014-08-31 8:10 GMT+09:00 Andres Freund and...@2ndquadrant.com: On 2014-08-31 01:50:48 +0300, Heikki Linnakangas wrote: If we're going to fsync between each file, there's no need to sort all the buffers at once. It's enough to pick one file as the target - like in my crude patch - and

Re: [HACKERS] PATCH: Allow distdir to be overridden on make command line

2014-08-30 Thread Michael Paquier
On Sun, Aug 31, 2014 at 10:37 AM, Peter Eisentraut pete...@gmx.net wrote: On Fri, 2014-08-29 at 10:04 +0800, Craig Ringer wrote: Not just a one line patch, a one character patch. Use ?= instead of = in distdir assignment, so it can be overridden on the command line when building dist

Re: [HACKERS] add line number as prompt option to psql

2014-08-30 Thread Sawada Masahiko
On Tue, Aug 26, 2014 at 10:23 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-21 11:43:48 +0900, Sawada Masahiko wrote: On Wed, Aug 20, 2014 at 9:00 PM, Jeevan Chalke jeevan.cha...@enterprisedb.com wrote: Hi, I have reviewed this: I have initialize cur_lineno to UINTMAX -

Re: [HACKERS] pg_dump refactor patch to remove global variables

2014-08-30 Thread Peter Eisentraut
The general idea of this patch is not disputed, I think. Some concerns about the details: - Why is quote_all_identifiers left behind as a global variable? - Shouldn't pg_dumpall also use DumpOptions? - What about binary_upgrade? - I think some of the variables in pg_dump's main() don't need

Re: [HACKERS] improving speed of make check-world

2014-08-30 Thread Peter Eisentraut
Updated, rebased patch. diff --git a/.gitignore b/.gitignore index 681af08..823d3ac 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,4 @@ lib*.pc /pgsql.sln.cache /Debug/ /Release/ +/tmp_install/ diff --git a/GNUmakefile.in b/GNUmakefile.in index 69e0824..5667943 100644 ---

[HACKERS] Tips/advice for implementing integrated RESTful HTTP API

2014-08-30 Thread Dobes Vandermeer
A while back I was working on a little proposal to create a RESTful HTTP front-end for PostgreSQL and recently I had the inspiration to work on this. So I successfully created a background worker for PostgreSQL 9.3 that can use the SPI to list off the databases in a JSON response. WIP on github:

Re: [HACKERS] [v9.5] Custom Plan API

2014-08-30 Thread Kohei KaiGai
2014-08-29 13:33 GMT-04:00 Robert Haas robertmh...@gmail.com: On Wed, Aug 27, 2014 at 6:51 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'd like to follow this direction, and start stripping the DDL support. ...please make it so. The attached patch eliminates DDL support. Instead of the