Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-26 Thread Michael Paquier
On Tue, Aug 26, 2014 at 3:48 AM, Heikki Linnakangas wrote: > AFAICS, the namespace can never be NULL in any of these. There is a > "selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name)" call before or > after printing the message, so if tbinfo->dobj.namespace is NULL, you'll > crash anyway.

Re: [HACKERS] Escaping from blocked send() reprised.

2014-08-26 Thread Kyotaro HORIGUCHI
Hello, > > I condiered it but select() frequently (rather in most cases when > > send() blocks by send buffer exhaustion) fails to predict that > > following send() will be blocked. (If my memory is correct.) So > > the final problem would be blocked send()... > > My point was to put the socket

[HACKERS] Commitfest status

2014-08-26 Thread Heikki Linnakangas
The first week of the commitfest is now behind us. There are still 15 patches in "Needs Review" state, with no reviewer assigned. Please pick a patch and review! There are 20 patches in "Needs Review" state, with a reviewer assigned. If you have signed up as the reviewer, please proceed with

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Amit, I think another thing to know here is why exactly checkpoint storm is causing tps to drop so steeply. Yep. Actually it is not strictly 0, but a "few" tps that I rounded to 0. progress: 63.0 s, 47.0 tps, lat 2.810 ms stddev 5.194, lag 0.354 ms progress: 64.1 s, 11.9 tps, lat 8

Re: [HACKERS] pg_receivexlog and replication slots

2014-08-26 Thread Michael Paquier
On Tue, Aug 19, 2014 at 2:49 PM, Michael Paquier wrote: > On Mon, Aug 18, 2014 at 4:01 PM, Michael Paquier > wrote: >> On Mon, Aug 18, 2014 at 3:48 PM, Fujii Masao wrote: >>> On Mon, Aug 18, 2014 at 2:38 PM, Michael Paquier >>> wrote: >> And now looking at your patch there is additional refacto

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello again, I have not found any mean to force bgwriter to send writes when it can. (Well, I have: create a process which sends "CHECKPOINT" every 0.2 seconds... it works more or less, but this is not my point:-) There is scan_whole_pool_milliseconds, which currently forces bgwriter to circl

Re: [HACKERS] Concurrently option for reindexdb

2014-08-26 Thread Andres Freund
On 2014-08-26 12:44:43 +0900, Michael Paquier wrote: > On Tue, Aug 26, 2014 at 12:28 PM, Fujii Masao wrote: > >> +many. Although I'm not sure if we managed to find a safe relation swap. > > Well we didn't AFAIK. With the latest patch provided I could not > really find any whole in the logic, and

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 08:12:48 +0200, Fabien COELHO wrote: > As for checkpoint spreading, raising checkpoint_completion_target to 0.9 > degrades the situation (20% of transactions are more than 200 ms late > instead of 10%, bgwriter wrote less that 1 page per second, on on 500s run). > So maybe there is a

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Erik Rijkers
On Mon, August 25, 2014 07:21, Andrew Gierth wrote: > Here is the new version of our grouping sets patch. This version > supersedes the previous post. The patches did not apply anymore so I applied at 73eba19aebe0. There they applied OK, and make && make check was OK. drop table if exists it

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Andres, checkpoint when the segments are full... the server is unresponsive about 10% of the time (one in ten transaction is late by more than 200 ms). That's ext4 I guess? Yes! Did you check whether xfs yields a, err, more predictable performance? No. I cannot test that easily wi

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 10:25:29 +0200, Fabien COELHO wrote: > >Did you check whether xfs yields a, err, more predictable performance? > > No. I cannot test that easily without reinstalling the box. I did some quick > tests with ZFS/FreeBSD which seemed to freeze the same, but not in the very > same conditi

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Rukh Meski
Hi Fabien, On Sun, Aug 24, 2014 at 9:16 AM, Fabien COELHO wrote: > Find attached a new version: > - fix dropped percent computation in the final report > - simplify progress report code I have reviewed this patch. Is the patch in a patch format which has context? Yes. Does it apply cleanly

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Heikki Linnakangas
On 08/16/2014 02:19 AM, Tom Lane wrote: I think the realistic alternatives at this point are either to switch to all-lengths as in my test patch, or to use the hybrid approach of Heikki's test patch. IMO the major attraction of Heikki's patch is that it'd be upward compatible with existing beta

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
What are the other settings here? checkpoint_segments, checkpoint_timeout, wal_buffers? They simply are the defaults: checkpoint_segments = 3 checkpoint_timeout = 5min wal_buffers = -1 I did some test checkpoint_segments = 1, the problem is just more frequent but shorter. I also reduc

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 10:10 AM, Michael Paquier wrote: On Tue, Aug 26, 2014 at 3:48 AM, Heikki Linnakangas wrote: AFAICS, the namespace can never be NULL in any of these. There is a "selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name)" call before or after printing the message, so if tbinfo->do

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 10:49:31 +0200, Fabien COELHO wrote: > > >What are the other settings here? checkpoint_segments, > >checkpoint_timeout, wal_buffers? > > They simply are the defaults: > > checkpoint_segments = 3 > checkpoint_timeout = 5min > wal_buffers = -1 > > I did some test checkpoint_se

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Andrew Gierth
> "Erik" == Erik Rijkers writes: Erik> The patches did not apply anymore so I applied at 73eba19aebe0. Erik> There they applied OK, and make && make check was OK. I'll look and rebase if need be. --> WARNING: unrecognized node type: 347 Can't reproduce this - are you sure it's not a mi

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Fri, 2014-06-13 at 13:24 +0300, Heikki Linnakangas wrote: > Attached is a new patch. It now keeps the current pg_clog unchanged, but > adds a new pg_csnlog besides it. pg_csnlog is more similar to > pg_subtrans than pg_clog: it's not WAL-logged, is reset at startup, and > segments older than

Re: [HACKERS] replicating DROP commands across servers

2014-08-26 Thread Andres Freund
On 2014-06-13 15:50:50 -0400, Alvaro Herrera wrote: > Here's a patch implementing the proposed idea. This is used in the > Bidirectional Replication stuff by Simon/Andres; it works well. I think there's been some changes to this patch since july, care to resend a new version? I think it's approp

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Fabien COELHO
Hello Rukh, I have reviewed this patch. Thanks! [...] I get: pgbench: invalid option -- L Which appears to be caused by the fact that the call to getopt_long() has not been updated to reflect the new parameter. Indeed, I only tested/used it with the --limit= syntax. Also this part: +

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: > "Erik" == Erik Rijkers writes: Erik> The patches did not apply anymore so I applied at 73eba19aebe0. Erik> There they applied OK, and make && make check was OK. Andrew> I'll look and rebase if need be. They apply cleanly for me at 2bde297 wheth

Re: [HACKERS] What in the world is happening with castoroides and protosciurus?

2014-08-26 Thread Dave Page
On Tue, Aug 26, 2014 at 1:46 AM, Tom Lane wrote: > For the last month or so, these two buildfarm animals (which I believe are > the same physical machine) have been erratically failing with errors that > reflect low-order differences in floating-point calculations. > > A recent example is at > > h

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Uh. I'm not surprised you're facing utterly horrible performance with this. Did you try using a *large* checkpoints_segments setting? To achieve high performance I do not seek "high performance" per se, I seek "lower maximum latency". I think that the current settings and parameters are desig

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Fabien COELHO
Marking Waiting for Author until these small issues have been fixed. I've put it back to "Needs review". Feel free to set it to "Ready" if it is ok for you. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.po

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-26 Thread Heikki Linnakangas
Summary of this thread so far: There was a lot of discussion comparing this with Tomas Vondra's Hash Join patch. The conclusion was that while it would be nice to be able to dump transition state to disk, for aggregates like array_agg, the patch is fine as it is. Dumping transition states woul

Re: [HACKERS] Selectivity estimation for inet operators

2014-08-26 Thread Emre Hasegeli
> I agree with you that we can support other join type and anti join later, > If others don’t have any objection in doing other parts later I will mark as > "Ready For Committer". I updated the patch to cover semi and anti joins with eqjoinsel_semi(). I think it is better than returning a constan

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Andres Freund
On 2014-08-26 11:34:36 +0200, Fabien COELHO wrote: > > >Uh. I'm not surprised you're facing utterly horrible performance with > >this. Did you try using a *large* checkpoints_segments setting? To > >achieve high performance > > I do not seek "high performance" per se, I seek "lower maximum latenc

Re: Compute attr_needed for child relations (was Re: [HACKERS] inherit support for foreign tables)

2014-08-26 Thread Shigeru Hanada
Hi Fujita-san, I reviewed the v4 patch, and here are some comments from me. * After applying this patch, pull_varattnos() should not called in unnecessary places. For developers who want list of columns-to-be-processed for some purpose, it would be nice to mention when they should use pull_varat

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Erik Rijkers
On Tue, August 26, 2014 11:13, Andrew Gierth wrote: >> "Andrew" == Andrew Gierth writes: > >> "Erik" == Erik Rijkers writes: > > Erik> The patches did not apply anymore so I applied at 73eba19aebe0. > Erik> There they applied OK, and make && make check was OK. > > Andrew> I'll look and

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 12:03 PM, Jeff Davis wrote: On Fri, 2014-06-13 at 13:24 +0300, Heikki Linnakangas wrote: Attached is a new patch. It now keeps the current pg_clog unchanged, but adds a new pg_csnlog besides it. pg_csnlog is more similar to pg_subtrans than pg_clog: it's not WAL-logged, is reset at

Re: [HACKERS] replication commands and log_statements

2014-08-26 Thread Fujii Masao
On Wed, Aug 20, 2014 at 1:14 PM, Michael Paquier wrote: > > > > On Wed, Aug 20, 2014 at 2:06 AM, Robert Haas wrote: >> >> On Sat, Aug 16, 2014 at 10:27 AM, Amit Kapila >> wrote: >> > I think ideally it would have been better if we could have logged >> > replication commands under separate log_le

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Andres Freund
On 2014-06-29 18:54:50 +0530, Abhijit Menon-Sen wrote: > At 2014-06-29 20:35:04 +0900, maumau...@gmail.com wrote: > > > > Thanks, I marked it as ready for committer. I hope Fujii san or > > another committer will commit this, refining English expression if > > necessary. > > Since it was just a m

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Oskari Saarenmaa
On 25/08/14 14:35, Andres Freund wrote: currently pg_basebackup uses fetch mode when only -x is specified - which imo isn't a very good thing to use due to the increased risk of not fetching everything. How about switching to stream mode for 9.5+? +1. I was just wondering why it's not the defa

Re: [HACKERS] Verbose output of pg_dump not show schema name

2014-08-26 Thread Fabrízio de Royes Mello
On Tue, Aug 26, 2014 at 4:10 AM, Michael Paquier wrote: > > On Tue, Aug 26, 2014 at 3:48 AM, Heikki Linnakangas > wrote: > > AFAICS, the namespace can never be NULL in any of these. There is a > > "selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name)" call before or > > after printing the

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Petr Jelinek
On 26/08/14 13:20, Andres Freund wrote: I'm looking at committing this, but I wonder: Shouldn't this be accessible from inside psql as well? I.e. as a backslash command? +1 -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Servic

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Pavel Stehule
2014-08-26 13:30 GMT+02:00 Petr Jelinek : > On 26/08/14 13:20, Andres Freund wrote: > >> >> I'm looking at committing this, but I wonder: Shouldn't this be >> accessible from inside psql as well? I.e. as a backslash command? >> >> > +1 > have you idea about command name? \?+ Pavel > > > --

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-08-26 Thread Andres Freund
On 2014-08-26 13:44:16 +0200, Pavel Stehule wrote: > 2014-08-26 13:30 GMT+02:00 Petr Jelinek : > > > On 26/08/14 13:20, Andres Freund wrote: > > > >> > >> I'm looking at committing this, but I wonder: Shouldn't this be > >> accessible from inside psql as well? I.e. as a backslash command? > >> > >

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-26 Thread Fujii Masao
On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed wrote: > Hello, > Thank you for comments. > >>Could you tell me where the patch for "single block in one run" is? > Please find attached patch for single block compression in one run. Thanks! I ran the benchmark using pgbench and compared the results.

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Andrew Gierth
> "Erik" == Erik Rijkers writes: >> They apply cleanly for me at 2bde297 whether with git apply or >> patch, except for the contrib one (which you don't need unless you >> want to run the contrib regression tests without applying the >> gsp-u patch). Erik> Ah, I had not realised that.

[HACKERS] a7ae1dc has broken the windows builds

2014-08-26 Thread David Rowley
I've attached a small patch which should get the windows builds up and running again. They're currently failing from this: "c:\prog\bf\root\HEAD\pgsql.build\pgsql.sln" (default target) (1) -> "c:\prog\bf\root\HEAD\pgsql.build\pg_upgrade_support.vcxproj" (default target) (67) -> (Link target) ->

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread David Rowley
On Tue, Aug 5, 2014 at 10:35 PM, David Rowley wrote: > > Currently most of my changes are in analyzejoin.c, but I did also have to > make changes to load the foreign key constraints so that they were > available to the planner. One thing that is currently lacking, which would > likely be needed,

Re: [HACKERS] Final Patch for GROUPING SETS - unrecognized node type: 347

2014-08-26 Thread Erik Rijkers
On Tue, August 26, 2014 14:24, Andrew Gierth wrote: >> "Erik" == Erik Rijkers writes: > > >> They apply cleanly for me at 2bde297 whether with git apply or > >> patch, except for the contrib one (which you don't need unless you > >> want to run the contrib regression tests without applying

Re: [HACKERS] Patch to add support of "IF NOT EXISTS" to others "CREATE" statements

2014-08-26 Thread Heikki Linnakangas
On 04/14/2014 10:31 PM, Fabrízio de Royes Mello wrote: On Tue, Apr 1, 2014 at 2:46 PM, Robert Haas wrote: Where this is a bit more interesting is in the case of sequences, where resetting the sequence to zero may cause further inserts into an existing table to fail. Yeah. Sequences do have

Re: [HACKERS] add line number as prompt option to psql

2014-08-26 Thread Andres Freund
On 2014-08-21 11:43:48 +0900, Sawada Masahiko wrote: > On Wed, Aug 20, 2014 at 9:00 PM, Jeevan Chalke > wrote: > > Hi, > > > > I have reviewed this: > > > > I have initialize cur_lineno to UINTMAX - 2. And then observed following > > behaviour to check wrap-around. > > > > postgres=# \set PROMPT1

Re: [HACKERS] a7ae1dc has broken the windows builds

2014-08-26 Thread Andres Freund
On 2014-08-27 00:25:43 +1200, David Rowley wrote: > I've attached a small patch which should get the windows builds up and > running again. Pushed, thanks. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] Patch to add support of "IF NOT EXISTS" to others "CREATE" statements

2014-08-26 Thread Fabrízio de Royes Mello
On Tue, Aug 26, 2014 at 10:20 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 04/14/2014 10:31 PM, Fabrízio de Royes Mello wrote: > >> On Tue, Apr 1, 2014 at 2:46 PM, Robert Haas >> wrote: >> >>> >>> Where this is a bit more interesting is in the case of sequences, where > rese

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-08-26 Thread Heikki Linnakangas
On 08/26/2014 03:28 PM, David Rowley wrote: Any ideas or feedback on this would be welcome Before someone spends time reviewing this patch, are you sure this is worth the effort? It seems like very narrow use case to me. I understand removing LEFT and INNER joins, but the case for SEMI and AN

Re: [HACKERS] replicating DROP commands across servers

2014-08-26 Thread Alvaro Herrera
Andres Freund wrote: > On 2014-06-13 15:50:50 -0400, Alvaro Herrera wrote: > > Here's a patch implementing the proposed idea. This is used in the > > Bidirectional Replication stuff by Simon/Andres; it works well. > > I think there's been some changes to this patch since july, care to > resend a

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Heikki Linnakangas writes: > On 08/16/2014 02:19 AM, Tom Lane wrote: >> I think the realistic alternatives at this point are either to >> switch to all-lengths as in my test patch, or to use the hybrid approach >> of Heikki's test patch. ... >> Personally I'd prefer to go to the all-lengths approa

Re: [HACKERS] Scaling shared buffer eviction

2014-08-26 Thread Amit Kapila
On Tue, Aug 5, 2014 at 9:21 PM, Robert Haas wrote: > > Incidentally, while I generally think your changes to the locking regimen in StrategyGetBuffer() are going in the right direction, they need significant cleanup. Your patch adds two new spinlocks, freelist_lck and victimbuf_lck, that mostly b

Re: [HACKERS] Scaling shared buffer eviction

2014-08-26 Thread Tom Lane
Amit Kapila writes: > On Tue, Aug 5, 2014 at 9:21 PM, Robert Haas wrote: >>> I think you should get rid of BufFreelistLock completely and just >>> decide that freelist_lck will protect everything the freeNext links, plus >>> everything in StrategyControl except for nextVictimBuffer. victimbuf_lc

Re: [HACKERS] [BUGS] BUG #11208: Refresh Materialized View Concurrently bug using user Postgres

2014-08-26 Thread Kevin Grittner
Kevin Grittner wrote: > Kevin Grittner wrote: >> "bemanuel...@gmail.com" wrote: >> >>> tjma_dw=> set role user_dw; >>> >>> tjma_dw=> CREATE TABLE foo_data AS SELECT i, md5(random()::text) FROM >>> generate_series(1, 10) i; >>> SELECT 10 >>> tjma_dw=> CREATE MATERIALIZED VIEW mv_foo AS SELE

Re: [HACKERS] Scaling shared buffer eviction

2014-08-26 Thread Amit Kapila
On Tue, Aug 26, 2014 at 8:40 PM, Tom Lane wrote: > Amit Kapila writes: > > Another point is I think it will be better to protect > > StrategyControl->completePasses with victimbuf_lck rather than > > freelist_lck, as when we are going to update it we will already be > > holding the victimbuf_lck

Re: [HACKERS] pg_dump refactor patch to remove global variables

2014-08-26 Thread Peter Eisentraut
On 8/15/14 7:30 PM, Joachim Wieland wrote: > Attached is a patch that doesn't add any new functionality or > features, all it does is get rid of the global variables that > pg_dump.c is full of. I'm getting a compiler error: In file included from pg_dump.c:60: In file included from ./pg_backup_ar

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-08-26 Thread Alvaro Herrera
Mark Kirkwood wrote: > On 06/05/14 16:28, Amit Kapila wrote: > >On Mon, May 5, 2014 at 11:57 AM, Mark Kirkwood > > wrote: > >I could think of 2 ways to change this: > > > >a. if user has specified cost_limit value for table, then it just uses it > > rather than rebalancing based on value of s

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Magnus Hagander
On Mon, Aug 25, 2014 at 1:35 PM, Andres Freund wrote: > Hi, > > currently pg_basebackup uses fetch mode when only -x is specified - > which imo isn't a very good thing to use due to the increased risk of > not fetching everything. > How about switching to stream mode for 9.5+? I think the origina

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Andres Freund
On 2014-08-26 18:40:27 +0200, Magnus Hagander wrote: > On Mon, Aug 25, 2014 at 1:35 PM, Andres Freund wrote: > > Hi, > > > > currently pg_basebackup uses fetch mode when only -x is specified - > > which imo isn't a very good thing to use due to the increased risk of > > not fetching everything. >

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Magnus Hagander
On Tue, Aug 26, 2014 at 6:51 PM, Andres Freund wrote: > On 2014-08-26 18:40:27 +0200, Magnus Hagander wrote: >> On Mon, Aug 25, 2014 at 1:35 PM, Andres Freund >> wrote: >> > Hi, >> > >> > currently pg_basebackup uses fetch mode when only -x is specified - >> > which imo isn't a very good thing t

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 13:45 +0300, Heikki Linnakangas wrote: > My current thinking is that access to the csnlog will need to be made > faster. Currently, it's just another SLRU, but I'm sure we can do better > than that. Or add a backend-private cache on top of it; that might > already alleviate

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Greg Stark
On Tue, Aug 26, 2014 at 11:45 AM, Heikki Linnakangas wrote: >> It appears that this patch weakens the idea of hint bits. Even if >> HEAP_XMIN_COMMITTED is set, it still needs to find out if it's in the >> snapshot. >> >> That's fast if the xid is less than snap->xmin, but otherwise it needs >> to

Re: Compute attr_needed for child relations (was Re: [HACKERS] inherit support for foreign tables)

2014-08-26 Thread Tom Lane
Etsuro Fujita writes: > [ attr_needed-v4.patch ] I looked this over, and TBH I'm rather disappointed. The patch adds 150 lines of dubiously-correct code in order to save ... uh, well, actually it *adds* code, because the places that are supposedly getting a benefit are changed like this: *** 79

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 07:51 AM, Tom Lane wrote: > My feeling about it at this point is that the apparent speed gain from > using offsets is illusory: in practically all real-world cases where there > are enough keys or array elements for it to matter, costs associated with > compression (or rather failure

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Josh Berkus writes: > Anyway, I called for feedback on by blog, and have gotten some: > http://www.databasesoup.com/2014/08/the-great-jsonb-tradeoff.html I was hoping you'd get some useful data from that, but so far it seems like a rehash of points made in the on-list thread :-(

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 11:40 AM, Tom Lane wrote: > Josh Berkus writes: >> Anyway, I called for feedback on by blog, and have gotten some: >> http://www.databasesoup.com/2014/08/the-great-jsonb-tradeoff.html > > I was hoping you'd get some useful data from that, but so far it seems > like a rehash of poin

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Josh Berkus writes: > On 08/26/2014 11:40 AM, Tom Lane wrote: >> I was hoping you'd get some useful data from that, but so far it seems >> like a rehash of points made in the on-list thread :-( > Unfortunately even the outside commentors don't seem to understand that > storage size *is* related t

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Andres Freund
On 2014-08-26 15:01:27 -0400, Tom Lane wrote: > Josh Berkus writes: > > On 08/26/2014 11:40 AM, Tom Lane wrote: > >> I was hoping you'd get some useful data from that, but so far it seems > >> like a rehash of points made in the on-list thread :-( > > > Unfortunately even the outside commentors d

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Laurence Rowe
On 26 August 2014 11:34, Josh Berkus wrote: > On 08/26/2014 07:51 AM, Tom Lane wrote: > > My feeling about it at this point is that the apparent speed gain from > > using offsets is illusory: in practically all real-world cases where > there > > are enough keys or array elements for it to matter,

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
Andres Freund writes: > On 2014-08-26 15:01:27 -0400, Tom Lane wrote: >> Yeah, exactly. Given current hardware trends, data compression is >> becoming more of a win not less as time goes on: CPU cycles are cheap >> even compared to main memory access, let alone mass storage. So I'm >> thinking w

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Claudio Freire
On Tue, Aug 26, 2014 at 4:01 PM, Tom Lane wrote: > Josh Berkus writes: >> On 08/26/2014 11:40 AM, Tom Lane wrote: >>> I was hoping you'd get some useful data from that, but so far it seems >>> like a rehash of points made in the on-list thread :-( > >> Unfortunately even the outside commentors do

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Andres Freund
On 2014-08-26 15:17:13 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-08-26 15:01:27 -0400, Tom Lane wrote: > >> Yeah, exactly. Given current hardware trends, data compression is > >> becoming more of a win not less as time goes on: CPU cycles are cheap > >> even compared to main memo

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 19:25 +0100, Greg Stark wrote: > I don't immediately see how to make that practical. One thought would > be to have a list of xids in the page header with their corresponding > csn -- which starts to sound a lot like Oralce's "Interested > Transaction List". But I don't see ho

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Peter Geoghegan
On Tue, Aug 26, 2014 at 12:27 PM, Andres Freund wrote: > Anyway, that's just to say that I don't really agree that CPU overhead > is a worthy price to pay for storage efficiency if the gains are small. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] New Model For Role Attributes and Fine Grained Permssions

2014-08-26 Thread Robert Haas
On Thu, Aug 21, 2014 at 10:49 PM, Stephen Frost wrote: > * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: >> On 08/19/2014 04:27 AM, Brightwell, Adam wrote: >> >This is a "proof-of-concept" patch for a new model around role attributes >> >and fine grained permissions meant to alleviate the cu

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 12:39 +0300, Heikki Linnakangas wrote: > I think this is enough for this commitfest - we have consensus on the > design. For the next one, please address those open items, and resubmit. Agreed, return with feedback. I need to get the accounting patch in first, which needs t

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 12:27 PM, Andres Freund wrote: > Anyway, that's just to say that I don't really agree that CPU overhead > is a worthy price to pay for storage efficiency if the gains are small. But in this case the gains aren't small; we're talking up to 60% smaller storage. Testing STORAGE EXTENDE

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-26 Thread Fabien COELHO
Hello Jeff, The culprit I found is "bgwriter", which is basically doing nothing to prevent the coming checkpoint IO storm, even though there would be ample time to write the accumulating dirty pages so that checkpoint would find a clean field and pass in a blink. Indeed, at the end of the 500 s

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-08-26 Thread Robert Haas
On Fri, Aug 22, 2014 at 2:46 PM, Peter Geoghegan wrote: > On Fri, Aug 22, 2014 at 7:19 AM, Robert Haas wrote: >> Patch 0002 no longer applies; please rebase. > > I attach rebased patch. > > Note that there is currently a bug in the master branch: > > + if (len2 >= tss->buflen2) > + { > +

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Greg Stark
On Tue, Aug 26, 2014 at 8:32 PM, Jeff Davis wrote: > We are mixing two kinds of data: user data and visibility information. > Each is changed under different circumstances and has different > characteristics, and I'm beginning to think they shouldn't be mixed at > all. > > What if we just devised

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-08-26 Thread Peter Geoghegan
On Tue, Aug 26, 2014 at 12:59 PM, Robert Haas wrote: > I have committed a fix for that problem. Let me know if I missed > something else. Yes, that's all I meant. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: h

Re: [HACKERS] [v9.5] Custom Plan API

2014-08-26 Thread Robert Haas
On Fri, Aug 22, 2014 at 9:48 PM, Kohei KaiGai wrote: > One thing I was pointed out, it is the reason why I implemented > DDL support, is that intermediation of c-language function also > loads the extension module implicitly. It is an advantage, but > not sure whether it shall be supported from th

Re: [HACKERS] Proposal for CSN based snapshots

2014-08-26 Thread Jeff Davis
On Tue, 2014-08-26 at 21:00 +0100, Greg Stark wrote: > Well fundamentally the reason the visibility information is with the > user data is so that we don't need to do two i/os to access the data. > The whole point of hint bits is to guarantee that after some amount of > time you can read data direc

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Robert Haas
On Sat, Aug 23, 2014 at 6:39 PM, Greg Stark wrote: > On Sat, Aug 23, 2014 at 9:38 PM, Tom Lane wrote: >> >> Ah. Okay, but then what's wrong with the original proposal of "use ceil() >> instead of floor()"? Basically I think the idea of treating fractions >> less than one differently from fracti

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Andres Freund
On 2014-08-26 16:16:32 -0400, Robert Haas wrote: > On Sat, Aug 23, 2014 at 6:39 PM, Greg Stark wrote: > > On Sat, Aug 23, 2014 at 9:38 PM, Tom Lane wrote: > >> > >> Ah. Okay, but then what's wrong with the original proposal of "use ceil() > >> instead of floor()"? Basically I think the idea of

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Peter Eisentraut
On 8/26/14 12:40 PM, Magnus Hagander wrote: > I think the first reason is gone now, and the risk/damage of the two > connections is probably smaller than running out of WAL. -x is a good > default for smaller systems, but -X is a safer one for bigger ones. So > I agree that changing the default mod

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Andres Freund
On 2014-08-26 16:41:44 -0400, Peter Eisentraut wrote: > On 8/26/14 12:40 PM, Magnus Hagander wrote: > > I think the first reason is gone now, and the risk/damage of the two > > connections is probably smaller than running out of WAL. -x is a good > > default for smaller systems, but -X is a safer o

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Peter Eisentraut
On 8/23/14 6:39 PM, Greg Stark wrote: > Or make it an error to specify a value that rounds to 0 but isn't 0. That's what I would prefer. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Peter Eisentraut
On 8/26/14 4:22 PM, Andres Freund wrote: > Is the whole topic actually practically relevant? It's clearly not all that important, or otherwise we'd have heard about before now. I suppose someone could do something like wal_receiver_status_interval = 10ms and end up silently turning the whole th

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-26 Thread Robert Haas
On Sat, Aug 23, 2014 at 11:59 PM, Michael Paquier wrote: > As mentioned in $subject, the header files in src/bin/pg_basebackup do > not have a comment block at the top and do not have any copyright > text. > Any reason for that? Shouldn't we have something for consistency with > the other files li

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-26 Thread Magnus Hagander
On Tue, Aug 26, 2014 at 11:03 PM, Robert Haas wrote: > On Sat, Aug 23, 2014 at 11:59 PM, Michael Paquier > wrote: >> As mentioned in $subject, the header files in src/bin/pg_basebackup do >> not have a comment block at the top and do not have any copyright >> text. >> Any reason for that? Shouldn

Re: [HACKERS] Switch pg_basebackup to use -X stream instead of -X fetch by default?

2014-08-26 Thread Magnus Hagander
On Tue, Aug 26, 2014 at 10:46 PM, Andres Freund wrote: > On 2014-08-26 16:41:44 -0400, Peter Eisentraut wrote: >> On 8/26/14 12:40 PM, Magnus Hagander wrote: >> > I think the first reason is gone now, and the risk/damage of the two >> > connections is probably smaller than running out of WAL. -x i

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread Tom Lane
Robert Haas writes: > I liked David Johnston's even stronger suggestion upthread: make it an > error to specify a value requires rounding of any kind. In other > words, if the minimum granularity is 1 minute, you can specify that as > 60 seconds instead, but if you write 59 seconds, we error out.

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-08-26 Thread David G Johnston
Tom Lane-2 wrote > Robert Haas < > robertmhaas@ > > writes: >> I liked David Johnston's even stronger suggestion upthread: make it an >> error to specify a value requires rounding of any kind. In other >> words, if the minimum granularity is 1 minute, you can specify that as >> 60 seconds instea

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Tom Lane
I wrote: > I wish it were cache-friendly too, per the upthread tangent about having > to fetch keys from all over the place within a large JSON object. > ... and while I was typing that sentence, lightning struck. The existing > arrangement of object subfields with keys and values interleaved is

Re: [HACKERS] Per table autovacuum vacuum cost limit behaviour strange

2014-08-26 Thread Mark Kirkwood
On 27/08/14 10:27, Alvaro Herrera wrote: Alvaro Herrera wrote: So my proposal is a bit more complicated. First we introduce the notion of a single number, to enable sorting and computations: the "delay equivalent", which is the cost_limit divided by cost_delay. Here's a patch that implements

Re: [HACKERS] delta relations in AFTER triggers

2014-08-26 Thread Kevin Grittner
Kevin Grittner wrote: > I think this is approaching a committable state, although I think > it should probably be broken down to four separate patches. And here they are. This should net to the same set of changes as the prior post, but the changes are logically separated. They are labeled as

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-26 Thread Andres Freund
Hi, On 2014-03-21 19:22:31 +0100, Andres Freund wrote: > Hi, > > I've been annoyed at the amount of memory used by the backend local > PrivateRefCount array for a couple of reasons: > > a) The performance impact of AtEOXact_Buffers() on Assert() enabled >builds is really, really annoying. >

Re: [HACKERS] pgbench throttling latency limit

2014-08-26 Thread Rukh Meski
Hi Fabien, On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO wrote: > > Please find attached a new version which fixes these two points. Indeed it does. Marking the patch ready for a committer. Thanks, ♜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Missing comment block at the top of streamutil.h and receivelog.h

2014-08-26 Thread Michael Paquier
On Wed, Aug 27, 2014 at 6:10 AM, Magnus Hagander wrote: > On Tue, Aug 26, 2014 at 11:03 PM, Robert Haas wrote: >> Probably that's a good idea, but do we really need Author: tags? I >> know we have those in a few places, but certainly not everywhere, and >> as time goes by they tend to be less ac

Re: [HACKERS] [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

2014-08-26 Thread Jim Nasby
On 8/26/14, 6:52 PM, Andres Freund wrote: On 2014-03-21 19:22:31 +0100, Andres Freund wrote: >Hi, > >I've been annoyed at the amount of memory used by the backend local >PrivateRefCount array for a couple of reasons: > >a) The performance impact of AtEOXact_Buffers() on Assert() enabled >bui

[HACKERS] Similar to csvlog but not really, json logs?

2014-08-26 Thread Michael Paquier
Hi all, As mentioned here, we support multiple logging format: http://www.postgresql.org/docs/devel/static/runtime-config-logging.html Now what about a json format logging with one json object per log entry? A single json entry would need more space than a csv one as we need to track the field na

  1   2   >