Re: [HACKERS] Support for N synchronous standby servers

2014-08-13 Thread Michael Paquier
On Wed, Aug 13, 2014 at 2:10 PM, Fujii Masao masao.fu...@gmail.com wrote: I sent the SIGSTOP signal to the walreceiver process in one of sync standbys, and then ran write transactions again. In this case, they must not be completed because their WAL cannot be replicated to the standby that

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Michael Paquier
On Tue, Aug 12, 2014 at 6:44 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 08/05/2014 03:50 PM, Michael Paquier wrote: - When testing pg_xlogdump I found an assertion failure for record XLOG_GIN_INSERT: Assertion failed: (((bool) (((const void*)(insertData-newitem.key) !=

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Michael Paquier
On Wed, Aug 13, 2014 at 4:49 PM, Michael Paquier michael.paqu...@gmail.com wrote: Yes indeed. As XLogBeginInsert() is already part of xlogconstruct.c, it is not weird. This approach has the merit to make XLogRecData completely bundled with the insertion and construction of the WAL records.

Re: [HACKERS] pg_receivexlog --status-interval add fsync feedback

2014-08-13 Thread furuyao
I don't think that it's good idea to control that behavior by using --status-interval. I'm sure that there are some users who both want that behavior and want set the maximum interval between a feedback is sent back to the server because these settings are available in walreceiver. But your

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-13 Thread MauMau
From: Pavel Stehule pavel.steh...@gmail.com There are two relative independent tasks a) monitor and show total lock time of living queries b) monitor and log total lock time of executed queries. I am interested by @b now. When we work with slow query log, then we would to identify reason for

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-13 Thread Pavel Stehule
2014-08-13 11:14 GMT+02:00 MauMau maumau...@gmail.com: From: Pavel Stehule pavel.steh...@gmail.com There are two relative independent tasks a) monitor and show total lock time of living queries b) monitor and log total lock time of executed queries. I am interested by @b now. When we

Re: [HACKERS] strncpy is not a safe version of strcpy

2014-08-13 Thread David Rowley
On Wed, Aug 13, 2014 at 3:19 PM, Noah Misch n...@leadboat.com wrote: On Sat, Nov 16, 2013 at 12:53:10PM +1300, David Rowley wrote: I went on a bit of a strncpy cleanup rampage this morning and ended up finding quite a few places where strncpy is used wrongly. I'm not quite sure if I have

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-13 Thread Tomas Vondra
On 13 Srpen 2014, 7:02, Jeff Davis wrote: On Tue, 2014-08-12 at 14:58 +0200, Tomas Vondra wrote: CREATE AGGREGATE myaggregate ( ... SERIALIZE_FUNC = 'dump_data', DESERIALIZE_FUNC = 'read_data', ... ); Seems reasonable. I don't see why it should get messy? In the end, you

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2014-08-13 Thread Dilip kumar
On 11 August 2014 10:29, Amit kapila wrote, 1. I have fixed all the review comments except few, and modified patch is attached. 2. For not fixed comments, find inline reply in the mail.. 1. +Number of parallel connections to perform the operation. This option will enable the

[HACKERS] Enable WAL archiving even in standby

2014-08-13 Thread Fujii Masao
Hi, I'd propose the attached WIP patch which allows us to enable WAL archiving even in standby. The patch adds always as the valid value of archive_mode. If it's set to always, the archiver is started when the server is in standby mode and all the WAL files that walreceiver wrote to the disk are

Re: [HACKERS] Scaling shared buffer eviction

2014-08-13 Thread Andres Freund
On 2014-08-13 09:51:58 +0530, Amit Kapila wrote: Overall, the main changes required in patch as per above feedback are: 1. add an additional counter for the number of those allocations not satisfied from the free list, with a name like buffers_alloc_clocksweep. 2. Autotune the low and high

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Andres Freund
On 2014-08-13 02:36:59 +0300, Heikki Linnakangas wrote: On 08/13/2014 01:04 AM, Andres Freund wrote: * I'm distinctly not a fan of passing around both the LSN and the struct XLogRecord to functions. We should bit the bullet and use something encapsulating both. You mean, the redo

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-13 Thread MauMau
From: Pavel Stehule pavel.steh...@gmail.com isn't it too heavy? Are you concerned about the impactof collection overhead on the queries diagnosed? Maybe not light, but I'm optimistic. Oracle has the track record of long use, and MySQL provides performance schema starting from 5.6. I have

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-13 Thread Pavel Stehule
2014-08-13 13:59 GMT+02:00 MauMau maumau...@gmail.com: From: Pavel Stehule pavel.steh...@gmail.com isn't it too heavy? Are you concerned about the impactof collection overhead on the queries diagnosed? Maybe not light, but I'm optimistic. Oracle has the track record of long use, and

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Heikki Linnakangas
On 08/13/2014 02:07 PM, Andres Freund wrote: On 2014-08-13 02:36:59 +0300, Heikki Linnakangas wrote: On 08/13/2014 01:04 AM, Andres Freund wrote: * The patch mixes the API changes around WAL records with changes of how individual actions are logged. That makes it rather hard to review -

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-13 Thread MauMau
From: Pavel Stehule pavel.steh...@gmail.com 2014-08-13 13:59 GMT+02:00 MauMau maumau...@gmail.com: Are you concerned about the impactof collection overhead on the queries diagnosed? Maybe not light, but I'm optimistic. Oracle has the track record of long use, and MySQL provides performance

Re: [HACKERS] strncpy is not a safe version of strcpy

2014-08-13 Thread Kevin Grittner
David Rowley dgrowle...@gmail.com wrote: I had a quick look at the usages of strncpy in master tonight and I've really just picked out the obviously broken ones for now. The other ones, on first look, either look safe, or require some more analysis to see what's actually done with the string.

Re: [HACKERS] ALTER TABLESPACE MOVE command tag tweak

2014-08-13 Thread Alvaro Herrera
Stephen Frost wrote: * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Stephen Frost wrote: Alright, sounds like this is more-or-less the concensus. I'll see about making it happen shortly. Were you able to work on this? Apologies, I've been gone more-or-less all of July. I'm

Re: [HACKERS] proposal for 9.5: monitoring lock time for slow queries

2014-08-13 Thread Pavel Stehule
2014-08-13 15:22 GMT+02:00 MauMau maumau...@gmail.com: From: Pavel Stehule pavel.steh...@gmail.com 2014-08-13 13:59 GMT+02:00 MauMau maumau...@gmail.com: Are you concerned about the impactof collection overhead on the queries diagnosed? Maybe not light, but I'm optimistic. Oracle has the

Re: [HACKERS] strncpy is not a safe version of strcpy

2014-08-13 Thread Heikki Linnakangas
On 08/13/2014 04:31 PM, Kevin Grittner wrote: David Rowley dgrowle...@gmail.com wrote: I had a quick look at the usages of strncpy in master tonight and I've really just picked out the obviously broken ones for now. The other ones, on first look, either look safe, or require some more analysis

Re: [HACKERS] strncpy is not a safe version of strcpy

2014-08-13 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: I am concerned that failure to check for truncation could allow deletion of unexpected files or directories. I believe that we deal with this by the expedient of checking the lengths of tablespace paths in advance, when the tablespace is created.

Re: [HACKERS] strncpy is not a safe version of strcpy

2014-08-13 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: I am concerned that failure to check for truncation could allow deletion of unexpected files or directories. I believe that we deal with this by the expedient of checking the lengths of tablespace paths in advance,

Re: [HACKERS] option -T in pg_basebackup doesn't work on windows

2014-08-13 Thread MauMau
From: Amit Kapila amit.kapil...@gmail.com During my recent work on pg_basebackup, I noticed that -T option doesn't seem to work on Windows. The reason for the same is that while updating symlinks it doesn't consider that on Windows, junction points can be directories due to which it is not able

Re: [HACKERS] failures on barnacle (CLOBBER_CACHE_RECURSIVELY) because of memory leaks

2014-08-13 Thread Tom Lane
Tomas Vondra t...@fuzzy.cz writes: So after 83 days, the regression tests on barnacle completed, and it smells like another memory leak in CacheMemoryContext, similar to those fixed in 078b2ed on May 18. I've pushed fixes for the issues I was able to identify by running the create_view test.

[HACKERS] how to implement selectivity injection in postgresql

2014-08-13 Thread Rajmohan C
SELECT c1, c2, c3, FROM T1, T2, T3 WHERE T1.x = T2.x AND T2.y=T3.y AND T1.x = ? selectivity 0.1 AND T2.y ? selectivity 0.5 AND T3.z = ? selectivity 0.2 AND T3.w = ? I need to implement Selectivity injection as shown in above query in PostgreSQL

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Heikki Linnakangas
On 08/13/2014 04:15 PM, Heikki Linnakangas wrote: Hmm, thinking about this some more, there is one sensible way to split this patch: We can add the XLogReplayBuffer() function and rewrite all the redo routines to use it, without changing any WAL record formats or anything in the way the WAL

Re: [HACKERS] failures on barnacle (CLOBBER_CACHE_RECURSIVELY) because of memory leaks

2014-08-13 Thread Tomas Vondra
On 13.8.2014 17:52, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: So after 83 days, the regression tests on barnacle completed, and it smells like another memory leak in CacheMemoryContext, similar to those fixed in 078b2ed on May 18. I've pushed fixes for the issues I was able to

Re: [HACKERS] Proposal: Incremental Backup

2014-08-13 Thread Claudio Freire
On Tue, Aug 12, 2014 at 8:26 PM, Stephen Frost sfr...@snowman.net wrote: * Claudio Freire (klaussfre...@gmail.com) wrote: I'm not talking about malicious attacks, with big enough data sets, checksum collisions are much more likely to happen than with smaller ones, and incremental backups are

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-13 Thread Claudio Freire
On Tue, Aug 12, 2014 at 8:00 PM, Bruce Momjian br...@momjian.us wrote: On Mon, Aug 11, 2014 at 01:44:05PM -0700, Peter Geoghegan wrote: On Mon, Aug 11, 2014 at 1:01 PM, Stephen Frost sfr...@snowman.net wrote: We've got a clear example of someone, quite reasonably, expecting their JSONB

Re: [HACKERS] 9.5: Memory-bounded HashAgg

2014-08-13 Thread Tomas Vondra
On 13.8.2014 12:31, Tomas Vondra wrote: On 13 Srpen 2014, 7:02, Jeff Davis wrote: On Tue, 2014-08-12 at 14:58 +0200, Tomas Vondra wrote: (b) bad estimate of required memory - this is common for aggregates passing 'internal' state (planner uses some quite high defaults) Maybe some

Re: [HACKERS] how to implement selectivity injection in postgresql

2014-08-13 Thread Euler Taveira
On 13-08-2014 13:33, Rajmohan C wrote: SELECT c1, c2, c3, FROM T1, T2, T3 WHERE T1.x = T2.x AND T2.y=T3.y AND T1.x = ? selectivity 0.1 AND T2.y ? selectivity 0.5 AND T3.z = ? selectivity 0.2 AND T3.w = ? I need to implement Selectivity

Re: [HACKERS] how to implement selectivity injection in postgresql

2014-08-13 Thread Euler Taveira
On 13-08-2014 15:28, Rajmohan C wrote: Yeah. I have to do it for my academic research. Is it available in catalogs? It is to be computed at run time from the predicates in the query right? The selectivity information is available at runtime. See backend/optimizer/path/costsize.c. --

Re: [HACKERS] PostrgeSQL vs oracle doing 1 million sqrts am I doing it wrong?

2014-08-13 Thread Pavel Stehule
2014-08-08 2:13 GMT+02:00 Josh Berkus j...@agliodbs.com: On 08/07/2014 04:48 PM, Tom Lane wrote: plpgsql is not efficient at all about coercions performed as a side effect of assignments; if memory serves, it always handles them by converting to text and back. So basically the added cost

Re: [HACKERS] how to implement selectivity injection in postgresql

2014-08-13 Thread Jeff Janes
On Wed, Aug 13, 2014 at 9:33 AM, Rajmohan C csrajmo...@gmail.com wrote: SELECT c1, c2, c3, FROM T1, T2, T3 WHERE T1.x = T2.x AND T2.y=T3.y AND T1.x = ? selectivity 0.1 AND T2.y ? selectivity 0.5 AND T3.z = ? selectivity 0.2 AND T3.w = ?

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Alvaro Herrera
Heikki Linnakangas wrote: Here's a full version of this refactoring patch, all the rmgr's have now been updated to use XLogReplayBuffer(). I think this is a worthwhile change on its own, even if we drop the ball on the rest of the WAL format patch, because it makes the redo-routines more

Re: [HACKERS] WAL format and API changes (9.5)

2014-08-13 Thread Alvaro Herrera
Heikki Linnakangas wrote: Here's a full version of this refactoring patch, all the rmgr's have now been updated to use XLogReplayBuffer(). I think this is a worthwhile change on its own, even if we drop the ball on the rest of the WAL format patch, because it makes the redo-routines more

Re: [HACKERS] how to implement selectivity injection in postgresql

2014-08-13 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: On Wed, Aug 13, 2014 at 9:33 AM, Rajmohan C csrajmo...@gmail.com wrote: I need to implement Selectivity injection as shown in above query in PostgreSQL by which we can inject selectivity of each predicate or at least selectivity at relation level

Re: [HACKERS] Proposal to add a QNX 6.5 port to PostgreSQL

2014-08-13 Thread Baker, Keith [OCDUS Non-JJ]
Robert and Tom, I assume you guys are working on other priorities, so I did some locking experiments on QNX. I know fcntl() locking has downsides, but I think it deserves a second look: - it is POSIX, so should be fairly consistent across platforms (at least more consistent than lockf and

[HACKERS] What happened to jsonb's JENTRY_ISFIRST?

2014-08-13 Thread Tom Lane
jsonb.h claims that the high order bit of a JEntry word is set on the first element of a JEntry array. However, AFAICS, JBE_ISFIRST() is not used anywhere, which is a good thing because it refers to a constant JENTRY_ISFIRST that's not defined anywhere. Is this comment just a leftover from a

Re: [HACKERS] Proposal to add a QNX 6.5 port to PostgreSQL

2014-08-13 Thread Tom Lane
Baker, Keith [OCDUS Non-JJ] kbak...@its.jnj.com writes: I assume you guys are working on other priorities, so I did some locking experiments on QNX. I know fcntl() locking has downsides, but I think it deserves a second look: - it is POSIX, so should be fairly consistent across platforms (at

Re: [HACKERS] What happened to jsonb's JENTRY_ISFIRST?

2014-08-13 Thread Peter Geoghegan
On Wed, Aug 13, 2014 at 3:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: If the bit is unused now, should we be worrying about reclaiming it for better use? Like say allowing jsonb's to be larger than just a quarter of the maximum datum size? Commit d9daff0e0cb15221789e6c50d9733c8754c054fb removed

Re: [HACKERS] Proposal to add a QNX 6.5 port to PostgreSQL

2014-08-13 Thread Baker, Keith [OCDUS Non-JJ]
Tom, I appreciate your patience and explanation. (I am new to PostgreSQL hacking. I have read many old posts but not all of it sticks, sorry). I know QNX support is not high on your TODO list, so I am trying to keep the effort moving without being a distraction. Couldn't backend random code

Re: [HACKERS] replication commands and log_statements

2014-08-13 Thread Stephen Frost
* Amit Kapila (amit.kapil...@gmail.com) wrote: On Wed, Aug 13, 2014 at 4:24 AM, Stephen Frost sfr...@snowman.net wrote: Not entirely sure what you're referring to as 'internally generated' here.. Here 'internally generated' means that user doesn't execute those statements, rather the

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-13 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Seems we have two issues: 1) the header makes testing for compression likely to fail 2) use of pointers rather than offsets reduces compression potential I understand we are focusing on #1, but how much does compression reduce the storage size with

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-13 Thread Tom Lane
I wrote: That's a fair question. I did a very very simple hack to replace the item offsets with item lengths -- turns out that that mostly requires removing some code that changes lengths to offsets ;-). I then loaded up Larry's example of a noncompressible JSON value, and compared

Re: [HACKERS] replication commands and log_statements

2014-08-13 Thread Fujii Masao
On Thu, Aug 14, 2014 at 9:26 AM, Stephen Frost sfr...@snowman.net wrote: * Amit Kapila (amit.kapil...@gmail.com) wrote: On Wed, Aug 13, 2014 at 4:24 AM, Stephen Frost sfr...@snowman.net wrote: Not entirely sure what you're referring to as 'internally generated' here.. Here 'internally

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-13 Thread Andrew Dunstan
On 08/13/2014 09:01 PM, Tom Lane wrote: I wrote: That's a fair question. I did a very very simple hack to replace the item offsets with item lengths -- turns out that that mostly requires removing some code that changes lengths to offsets ;-). I then loaded up Larry's example of a

Re: [HACKERS] option -T in pg_basebackup doesn't work on windows

2014-08-13 Thread Michael Paquier
On Thu, Aug 14, 2014 at 12:50 AM, MauMau maumau...@gmail.com wrote: The code change appears correct, but the patch application failed against the latest source code. I don't know why. Could you confirm this? patching file src/bin/pg_basebackup/pg_basebackup.c Hunk #1 FAILED at 1119. 1 out

Re: [HACKERS] replication commands and log_statements

2014-08-13 Thread Michael Paquier
On Thu, Aug 14, 2014 at 10:40 AM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Aug 14, 2014 at 9:26 AM, Stephen Frost sfr...@snowman.net wrote: * Amit Kapila (amit.kapil...@gmail.com) wrote: On Wed, Aug 13, 2014 at 4:24 AM, Stephen Frost sfr...@snowman.net wrote: Not entirely sure what

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-13 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 08/13/2014 09:01 PM, Tom Lane wrote: That's a fair question. I did a very very simple hack to replace the item offsets with item lengths -- turns out that that mostly requires removing some code that changes lengths to offsets ;-). What does

Re: [HACKERS] Scaling shared buffer eviction

2014-08-13 Thread Amit Kapila
On Wed, Aug 13, 2014 at 4:25 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-08-13 09:51:58 +0530, Amit Kapila wrote: Overall, the main changes required in patch as per above feedback are: 1. add an additional counter for the number of those allocations not satisfied from the

[HACKERS] pg_dump bug in 9.4beta2 and HEAD

2014-08-13 Thread Joachim Wieland
I'm seeing an assertion failure with pg_dump -c --if-exists which is not ready to handle BLOBs it seems: pg_dump: pg_backup_archiver.c:472: RestoreArchive: Assertion `mark != ((void *)0)' failed. To reproduce: $ createdb test $ pg_dump -c --if-exists test (works, dumps empty database) $ psql

Re: [HACKERS] replication commands and log_statements

2014-08-13 Thread Amit Kapila
On Thu, Aug 14, 2014 at 5:56 AM, Stephen Frost sfr...@snowman.net wrote: * Amit Kapila (amit.kapil...@gmail.com) wrote: Oh, to be clear- I agree that logging of queries executed through SPI is desirable; I'd certainly like to have that capability without having to go through the auto-explain

Re: [HACKERS] strncpy is not a safe version of strcpy

2014-08-13 Thread Noah Misch
On Wed, Aug 13, 2014 at 10:21:50AM -0400, Tom Lane wrote: Kevin Grittner kgri...@ymail.com writes: I am concerned that failure to check for truncation could allow deletion of unexpected files or directories. I believe that we deal with this by the expedient of checking the lengths of

[HACKERS] proposal: allow to specify result tupdesc and mode in SPI API

2014-08-13 Thread Pavel Stehule
Hello we have not possibility to simple specify result types in SPI API functions. Planner has this functionality - see transformInsertRow function, but it is not visible from SPI. A new function should to look like: SPIPlanPtr SPI_prepare_params_rettupdesc(const char *src,

Compute attr_needed for child relations (was Re: [HACKERS] inherit support for foreign tables)

2014-08-13 Thread Etsuro Fujita
(2014/08/08 18:51), Etsuro Fujita wrote: (2014/06/30 22:48), Tom Lane wrote: I wonder whether it isn't time to change that. It was coded like that originally only because calculating the values would've been a waste of cycles at the time. But this is at least the third place where it'd be

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-13 Thread MauMau
I fixed some minor mistakes. Regards MauMau pg_copy_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers