Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-15 Thread Heikki Linnakangas
On 01/15/2014 08:46 AM, Jim Nasby wrote: On 1/14/14, 7:41 AM, Magnus Hagander wrote: Yes, it would be necessary to scan the whole database as the LSN to be checked is kept in PageHeaderData :). Perhaps it is not that performant, but my initial thought was that perhaps the amount of

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Heikki Linnakangas
On 01/15/2014 06:01 AM, Jim Nasby wrote: For the sake of completeness... it's theoretically silly that Postgres is doing all this stuff with WAL when the filesystem is doing something very similar with it's journal. And an SSD drive (and next generation spinning rust) is doing the same thing

Re: [HACKERS] nested hstore patch - FailedAssertion(!(value-array.nelems == 1)

2014-01-15 Thread Oleg Bartunov
https://github.com/feodor/postgres On Wed, Jan 15, 2014 at 11:10 AM, Erik Rijkers e...@xs4all.nl wrote: On Wed, January 15, 2014 08:01, Oleg Bartunov wrote: It doesn't crashed in the last version in our repository. =# select 'x'::hstore || ('a=1':: hstore) ; ?column? ---

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 7:07 AM, Florian Pflug wrote: On Jan15, 2014, at 01:34 , Marko Tiikkaja ma...@joh.to wrote: It's me again, trying to find a solution to the most common mistakes I make. This time it's accidental shadowing of variables, especially input variables. I've wasted several hours banging

Re: [HACKERS] Failed assertion root-hasLateralRTEs on initsplan.c

2014-01-15 Thread Dean Rasheed
On 9 January 2014 15:33, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 7, 2014 at 1:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: The next question is if we should allow it with LATERAL. That would essentially be treating subscriber as having implicitly

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Mel Gorman
(This thread is now massive and I have not read it all yet. If anything I say has already been discussed then whoops) On Tue, Jan 14, 2014 at 12:09:46PM +1100, Dave Chinner wrote: On Mon, Jan 13, 2014 at 09:29:02PM +, Greg Stark wrote: On Mon, Jan 13, 2014 at 9:12 PM, Andres Freund

[HACKERS] identify table oid for an AggState during plan tree initialization

2014-01-15 Thread Masterprojekt Naumann1
Hi, during the initialization of the nodes in the plan tree (in ExecInitNode in the file execProcnode.c) I want to find out for a node with the type T_Agg which table will be aggregated. I tried the following: resultAsAggState = ExecInitAgg((Agg *) node, estate, eflags); if (resultAsAggState) {

Re: [HACKERS] [bug fix] PostgreSQL fails to start on Windows if it crashes after tablespace creation

2014-01-15 Thread Asif Naeem
I did worked on testing the patch on Windows, test scenario mentioned above thread is reproducible and the provided patch resolve the issue. In case of junction or directory unlink function (deprecatedhttp://msdn.microsoft.com/en-us/library/ms235350.aspx) returns -1 with errno “EACCES” (i.e.

Re: [HACKERS] identify table oid for an AggState during plan tree initialization

2014-01-15 Thread Masterprojekt Naumann1
2014/1/15 Masterprojekt Naumann1 mpws201...@gmail.com Hi, during the initialization of the nodes in the plan tree (in ExecInitNode in the file execProcnode.c) I want to find out for a node with the type T_Agg which table will be aggregated. I tried the following: resultAsAggState =

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Mel Gorman
On Tue, Jan 14, 2014 at 09:30:19AM -0800, Jeff Janes wrote: What's not so simple, is figuring out what policy to use. Remember, you cannot tell the kernel to put some page in its page cache without reading it or writing it. So, once you make the kernel forget a page, evicting it from

Re: [HACKERS] plpgsql.consistent_into

2014-01-15 Thread Pavel Stehule
2014/1/15 Jim Nasby j...@nasby.net On 1/14/14, 11:15 AM, Tom Lane wrote: How about: (a) = SELECT 1; (a, b) = SELECT 1, 2; (a, b) = INSERT INTO foo RETURNING col1, col2; Same semantics: TOO_MANY_ROWS on rows 1, sets FOUND and row_count. AFAICT this can be parsed unambiguously,

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Pavel Stehule
2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 7:07 AM, Florian Pflug wrote: On Jan15, 2014, at 01:34 , Marko Tiikkaja ma...@joh.to wrote: It's me again, trying to find a solution to the most common mistakes I make. This time it's accidental shadowing of variables, especially input

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 11:20 AM, Pavel Stehule wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to Hmm. How about: plpgsql.warnings = 'all' # enable all warnings, defauls to the empty list, i.e. no warnings plpgsql.warnings = 'shadow, unused' # enable just shadow and unused warnings

Re: [HACKERS] identify table oid for an AggState during plan tree initialization

2014-01-15 Thread Ashutosh Bapat
Hi Cathleen, An aggregate can be working on more than one table e.g. select count(*) from a, b, c where a.c1 = b.c1 and b.c1 = c.c1; In such a case, which table would you like to be reported? IOW, it doesn't look to be sensible to attach and aggregate with a table. If you can explain what you

Re: [HACKERS] nested hstore patch - FailedAssertion(!(value-array.nelems == 1)

2014-01-15 Thread Erik Rijkers
On Wed, January 15, 2014 09:46, Oleg Bartunov wrote: On Wed, January 15, 2014 08:01, Oleg Bartunov wrote: It doesn't crashed in the last version in our repository. =# select 'x'::hstore || ('a=1':: hstore) ; ?column? --- x, a, 1 (1 row) OK, shall I use that repository

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Pavel Stehule
2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 11:20 AM, Pavel Stehule wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to Hmm. How about: plpgsql.warnings = 'all' # enable all warnings, defauls to the empty list, i.e. no warnings plpgsql.warnings = 'shadow, unused' # enable just

Re: [HACKERS] ISN extension bug? (with patch)

2014-01-15 Thread Fabien COELHO
With this caveat, the one-liner patch (4 characters removed) reattached does compile for me: Thanks, applied. Thanks! -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 11:33 AM, Pavel Stehule wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to I agree, it's better to include the word compiler in the GUC name. But do we really need WARNING, ERROR and FATAL levels though? Would WARNING and ERROR not be enough? I am not strong in level names - and it is

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Pavel Stehule
2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 11:33 AM, Pavel Stehule wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to I agree, it's better to include the word compiler in the GUC name. But do we really need WARNING, ERROR and FATAL levels though? Would WARNING and ERROR not be enough?

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Mel Gorman
On Wed, Jan 15, 2014 at 09:44:21AM +, Mel Gorman wrote: SNIP H. What happens if the process crashes after pinning the dirty pages? How do we even know what process pinned the dirty pages so we can clean up after it? What happens if the same page is pinned by multiple processes?

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Hannu Krosing
On 01/14/2014 06:12 PM, Robert Haas wrote: This would be pretty similar to copy-on-write, except without the copying. It would just be forget-from-the-buffer-pool-on-write. +1 A version of this could probably already be implement using MADV_DONTNEED and MADV_WILLNEED Thet is, just after

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Hannu Krosing
On 01/15/2014 12:16 PM, Hannu Krosing wrote: On 01/14/2014 06:12 PM, Robert Haas wrote: This would be pretty similar to copy-on-write, except without the copying. It would just be forget-from-the-buffer-pool-on-write. +1 A version of this could probably already be implement using

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Florian Pflug
On Jan15, 2014, at 10:08 , Marko Tiikkaja ma...@joh.to wrote: On 1/15/14 7:07 AM, Florian Pflug wrote: On Jan15, 2014, at 01:34 , Marko Tiikkaja ma...@joh.to wrote: It's me again, trying to find a solution to the most common mistakes I make. This time it's accidental shadowing of variables,

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Florian Pflug
On Jan15, 2014, at 11:20 , Pavel Stehule pavel.steh...@gmail.com wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 7:07 AM, Florian Pflug wrote: On Jan15, 2014, at 01:34 , Marko Tiikkaja ma...@joh.to wrote: It's me again, trying to find a solution to the most common mistakes I make.

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Mel Gorman
On Mon, Jan 13, 2014 at 02:19:56PM -0800, James Bottomley wrote: On Mon, 2014-01-13 at 22:12 +0100, Andres Freund wrote: On 2014-01-13 12:34:35 -0800, James Bottomley wrote: On Mon, 2014-01-13 at 14:32 -0600, Jim Nasby wrote: Well, if we were to collaborate with the kernel community on

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Pavel Stehule
2014/1/15 Florian Pflug f...@phlo.org On Jan15, 2014, at 11:20 , Pavel Stehule pavel.steh...@gmail.com wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 7:07 AM, Florian Pflug wrote: On Jan15, 2014, at 01:34 , Marko Tiikkaja ma...@joh.to wrote: It's me again, trying to find a

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Florian Pflug
On Jan15, 2014, at 13:08 , Pavel Stehule pavel.steh...@gmail.com wrote: 2014/1/15 Florian Pflug f...@phlo.org On Jan15, 2014, at 11:20 , Pavel Stehule pavel.steh...@gmail.com wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to plpgsql.warnings = 'all' # enable all warnings, defauls to the empty

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2014-01-15 Thread Amit Kapila
On Wed, Jan 15, 2014 at 5:58 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Fri, Jan 10, 2014 at 9:12 PM, Robert Haas robertmh...@gmail.com wrote: Performance Data - Non-default settings: autovacuum =off checkpoint_segments =128 checkpoint_timeout = 10min

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 1:23 PM, Florian Pflug wrote: The fact that it's named plpgsql.warnings already clearly documents that this only affects plpgsql. But whether a particular warning is emitted during compilation or during execution it largely irrelevant, I think. For example, if we called this

Re: [HACKERS] identify table oid for an AggState during plan tree initialization

2014-01-15 Thread Masterprojekt Naumann1
2014/1/15 Ashutosh Bapat ashutosh.ba...@enterprisedb.com Hi Cathleen, An aggregate can be working on more than one table e.g. select count(*) from a, b, c where a.c1 = b.c1 and b.c1 = c.c1; In such a case, which table would you like to be reported? IOW, it doesn't look to be sensible to

Re: [HACKERS] WAL Rate Limiting

2014-01-15 Thread Magnus Hagander
On Wed, Jan 15, 2014 at 3:20 AM, Simon Riggs si...@2ndquadrant.com wrote: We've discussed previously the negative impact of large bulk operations, especially wrt WAL writes. Patch here allows maintenance operations to have their WAL generation slowed down as a replication lag prevention

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Florian Pflug
On Jan15, 2014, at 13:32 , Marko Tiikkaja ma...@joh.to wrote: On 1/15/14 1:23 PM, Florian Pflug wrote: The fact that it's named plpgsql.warnings already clearly documents that this only affects plpgsql. But whether a particular warning is emitted during compilation or during execution it

Re: [HACKERS] identify table oid for an AggState during plan tree initialization

2014-01-15 Thread Ashutosh Bapat
Hmm, ok, this is slighly involved. Is it important that you should do this at the execution time? At planner level where Agg node is involved, one can look for Aggref node and traverse down the args list to find any vars are involved in aggregation. You can find Var nodes involved in an Aggref by

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 2:00 PM, Florian Pflug wrote: On Jan15, 2014, at 13:32 , Marko Tiikkaja ma...@joh.to wrote: On 1/15/14 1:23 PM, Florian Pflug wrote: The fact that it's named plpgsql.warnings already clearly documents that this only affects plpgsql. But whether a particular warning is emitted

[HACKERS] What's the condition of bug PANIC: WAL contains references to invalid pages?

2014-01-15 Thread MauMau
Hello, Please tell me a bit about the following bug which has just been solved. I wish this is exactly what has been annoying for a year. Hot standby 9.2.6 - 9.2.6 PANIC: WAL contains references to invalid pages

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Dave Chinner
On Tue, Jan 14, 2014 at 09:54:20PM -0600, Jim Nasby wrote: On 1/14/14, 3:41 PM, Dave Chinner wrote: On Tue, Jan 14, 2014 at 09:40:48AM -0500, Robert Haas wrote: On Mon, Jan 13, 2014 at 5:26 PM, Mel Gorman mgor...@suse.de wrote: Whether the problem is with the system call or the programmer is

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Jan Kara
On Wed 15-01-14 10:27:26, Heikki Linnakangas wrote: On 01/15/2014 06:01 AM, Jim Nasby wrote: For the sake of completeness... it's theoretically silly that Postgres is doing all this stuff with WAL when the filesystem is doing something very similar with it's journal. And an SSD drive (and next

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Pavel Stehule
2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 2:00 PM, Florian Pflug wrote: On Jan15, 2014, at 13:32 , Marko Tiikkaja ma...@joh.to wrote: On 1/15/14 1:23 PM, Florian Pflug wrote: The fact that it's named plpgsql.warnings already clearly documents that this only affects plpgsql. But

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Jan Kara
On Wed 15-01-14 12:16:50, Hannu Krosing wrote: On 01/14/2014 06:12 PM, Robert Haas wrote: This would be pretty similar to copy-on-write, except without the copying. It would just be forget-from-the-buffer-pool-on-write. +1 A version of this could probably already be implement using

Re: [HACKERS] trgm regex index peculiarity

2014-01-15 Thread Alexander Korotkov
On Fri, Jun 21, 2013 at 5:39 PM, Erik Rijkers e...@xs4all.nl wrote: On Fri, June 21, 2013 15:11, Alexander Korotkov wrote: On Fri, Jun 21, 2013 at 2:40 PM, Erik Rijkers e...@xs4all.nl wrote: On Fri, June 21, 2013 05:25, Tom Lane wrote: Erik Rijkers e...@xs4all.nl writes: In a 112 MB

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Hannu Krosing
On 01/15/2014 02:01 PM, Jan Kara wrote: On Wed 15-01-14 12:16:50, Hannu Krosing wrote: On 01/14/2014 06:12 PM, Robert Haas wrote: This would be pretty similar to copy-on-write, except without the copying. It would just be forget-from-the-buffer-pool-on-write. +1 A version of this could

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 4:23 PM, James Bottomley james.bottom...@hansenpartnership.com wrote: Yes, that's what I was thinking: it's a cache. About how many files comprise this cache? Are you thinking it's too difficult for every process to map the files? No, I'm thinking that would throw

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 5:23 PM, Dave Chinner da...@fromorbit.com wrote: By default, background writeback doesn't start until 10% of memory is dirtied, and on your machine that's 25GB of RAM. That's way to high for your workload. It appears to me that we are seeing large memory machines much

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 2:27 PM, Pavel Stehule wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to Yeah, me neither, it's just something that needs to be communicated very clearly. So probably just a list plpgsql.warnings would be the most appropriate then. I am thinking so the name is not good. Changing

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Pavel Stehule
2014/1/15 Marko Tiikkaja ma...@joh.to On 1/15/14 2:27 PM, Pavel Stehule wrote: 2014/1/15 Marko Tiikkaja ma...@joh.to Yeah, me neither, it's just something that needs to be communicated very clearly. So probably just a list plpgsql.warnings would be the most appropriate then. I am

[HACKERS] Re: Linux kernel impact on PostgreSQL performance (summary v1 2014-1-15)

2014-01-15 Thread Mel Gorman
One assumption would be that Postgres is perfectly happy with the current kernel behaviour in which case our discussion here is done. It has been demonstrated that this statement was farcical. The thread is massive just from interaction with the LSF/MM program committee. I'm hoping that there

Re: [HACKERS] [bug fix] PostgreSQL fails to start on Windows if it crashes after tablespace creation

2014-01-15 Thread MauMau
From: Asif Naeem anaeem...@gmail.com As you have followed destroy_tablespace_directories() function, Is there any specific reason not to use same logic to detect type of the file/link i.e. “(lstat(linkloc, st) == 0 S_ISDIR(st.st_mode))”, It also seems have more appropriate error message i.e.

Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2014-01-15 Thread Simon Riggs
On 1 August 2013 01:53, Noah Misch n...@leadboat.com wrote: On Mon, Jul 15, 2013 at 05:50:40PM +0100, Simon Riggs wrote: On 15 July 2013 15:06, Robert Haas robertmh...@gmail.com wrote: Generally, the question on the table is: to what extent do MVCC catalog scans make the world safe for

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Heikki Linnakangas
On 01/15/2014 07:50 AM, Dave Chinner wrote: However, the first problem is dealing with the IO storm problem on fsync. Then we can measure the effect of spreading those writes out in time and determine what triggers read starvations (if they are apparent). The we can look at whether IO scheduling

Re: [HACKERS] Portal suddenly disappears?

2014-01-15 Thread Tatsuo Ishii
I'm confused too. Surely there are lots of ways a portal could get dropped, but most of them would have left traces in the postmaster log, I'd think, since you evidently have log_statement == LOGSTMT_ALL. What was log_min_messages set to? I don't have it at this moment. I requested the

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 01/15/2014 07:50 AM, Dave Chinner wrote: FWIW [and I know you're probably sick of hearing this by now], but the blk-io throttling works almost perfectly with applications that use direct IO. For checkpoint writes, direct I/O actually

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 4:35 AM, Jan Kara j...@suse.cz wrote: Filesystems could in theory provide facility like atomic write (at least up to a certain size say in MB range) but it's not so easy and when there are no strong usecases fs people are reluctant to make their code more complex

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 4:44 AM, Mel Gorman mgor...@suse.de wrote: That applies if the dirty pages are forced to be kept dirty. You call this pinned but pinned has special meaning so I would suggest calling it something like dirty-sticky pages. It could be the case that such hinting will have

Re: [HACKERS] shared memory message queues

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 9:28 PM, Peter Eisentraut pete...@gmx.net wrote: Something is causing this new compiler warning: setup.c: In function ‘setup_dynamic_shared_memory’: setup.c:102:3: error: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 2 has type ‘Size’

Re: [HACKERS] shared memory message queues

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 9:53 PM, Kevin Grittner kgri...@ymail.com wrote: I'm not seeing that one but I am now getting these: setup.c: In function ‘test_shm_mq_setup’: setup.c:65:25: warning: ‘outq’ may be used uninitialized in this function [-Wmaybe-uninitialized] setup.c:66:24: warning:

Re: [HACKERS] shared memory message queues

2014-01-15 Thread Andres Freund
On 2014-01-15 10:19:32 -0500, Robert Haas wrote: On Tue, Jan 14, 2014 at 9:28 PM, Peter Eisentraut pete...@gmx.net wrote: Something is causing this new compiler warning: setup.c: In function ‘setup_dynamic_shared_memory’: setup.c:102:3: error: format ‘%llu’ expects argument of type ‘long

Re: [HACKERS] COPY and Volatile default expressions

2014-01-15 Thread Simon Riggs
On 16 April 2013 14:37, Simon Riggs si...@2ndquadrant.com wrote: On 16 April 2013 13:57, Heikki Linnakangas hlinnakan...@vmware.com wrote: You still need to check the args, if the function is nextval, otherwise you incorrectly perform the optimization for something like

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Mel Gorman
On Wed, Jan 15, 2014 at 10:16:27AM -0500, Robert Haas wrote: On Wed, Jan 15, 2014 at 4:44 AM, Mel Gorman mgor...@suse.de wrote: That applies if the dirty pages are forced to be kept dirty. You call this pinned but pinned has special meaning so I would suggest calling it something like

Re: [HACKERS] plpgsql.warn_shadow

2014-01-15 Thread Marko Tiikkaja
On 1/15/14 3:09 PM, Pavel Stehule wrote: You first should to say, what is warning and why it is only warning and not error. Personally, I'm a huge fan of the computer telling me that I might have made a mistake. But on the other hand, I also really hate it when the computer gets in my way

Re: [HACKERS] Soften pg_[start|stop]_backup to allow them on a standby?

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 7:54 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-14 12:31:09 +0900, Michael Paquier wrote: Currently, pg_start_backup and pg_stop_backup cannot run on a standby because it is not possible to write a backup_label file to disk, because of the nature of a

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 10:53 AM, Mel Gorman mgor...@suse.de wrote: I realise that now and sorry for the noise. I later read the parts of the thread that covered the strict ordering requirements and in a summary mail I split the requirements in two. In one, there are dirty sticky pages that

Re: [HACKERS] Soften pg_[start|stop]_backup to allow them on a standby?

2014-01-15 Thread Andres Freund
On 2014-01-15 11:19:52 -0500, Robert Haas wrote: On Tue, Jan 14, 2014 at 7:54 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-14 12:31:09 +0900, Michael Paquier wrote: Currently, pg_start_backup and pg_stop_backup cannot run on a standby because it is not possible to write a

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Stephen Frost
* Claudio Freire (klaussfre...@gmail.com) wrote: Yes, that's basically zero-copy reads. It could be done. The kernel can remap the page to the physical page holding the shared buffer and mark it read-only, then expire the buffer and transfer ownership of the page if any page fault happens.

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 4:13 PM, Simon Riggs si...@2ndquadrant.com wrote: On 8 January 2014 08:33, Simon Riggs si...@2ndquadrant.com wrote: VACUUM cleans up blocks, which is nice because it happens offline in a lazy manner. We also make SELECT clean up blocks as it goes. That is useful in

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Claudio Freire
On Wed, Jan 15, 2014 at 1:35 PM, Stephen Frost sfr...@snowman.net wrote: And there's a nice bingo. Had forgotten about KSM. KSM could help lots. I could try to see of madvising shared_buffers as mergeable helps. But this should be an automatic case of KSM - ie, when reading into a

Re: [HACKERS] shared memory message queues

2014-01-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jan 14, 2014 at 9:28 PM, Peter Eisentraut pete...@gmx.net wrote: Something is causing this new compiler warning: setup.c: In function ‘setup_dynamic_shared_memory’: setup.c:102:3: error: format ‘%llu’ expects argument of type ‘long long

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-15 Thread Robert Haas
This 0001 patch, to log running transactions more frequently, has been pending for a long time now, and I haven't heard any objections, so I've gone ahead and committed that part. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] shared memory message queues

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 12:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jan 14, 2014 at 9:28 PM, Peter Eisentraut pete...@gmx.net wrote: Something is causing this new compiler warning: setup.c: In function ‘setup_dynamic_shared_memory’:

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I think that the bottom line is that we're not likely to make massive changes to the way that we do block caching now. Even if some other scheme could work much better on Linux (and so far I'm unconvinced that any of the proposals made here would in

Re: [HACKERS] Deprecations in authentication

2014-01-15 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: One thing I noticed - in MSVC, the config parameter krb5 (equivalent of the removed --with-krb5) enabled *both* krb5 and gssapi, and there is no separate config parameter for gssapi. Do we want to rename that one to gss, or do we want to keep it as

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Claudio Freire
On Wed, Jan 15, 2014 at 2:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I think that the bottom line is that we're not likely to make massive changes to the way that we do block caching now. Even if some other scheme could work much better on Linux (and

Re: [HACKERS]

2014-01-15 Thread Fabrízio de Royes Mello
On Thu, Jan 9, 2014 at 2:50 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jan 7, 2014 at 10:55 PM, Dilip kumar dilip.ku...@huawei.com wrote: Below attached patch is implementing following todo item.. machine-readable pg_controldata?

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-15 Thread Robert Haas
Review of patch 0002: - I think you should just regard ReplicationSlotCtlLock as protecting the name and active flags of every slot. ReplicationSlotCreate() would then not need to mess with the spinlocks at all, and ReplicationSlotAcquire and ReplicationSlotDrop get a bit simpler too I think.

Re: [HACKERS] drop duplicate buffers in OS

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 1:53 AM, KONDO Mitsumasa kondo.mitsum...@lab.ntt.co.jp wrote: I create patch that can drop duplicate buffers in OS using usage_count alogorithm. I have developed this patch since last summer. This feature seems to be discussed in hot topic, so I submit it more faster

Re: [HACKERS] Turning recovery.conf into GUCs

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 2:06 AM, Jaime Casanova ja...@2ndquadrant.com wrote: On Wed, Jan 15, 2014 at 2:00 AM, Jaime Casanova ja...@2ndquadrant.com wrote: On Mon, Nov 18, 2013 at 12:27 PM, Andres Freund and...@2ndquadrant.com wrote: * Maybe we should rename names like pause_at_recovery_target

Re: [HACKERS] tests for client programs

2014-01-15 Thread Peter Eisentraut
On 1/15/14, 1:46 AM, Erik Rijkers wrote: The seems to be a dependency on IPC::Run I can install that, of course... but I suppose you want to make this work without that. No, IPC::Run will be required. It looked like it was part of the default installation where I tested, but apparently

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Stephen Frost
* Claudio Freire (klaussfre...@gmail.com) wrote: But, still, the implementation is very similar to what postgres needs: sharing a physical page for two distinct logical pages, efficiently, with efficient copy-on-write. Agreed, except that KSM seems like it'd be slow/lazy about it and I'm

[HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Josh Berkus
Hackers, ALTER SYSTEM SET has been committed and recovery.conf GUCs are being reviewed. I'm going to make a last case for conf.d in relation to these two patches before 9.4 goes out the door. In 9.3, we added support for a config directory (conf.d), but have it disabled by default. For tool

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-15 Thread Robert Haas
On Tue, Jan 14, 2014 at 4:16 AM, David Rowley dgrowle...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, David Rowley dgrowle...@gmail.com wrote: I think unless anyone has some objections I'm going to remove the inverse transition for SUM(numeric) and modify the documents to tell the user

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Claudio Freire
On Wed, Jan 15, 2014 at 3:41 PM, Stephen Frost sfr...@snowman.net wrote: * Claudio Freire (klaussfre...@gmail.com) wrote: But, still, the implementation is very similar to what postgres needs: sharing a physical page for two distinct logical pages, efficiently, with efficient copy-on-write.

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Stephen Frost
Josh, * Josh Berkus (j...@agliodbs.com) wrote: In 9.3, we added support for a config directory (conf.d), but have it disabled by default. For tool authors, this makes conf.d useless since you never know, on any given installation, whether it will be present/enabled or not. While we don't

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Jan Kara
On Wed 15-01-14 14:38:44, Hannu Krosing wrote: On 01/15/2014 02:01 PM, Jan Kara wrote: On Wed 15-01-14 12:16:50, Hannu Krosing wrote: On 01/14/2014 06:12 PM, Robert Haas wrote: This would be pretty similar to copy-on-write, except without the copying. It would just be

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2014-01-15 Thread Robert Haas
On Wed, Jan 15, 2014 at 7:28 AM, Amit Kapila amit.kapil...@gmail.com wrote: Unpatched --- testname | wal_generated | duration --+--+--

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Peter Eisentraut
On 1/15/14, 1:53 PM, Josh Berkus wrote: I'm particularly thinking about this in relation to the merger of recovery.conf and postgresql.conf. There are several tools already (RepMgr, OminPITR, HandyRep, pgPool, etc.) which manage recovery.conf separately from postgresql.conf. These tools will

Re: [HACKERS] Add %z support to elog/ereport?

2014-01-15 Thread Andres Freund
On 2013-12-08 01:32:45 +0100, Andres Freund wrote: On 2013-12-06 09:54:59 -0500, Peter Eisentraut wrote: On 11/11/13, 12:01 PM, Tom Lane wrote: I do recall Peter saying that the infrastructure knows how to verify conversion specs in translated strings, so it would have to be aware of

Re: [HACKERS] tests for client programs

2014-01-15 Thread Erik Rijkers
On Wed, January 15, 2014 06:30, Peter Eisentraut wrote: As we all know, the client programs (src/bin/) don't have any real test So I wrote something. I chose to use Perl-based tools, prove and Test::More, because those are [ 0001-Add-TAP-tests-for-client-programs.patch ] 32 k I gave this

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Josh Berkus
On 01/15/2014 11:10 AM, Stephen Frost wrote: I don't buy this argument one bit- certainly, on Debian, the defaults are overridden for a number of variables already and could be done to enable a conf.d directory as well. Directory creation would, of course, also be able to be handled by the

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Stephen Frost
Josh, * Josh Berkus (j...@agliodbs.com) wrote: However, Debian is *never* going to add conf.d to the packages if we don't recommend it as an upstream project. And, frankly, I use the apt.postgresql.org packages anyway, which would certainly include anything which was decided on this list.

Re: [HACKERS] Changeset Extraction v7.0 (was logical changeset generation)

2014-01-15 Thread Andres Freund
Hi, On 2014-01-15 13:28:25 -0500, Robert Haas wrote: - I think you should just regard ReplicationSlotCtlLock as protecting the name and active flags of every slot. ReplicationSlotCreate() would then not need to mess with the spinlocks at all, and ReplicationSlotAcquire and

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-15 Thread Florian Pflug
On Jan15, 2014, at 19:56 , Robert Haas robertmh...@gmail.com wrote: It strikes me that for numeric what you really need is to just tell the sum operation, whether through a parameter or otherwise, how many decimal places to show in the output. Obviously that's not a practical change for sum()

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-15 Thread Tom Lane
Florian Pflug f...@phlo.org writes: I'm currently thinking the best way forward is to get a basic patch without any NUMERIC stuff committed, and to revisit this after that's done. +1. I liked Robert's suggestion that the fast aggregates be implemented as a contrib module rather than directly

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On 1/15/14, 1:53 PM, Josh Berkus wrote: Yes, I'm also arguing that postgresql.auto.conf should go into conf.d. I said I'd bring that up again after ALTER SYSTEM SET was committed, and here it is. Independent of the above, I don't agree with that.

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Peter Eisentraut
On 1/15/14, 4:35 PM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On 1/15/14, 1:53 PM, Josh Berkus wrote: Yes, I'm also arguing that postgresql.auto.conf should go into conf.d. I said I'd bring that up again after ALTER SYSTEM SET was committed, and here it is. Independent of

Re: [HACKERS] Why conf.d should be default, and auto.conf and recovery.conf should be in it

2014-01-15 Thread Peter Eisentraut
On 1/15/14, 3:01 PM, Josh Berkus wrote: Three issues: a) if postgresql is still going to look for a recovery.conf file in the usual place, but we are changing the names and meaning of some of the parameters, then aren't we making the upgrade problem much worse? That assumes that we are

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-01-15 Thread Simon Riggs
On 15 January 2014 16:47, Robert Haas robertmh...@gmail.com wrote: There would be a postgresql.conf parameter prune_cost_limit, as well as a table level parameter that would prevent pruning except via VACUUM. This will help in these ways * Reduce write I/O from SELECTs and pg_dump -

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Jeff Janes
On Wed, Jan 15, 2014 at 7:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: On 01/15/2014 07:50 AM, Dave Chinner wrote: FWIW [and I know you're probably sick of hearing this by now], but the blk-io throttling works almost perfectly with

Re: [HACKERS] Backup throttling

2014-01-15 Thread Alvaro Herrera
Alvaro Herrera escribió: Antonin Houska escribió: Thanks for checking. The new version addresses your findings. I gave this patch a look. BTW I also moved the patch the commitfest currently running, and set it waiting-on-author. Your move. -- Álvaro Herrera

Re: [HACKERS] trgm regex index peculiarity

2014-01-15 Thread Tom Lane
Alexander Korotkov aekorot...@gmail.com writes: Revised version of patch with necessary comments. I looked at this patch a bit. It seems like this: + *BLANK_COLOR_SIZE - How much blank character is more frequent than + * other character in average + #define

Re: [HACKERS] CREATE TABLESPACE WITH

2014-01-15 Thread Vik Fearing
On 01/15/2014 03:07 AM, Michael Paquier wrote: I just had a quick look at this patch, no testing at all. Thank you for looking at it. I am seeing that regression tests are still missing here, they should be added in src/test/regress/input/tablespace.source. New patch attached, with

Re: [HACKERS] CREATE FOREIGN TABLE ( ... LIKE ... )

2014-01-15 Thread Vik Fearing
On 11/24/2013 02:03 AM, Vik Fearing wrote: On 10/15/2013 07:50 AM, David Fetter wrote: On Mon, Oct 07, 2013 at 11:16:56PM -0700, David Fetter wrote: Folks, Please find attached a patch implementing and documenting, to some extent, $subject. I did this in aid of being able to import SQL

  1   2   >