Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-13 Thread Craig Ringer
On 01/09/2014 11:19 PM, Tom Lane wrote: Dean Rasheed dean.a.rash...@gmail.com writes: My first thought was that it should just preprocess any security barrier quals in subquery_planner() in the same way as other quals are preprocessed. But thinking about it further, those quals are destined

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Heikki Linnakangas
On 01/11/2014 12:40 AM, Peter Geoghegan wrote: My problem is that in general I'm not sold on the actual utility of making this kind of row locking work with exclusion constraints. I'm sincerely having a hard time thinking of a practical use-case (although, as I've said, I want to make it work

Re: [HACKERS] [PATCH] Filter error log statements by sqlstate

2014-01-13 Thread Jeevan Chalke
Hi Oskari, I had a quick look over the patch (Not compiled though). Here are few comments on the changes: 1. Documentation is missing and thus becomes difficult to understand what exactly you are trying to do. Or in other words, user will be uncertain about using it more efficiently. 2. Some

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Heikki Linnakangas
On 01/11/2014 12:39 PM, Peter Geoghegan wrote: In any case, my patch is bound to win decisively for the other extreme, the insert-only case, because the overhead of doing an index scan first is always wasted there with your approach, and the overhead of extended btree leaf page locking has been

Re: [HACKERS] ECPG regression tests generating warnings

2014-01-13 Thread Michael Meskes
On Sun, Jan 12, 2014 at 08:28:57AM -0800, Kevin Grittner wrote: desc.pgc:55: WARNING: descriptor outdesc does not exist desc.pgc:86: WARNING: descriptor outdesc does not exist Thanks, I didn't notice, fixed. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot

Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-13 Thread Dean Rasheed
On 12 January 2014 10:12, Craig Ringer cr...@2ndquadrant.com wrote: On 01/09/2014 06:48 PM, Dean Rasheed wrote: On 8 January 2014 10:19, Dean Rasheed dean.a.rash...@gmail.com wrote: The assertion failure with inheritance and sublinks is a separate issue --- adjust_appendrel_attrs() is not

Re: [HACKERS] [PATCH] Filter error log statements by sqlstate

2014-01-13 Thread Oskari Saarenmaa
Hi, On 13/01/14 10:26, Jeevan Chalke wrote: 1. Documentation is missing and thus becomes difficult to understand what exactly you are trying to do. Or in other words, user will be uncertain about using it more efficiently. I figured I'd write documentation for this if it looks like a useful

Re: [HACKERS] [BUG] Archive recovery failure on 9.3+.

2014-01-13 Thread Heikki Linnakangas
On 01/09/2014 10:55 PM, Josh Berkus wrote: On 01/09/2014 12:05 PM, Heikki Linnakangas wrote: Actually, why is the partially-filled 00010002 file archived in the first place? Looking at the code, it's been like that forever, but it seems like a bad idea. If the original server

Re: [HACKERS] generic pseudotype IO functions?

2014-01-13 Thread Peter Eisentraut
On Mon, 2014-01-06 at 17:36 +0100, Andres Freund wrote: FWIW, I am perfectly fine with duplicating the functions for now - I just thought that that might not be the best way but I didn't (and still don't) have a strong opinion. Could we just put 0 in for the functions' OID and have code

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Rajeev rastogi
On Sun, Jan 12, Amit Kapila wrote: How would that work? Would it be a tool in contrib? There already is a timeout, so if a tool checked more frequently than the timeout, it should work. The durable notification of the admin would happen in the tool, right? Well, you know what

Re: [HACKERS] ISN extension bug? (with patch)

2014-01-13 Thread Heikki Linnakangas
On 01/03/2014 07:53 PM, Fabien COELHO wrote: If so, there is only the one-liner patch to consider. This patch doesn't apply anymore. Please submit an updated patch for the commit fest. In src/include/utils/elog.h there is an include for utils/errcodes.h which is generated somehow when

Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-13 Thread Craig Ringer
On 01/09/2014 11:19 PM, Tom Lane wrote: Dean Rasheed dean.a.rash...@gmail.com writes: My first thought was that it should just preprocess any security barrier quals in subquery_planner() in the same way as other quals are preprocessed. But thinking about it further, those quals are destined

Re: [HACKERS] nested hstore patch

2014-01-13 Thread Oleg Bartunov
Thank you, Erik ! Oleg On Mon, Jan 13, 2014 at 12:25 PM, Erik Rijkers e...@xs4all.nl wrote: On Mon, January 13, 2014 00:24, Erik Rijkers wrote: On Sat, January 11, 2014 22:47, Andrew Dunstan wrote: On 01/11/2014 03:03 PM, Erik Rijkers wrote: On Sat, January 11, 2014 20:30, Peter Eisentraut

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Florian Pflug
On Jan12, 2014, at 04:18 , Josh Berkus j...@agliodbs.com wrote: Thing is, when we talk about auto-degrade, we need to determine things like Is the replica down or is this just a network blip? and take action according to the user's desired configuration. This is not something, realistically,

[HACKERS] Where do we stand on 9.3 bugs?

2014-01-13 Thread Tom Lane
What remaining issues are there blocking a 9.3.3 release? I know that there were unresolved multixact issues when we put out 9.3.2 --- are those all dealt with now? What else do people see as release-blockers? regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] generic pseudotype IO functions?

2014-01-13 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On Mon, 2014-01-06 at 17:36 +0100, Andres Freund wrote: FWIW, I am perfectly fine with duplicating the functions for now - I just thought that that might not be the best way but I didn't (and still don't) have a strong opinion. Could we just put 0 in

Re: [HACKERS] Where do we stand on 9.3 bugs?

2014-01-13 Thread Alvaro Herrera
Tom Lane wrote: What remaining issues are there blocking a 9.3.3 release? I know that there were unresolved multixact issues when we put out 9.3.2 --- are those all dealt with now? What else do people see as release-blockers? The only thing I'm aware still outstanding in multixact land is

Re: [HACKERS] Where do we stand on 9.3 bugs?

2014-01-13 Thread Andres Freund
On 2014-01-13 12:26:45 -0300, Alvaro Herrera wrote: Tom Lane wrote: What remaining issues are there blocking a 9.3.3 release? I know that there were unresolved multixact issues when we put out 9.3.2 --- are those all dealt with now? What else do people see as release-blockers? The only

Re: [HACKERS] Where do we stand on 9.3 bugs?

2014-01-13 Thread Alvaro Herrera
Andres Freund wrote: On 2014-01-13 12:26:45 -0300, Alvaro Herrera wrote: Tom Lane wrote: What remaining issues are there blocking a 9.3.3 release? I know that there were unresolved multixact issues when we put out 9.3.2 --- are those all dealt with now? What else do people see as

Re: [HACKERS] nested hstore patch

2014-01-13 Thread Alvaro Herrera
Andrew Dunstan wrote: On 01/13/2014 03:25 AM, Erik Rijkers wrote: There are errors in the example expressions in Table F-6. hstore Operators. Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these. I also attach an test perl program that

Re: [HACKERS] nested hstore patch

2014-01-13 Thread Oleg Bartunov
Andrew, did you run perl script ? Actually, I found, that operator table needs to be fixed. Oleg On Mon, Jan 13, 2014 at 7:36 PM, Andrew Dunstan and...@dunslane.net wrote: On 01/13/2014 03:25 AM, Erik Rijkers wrote: There are errors in the example expressions in Table F-6. hstore

Re: [HACKERS] Add CREATE support to event triggers

2014-01-13 Thread Alvaro Herrera
Alvaro Herrera escribió: In an event trigger, the function pg_event_trigger_get_creation_commands() returns the following JSON blob: After playing with this for a while, I realized something that must have seemed quite obvious to those paying attention: what this function is, is just a

Re: [HACKERS] nested hstore patch

2014-01-13 Thread Andrew Dunstan
On 01/13/2014 11:03 AM, Alvaro Herrera wrote: Andrew Dunstan wrote: On 01/13/2014 03:25 AM, Erik Rijkers wrote: There are errors in the example expressions in Table F-6. hstore Operators. Attached is a cumulative doc-patch (which includes the changes I sent earlier) which fixes these. I

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Merlin Moncure
On Sun, Jan 12, 2014 at 4:38 AM, Craig Ringer cr...@2ndquadrant.com wrote: Implicit casts to text, anybody? This backward compatibility break orphaned the company I work for on 8.1 until last year and very nearly caused postgres to be summarily extirpated (only rescued at the last minute by my

[HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Mel Gorman
Hi, I'm the chair for Linux Storage, Filesystem and Memory Management Summit 2014 (LSF/MM). A CFP was sent out last month (https://lwn.net/Articles/575681/) that you may have seen already. In recent years we have had at least one topic that was shared between all three tracks that was lead by a

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-13 Thread Alexander Korotkov
On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with the current patch, and once again I got PANIC: could not split GIN page,

[HACKERS] KNN-GiST with recheck

2014-01-13 Thread Alexander Korotkov
Hackers! This patch was split from thread: http://www.postgresql.org/message-id/CAPpHfdscOX5an71nHd8WSUH6GNOCf=V7wgDaTXdDd9=gon-...@mail.gmail.com I've split it to separate thead, because it's related to partial sort only conceptually not technically. Also I renamed it to knn-gist-recheck from

Re: [HACKERS] [PATCH] Add transforms feature

2014-01-13 Thread Robert Haas
On Fri, Jan 10, 2014 at 10:40 PM, Peter Eisentraut pete...@gmx.net wrote: On Wed, 2013-12-11 at 11:07 -0500, Tom Lane wrote: We should have learned by now that those are usually a bad idea. In this case, we've got changes in the behavior of function calling, which seems like not only a

Re: [HACKERS] nested hstore patch

2014-01-13 Thread Andrew Dunstan
On 01/13/2014 11:16 AM, Oleg Bartunov wrote: Andrew, did you run perl script ? Actually, I found, that operator table needs to be fixed. No. My build machine doesn't actually have DBD::Pg installed. Can you send me a patch if you don't want to push it yourself, or maybe Erik can send a

Re: [HACKERS] PoC: Partial sort

2014-01-13 Thread Alexander Korotkov
On Tue, Dec 31, 2013 at 5:41 AM, Andreas Karlsson andr...@proxel.se wrote: On 12/29/2013 08:24 AM, David Rowley wrote: If it was possible to devise some way to reuse any previous tuplesortstate perhaps just inventing a reset method which clears out tuples, then we could see performance

Re: [HACKERS] Patch: show xid and xmin in pg_stat_activity and pg_stat_replication

2014-01-13 Thread Heikki Linnakangas
On 12/17/2013 04:58 PM, Christian Kruse wrote: attached you will find a patch for showing the current transaction id (xid) and the xmin of a backend in pg_stat_activty and the xmin in pg_stat_replication. Docs. When an admin is looking for a long-running transaction that's blocking vacuum,

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Hannu Krosing
On 01/13/2014 04:12 PM, Florian Pflug wrote: On Jan12, 2014, at 04:18 , Josh Berkus j...@agliodbs.com wrote: Thing is, when we talk about auto-degrade, we need to determine things like Is the replica down or is this just a network blip? and take action according to the user's desired

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Joshua D. Drake
On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the *easiest* place. Anything we do *inside* Postgres is going to have a really, really hard time

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Josh Berkus
Mel, I'm the chair for Linux Storage, Filesystem and Memory Management Summit 2014 (LSF/MM). A CFP was sent out last month (https://lwn.net/Articles/575681/) that you may have seen already. In recent years we have had at least one topic that was shared between all three tracks that was

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Kevin Grittner
Josh Berkus j...@agliodbs.com wrote: Does the PostgreSQL community have a problem with recent kernels, particularly with respect to the storage, filesystem or memory management layers? How about don't add major IO behavior changes with no backwards-compatibility switches?  ;-) I notice,

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 12:23 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Exclusion constraints can be used to implement uniqueness checks with SP-GiST or GiST indexes. For example, if you want to enforce that there are no two tuples with the same x and y coordinates, ie. use a point

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Josh Berkus
On 01/13/2014 10:51 AM, Kevin Grittner wrote: How about don't add major IO behavior changes with no backwards-compatibility switches? ;-) I notice, Josh, that you didn't mention the problems many people have run into with Transparent Huge Page defrag and with NUMA access. Is that because

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-13 Thread Robert Haas
On Thu, Jan 9, 2014 at 10:57 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I wrote: Robert Haas wrote: Hmm, fair point. But I'm still not convinced that we really need to add extra accounting for this. What's wrong with just reporting the number of exact and lossy pages? No. I

Re: [HACKERS] ALTER SYSTEM SET command to change postgresql.conf parameters

2014-01-13 Thread Robert Haas
On Thu, Jan 9, 2014 at 11:06 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Thu, Jan 9, 2014 at 12:26 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas robertmh...@gmail.com

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Robert Haas
On Thu, Jan 9, 2014 at 11:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Greg Stark st...@mit.edu writes: On Thu, Jan 9, 2014 at 9:14 PM, Tom Lane t...@sss.pgh.pa.us wrote: In short then, I think we should just add this to EXPLAIN and be done. -1 for sticking the info into PlannedStmt or anything

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: Currently the patch includes changes to prepare.c which is what seems odd to me. I think it'd be fine to say, hey, I can't give you the planning time in this EXPLAIN ANALYZE because I just used a cached plan and did not re-plan. But saying, hey,

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Robert Haas
On Mon, Jan 13, 2014 at 1:51 PM, Kevin Grittner kgri...@ymail.com wrote: I notice, Josh, that you didn't mention the problems many people have run into with Transparent Huge Page defrag and with NUMA access. Amen to that. Actually, I think NUMA can be (mostly?) fixed by setting

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Robert Haas
On Mon, Jan 13, 2014 at 1:53 PM, Peter Geoghegan p...@heroku.com wrote: On Mon, Jan 13, 2014 at 12:23 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Exclusion constraints can be used to implement uniqueness checks with SP-GiST or GiST indexes. For example, if you want to enforce that

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Claudio Freire
On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote: On a related note, there's also the problem of double-buffering. When we read a page into shared_buffers, we leave a copy behind in the OS buffers, and similarly on write-out. It's very unclear what to do about this,

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Andreas Karlsson
On 01/13/2014 09:06 PM, Stephen Frost wrote: My thought, at least, was to always grab the planning time and then provide it for explain and/or explain analyze, and then for re-plan cases, indicate if a cached plan was returned, if a replan happened, and if a replan happened, what the old plan

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 9, 2014 at 11:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Uh, no, wasn't my suggestion. Doesn't that design imply measuring *every* planning cycle, explain or no? I was thinking more of just putting the timing calls into explain.c.

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Jim Nasby
On 1/13/14, 2:19 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote: On a related note, there's also the problem of double-buffering. When we read a page into shared_buffers, we leave a copy behind in the OS buffers, and similarly on write-out.

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-13 Thread Robert Haas
On Fri, Jan 10, 2014 at 2:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Florian Pflug f...@phlo.org writes: On Jan10, 2014, at 19:08 , Tom Lane t...@sss.pgh.pa.us wrote: Although, having said that ... maybe build your own aggregate would be a reasonable suggestion for people who need this? I

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Claudio Freire
On Mon, Jan 13, 2014 at 5:23 PM, Jim Nasby j...@nasby.net wrote: On 1/13/14, 2:19 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote: On a related note, there's also the problem of double-buffering. When we read a page into shared_buffers, we

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Robert Haas
On Mon, Jan 13, 2014 at 3:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Jan 9, 2014 at 11:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Uh, no, wasn't my suggestion. Doesn't that design imply measuring *every* planning cycle, explain or no? I was

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Jim Nasby
On 1/13/14, 2:27 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:23 PM, Jim Nasby j...@nasby.net wrote: On 1/13/14, 2:19 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote: On a related note, there's also the problem of double-buffering.

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Claudio Freire
On Mon, Jan 13, 2014 at 5:32 PM, Jim Nasby j...@nasby.net wrote: That's my point. In terms of kernel-postgres interaction, it's fairly simple. What's not so simple, is figuring out what policy to use. Remember, you cannot tell the kernel to put some page in its page cache without reading it

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jan 13, 2014 at 3:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Meh. Why? This would only come into play for EXPLAIN EXECUTE stmtname. I don't think users would be surprised to see a report of minimal planning time for that. In fact, it might be

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Andres Freund
On 2014-01-13 15:15:16 -0500, Robert Haas wrote: On Mon, Jan 13, 2014 at 1:51 PM, Kevin Grittner kgri...@ymail.com wrote: I notice, Josh, that you didn't mention the problems many people have run into with Transparent Huge Page defrag and with NUMA access. Amen to that. Actually, I

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2014-01-13 Thread Robert Haas
On Sat, Jan 11, 2014 at 1:08 AM, Amit Kapila amit.kapil...@gmail.com wrote: Yes, currently this applies to update, what I have in mind is that in future if some one wants to use WAL compression for any other operation like 'full_page_writes', then it can be easily extendible. To be honest, I

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-13 Thread Robert Haas
On Mon, Jan 13, 2014 at 3:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jan 13, 2014 at 3:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Meh. Why? This would only come into play for EXPLAIN EXECUTE stmtname. I don't think users would be surprised to

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 12:17 PM, Robert Haas robertmh...@gmail.com wrote: For what it's worth, I agree with Heikki. There's probably nothing sensible an upsert can do if it conflicts with more than one tuple, but if it conflicts with just exactly one, it oughta be OK. If there is exactly

Re: [HACKERS] PoC: Partial sort

2014-01-13 Thread Marti Raudsepp
Hi Alexander, First, thanks a lot for working on this feature. This PostgreSQL shortcoming crops up in all the time in web applications that implement paging by multiple sorted columns. I've been trying it out in a few situations. I implemented a new enable_partialsort GUC to make it easier to

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Jim Nasby
On 1/13/14, 2:37 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:32 PM, Jim Nasby j...@nasby.net wrote: That's my point. In terms of kernel-postgres interaction, it's fairly simple. What's not so simple, is figuring out what policy to use. Remember, you cannot tell the kernel to put some

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Heikki Linnakangas
On 01/13/2014 10:53 PM, Peter Geoghegan wrote: On Mon, Jan 13, 2014 at 12:17 PM, Robert Haas robertmh...@gmail.com wrote: For what it's worth, I agree with Heikki. There's probably nothing sensible an upsert can do if it conflicts with more than one tuple, but if it conflicts with just exactly

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Andres Freund
On 2014-01-13 15:53:36 -0500, Trond Myklebust wrote: I've wondered before if there wouldn't be a chance for postgres to say my dear OS, that the file range 0-8192 of file x contains y, no need to reread and do that when we evict a page from s_b but I never dared to actually propose that to

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Jeff Janes
On Mon, Jan 13, 2014 at 12:32 PM, Jim Nasby j...@nasby.net wrote: On 1/13/14, 2:27 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:23 PM, Jim Nasby j...@nasby.net wrote: On 1/13/14, 2:19 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Robert Haas
On Mon, Jan 13, 2014 at 3:53 PM, Trond Myklebust tron...@gmail.com wrote: O_DIRECT was specifically designed to solve the problem of double buffering between applications and the kernel. Why are you not able to use that in these situations? O_DIRECT was apparently designed by a deranged

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread James Bottomley
On Mon, 2014-01-13 at 14:32 -0600, Jim Nasby wrote: On 1/13/14, 2:27 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:23 PM, Jim Nasby j...@nasby.net wrote: On 1/13/14, 2:19 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote: On a

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Trond Myklebust
On Jan 13, 2014, at 15:40, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-13 15:15:16 -0500, Robert Haas wrote: On Mon, Jan 13, 2014 at 1:51 PM, Kevin Grittner kgri...@ymail.com wrote: I notice, Josh, that you didn't mention the problems many people have run into with Transparent

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Kevin Grittner
Josh Berkus j...@agliodbs.com wrote: Wanna go to Collab? I don't think that works out for me, but thanks for suggesting it. I'd be happy to brainstorm with anyone who does go about issues to discuss; although the ones I keep running into have already been mentioned. Regarding the problems

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Andres Freund
On 2014-01-13 12:34:35 -0800, James Bottomley wrote: On Mon, 2014-01-13 at 14:32 -0600, Jim Nasby wrote: Well, if we were to collaborate with the kernel community on this then presumably we can do better than that for eviction... even to the extent of here's some data from this range in

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Jim Nasby
On 1/13/14, 12:21 PM, Joshua D. Drake wrote: On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the *easiest* place. Anything we do *inside* Postgres

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Andres Freund
On 2014-01-13 15:14:21 -0600, Jim Nasby wrote: On 1/13/14, 12:21 PM, Joshua D. Drake wrote: On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Jim Nasby
On 1/13/14, 10:40 AM, Merlin Moncure wrote: On Sun, Jan 12, 2014 at 4:38 AM, Craig Ringercr...@2ndquadrant.com wrote: Implicit casts to text, anybody? This backward compatibility break orphaned the company I work for on 8.1 until last year and very nearly caused postgres to be summarily

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Greg Stark
On Mon, Jan 13, 2014 at 9:12 PM, Andres Freund and...@2ndquadrant.com wrote: For one, postgres doesn't use mmap for files (and can't without major new interfaces). Frequently mmap()/madvise()/munmap()ing 8kb chunks has horrible consequences for performance/scalability - very quickly you

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Joshua D. Drake
On 01/13/2014 01:14 PM, Jim Nasby wrote: On 1/13/14, 12:21 PM, Joshua D. Drake wrote: On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread David Fetter
On Mon, Jan 13, 2014 at 10:40:57AM -0600, Merlin Moncure wrote: On Sun, Jan 12, 2014 at 4:38 AM, Craig Ringer cr...@2ndquadrant.com wrote: Implicit casts to text, anybody? This backward compatibility break orphaned the company I work for on 8.1 until last year and very nearly caused

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Jim Nasby
On 1/13/14, 1:44 AM, Pavel Stehule wrote: 2014/1/12 Florian Pflug f...@phlo.org mailto:f...@phlo.org On Jan12, 2014, at 22:37 , Pavel Stehule pavel.steh...@gmail.com mailto:pavel.steh...@gmail.com wrote: There is GUC for variable_conflict already too. In this case I would to

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Robert Haas
On Mon, Jan 13, 2014 at 4:45 PM, David Fetter da...@fetter.org wrote: On Mon, Jan 13, 2014 at 10:40:57AM -0600, Merlin Moncure wrote: On Sun, Jan 12, 2014 at 4:38 AM, Craig Ringer cr...@2ndquadrant.com wrote: Implicit casts to text, anybody? This backward compatibility break orphaned the

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Josh Berkus
Everyone, I am looking for one or more hackers to go to Collab with me to discuss this. If you think that might be you, please let me know and I'll look for funding for your travel. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Syntax of INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Jim Nasby
On 1/12/14, 9:35 PM, Andreas Karlsson wrote: On 01/12/2014 11:20 PM, Peter Geoghegan wrote: On Sun, Jan 12, 2014 at 8:12 AM, Andreas Karlsson andr...@proxel.se wrote: On 01/11/2014 11:42 PM, Peter Geoghegan wrote: I recently suggested that rather than RETURNING REJECTS, we could have a

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Merlin Moncure
On Mon, Jan 13, 2014 at 3:45 PM, David Fetter da...@fetter.org wrote: On Mon, Jan 13, 2014 at 10:40:57AM -0600, Merlin Moncure wrote: This project has no deprecation policy, I believe it actually does, although it's not a formal, written policy. Would you like to help draft one up? Lack of

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Mel Gorman
On Mon, Jan 13, 2014 at 03:15:16PM -0500, Robert Haas wrote: On Mon, Jan 13, 2014 at 1:51 PM, Kevin Grittner kgri...@ymail.com wrote: I notice, Josh, that you didn't mention the problems many people have run into with Transparent Huge Page defrag and with NUMA access. Ok, there are at

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Mel Gorman
On Mon, Jan 13, 2014 at 06:27:03PM -0200, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:23 PM, Jim Nasby j...@nasby.net wrote: On 1/13/14, 2:19 PM, Claudio Freire wrote: On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote: On a related note, there's also the

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 12:58 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Well, even if you don't agree that locking all the conflicting rows for update is sensible, it's still perfectly sensible to return the rejected rows to the user. For example, you're inserting N rows, and if

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Andres Freund
On 2014-01-13 14:19:56 -0800, James Bottomley wrote: Frequently mmap()/madvise()/munmap()ing 8kb chunks has horrible consequences for performance/scalability - very quickly you contend on locks in the kernel. Is this because of problems in the mmap_sem? It's been a while since I looked

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Mel Gorman
On Mon, Jan 13, 2014 at 11:38:44PM +0100, Jan Kara wrote: On Mon 13-01-14 22:26:45, Mel Gorman wrote: The flipside is also meant to hold true. If you know data will be needed in the near future then posix_fadvise(POSIX_FADV_WILLNEED). Glancing at the implementation it does a forced

Re: [HACKERS] Syntax of INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 2:20 PM, Jim Nasby j...@nasby.net wrote: Well, a common case for INSERT RETURNING is to get your set of surrogate keys back; so I think users would want the ability to RETURN what finally made it into the table. Your update can also have a RETURNING clause. I'm not

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Florian Pflug
On Jan13, 2014, at 22:30 , Joshua D. Drake j...@commandprompt.com wrote: On 01/13/2014 01:14 PM, Jim Nasby wrote: On 1/13/14, 12:21 PM, Joshua D. Drake wrote: On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Josh Berkus
On 01/13/2014 02:26 PM, Mel Gorman wrote: Really? zone_reclaim_mode is often a complete disaster unless the workload is partitioned to fit within NUMA nodes. On older kernels enabling it would sometimes cause massive stalls. I'm actually very surprised to hear it fixes anything and would be

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Craig Ringer
On 01/14/2014 12:40 AM, Merlin Moncure wrote: On Sun, Jan 12, 2014 at 4:38 AM, Craig Ringer cr...@2ndquadrant.com wrote: Implicit casts to text, anybody? This backward compatibility break orphaned the company I work for on 8.1 until last year and very nearly caused postgres to be summarily

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-13 Thread Tomas Vondra
On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Florian Pflug
On Jan13, 2014, at 22:49 , Jim Nasby j...@nasby.net wrote: ISTM that in this case, it should be safe to make the new default behavior STRICT; if you forget to set the GUC to disable than you'll get an error that points directly at the problem, at which point you'll go Oh, yeah... I forgot

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Marko Tiikkaja
On 1/14/14, 12:41 AM, Florian Pflug wrote: In fact, after reading the documentation on SELECT ... INTO, I'm convinced the the whole consistent_into thing is a bad idea. The documentation states clearly that For INSERT/UPDATE/DELETE with RETURNING, PL/pgSQL reports an error for more than

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Josh Berkus
On 01/13/2014 03:41 PM, Florian Pflug wrote: It therefor isn't an oversight that SELECT ... INTO allows multiple result rows but INSERT/UPDATE/DELETE forbids them, it's been done that way on purpose and for a reason. We shouldn't be second-guessing ourselves by changing that later - not,

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Hannu Krosing
On 01/13/2014 09:53 PM, Trond Myklebust wrote: On Jan 13, 2014, at 15:40, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-13 15:15:16 -0500, Robert Haas wrote: On Mon, Jan 13, 2014 at 1:51 PM, Kevin Grittner kgri...@ymail.com wrote: I notice, Josh, that you didn't mention the problems

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Hannu Krosing
On 01/14/2014 12:33 AM, Craig Ringer wrote: On 01/14/2014 12:40 AM, Merlin Moncure wrote: On Sun, Jan 12, 2014 at 4:38 AM, Craig Ringer cr...@2ndquadrant.com wrote: Implicit casts to text, anybody? This backward compatibility break orphaned the company I work for on 8.1 until last year and

Re: [HACKERS] Syntax of INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Jim Nasby
On 1/13/14, 5:05 PM, Peter Geoghegan wrote: On Mon, Jan 13, 2014 at 2:20 PM, Jim Nasby j...@nasby.net wrote: Well, a common case for INSERT RETURNING is to get your set of surrogate keys back; so I think users would want the ability to RETURN what finally made it into the table. Your update

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Florian Pflug
On Jan14, 2014, at 00:52 , Marko Tiikkaja ma...@joh.to wrote: When I've worked with PL/PgSQL, this has been a source of a few bugs that would have been noticed during testing if the behaviour of INTO wasn't as dangerous as it is right now. The question is, how many bugs stemmed from wrong SQL

Re: [HACKERS] nested hstore patch

2014-01-13 Thread Erik Rijkers
On Mon, January 13, 2014 18:30, Andrew Dunstan wrote: On 01/13/2014 11:16 AM, Oleg Bartunov wrote: Andrew, did you run perl script ? Actually, I found, that operator table needs to be fixed. No. My build machine doesn't actually have DBD::Pg installed. Can you send me a patch if you

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Jim Nasby
On 1/13/14, 5:57 PM, Josh Berkus wrote: On 01/13/2014 03:41 PM, Florian Pflug wrote: It therefor isn't an oversight that SELECT ... INTO allows multiple result rows but INSERT/UPDATE/DELETE forbids them, it's been done that way on purpose and for a reason. We shouldn't be second-guessing

Re: [HACKERS] plpgsql.consistent_into

2014-01-13 Thread Jim Nasby
On 1/13/14, 6:16 PM, Florian Pflug wrote: On Jan14, 2014, at 00:52 , Marko Tiikkaja ma...@joh.to wrote: When I've worked with PL/PgSQL, this has been a source of a few bugs that would have been noticed during testing if the behaviour of INTO wasn't as dangerous as it is right now. The

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Claudio Freire
On Mon, Jan 13, 2014 at 7:36 PM, Mel Gorman mgor...@suse.de wrote: That could be something we look at. There are cases buried deep in the VM where pages get shuffled to the end of the LRU and get tagged for reclaim as soon as possible. Maybe you need access to something like that via

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Jim Nasby
On 1/13/14, 5:33 PM, Craig Ringer wrote: So I guess the question is: Is it worth all that hassle to remove a misfeature you have to go out of your way to use? Is support for non-1 lower bounds stopping us from doing something useful and important? Or is it just an irritation that it exists?

  1   2   >