Re: [HACKERS] New functions in sslinfo module

2014-04-21 Thread Michael Paquier
On Mon, Apr 21, 2014 at 2:51 PM, Воронин Дмитрий carriingfat...@yandex.ruwrote: I put patch generated on git diffs to this letter. I make an a thread in postgresql commit fest: https://commitfest.postgresql.org/action/patch_view?id=1438 Thanks! -- Michael

Re: [HACKERS] DISCARD ALL (Again)

2014-04-21 Thread Hannu Krosing
On 04/18/2014 01:38 AM, Fabrízio de Royes Mello wrote: On Thu, Apr 17, 2014 at 6:51 PM, Alvaro Herrera alvhe...@2ndquadrant.com mailto:alvhe...@2ndquadrant.com wrote: It does sounds a legitimate feature request to me. I don't remember if we honored the request to add resetting of

Re: [HACKERS] DISCARD ALL (Again)

2014-04-21 Thread Hannu Krosing
On 04/18/2014 06:44 PM, Joshua D. Drake wrote: On 04/18/2014 08:01 AM, Peter Eisentraut wrote: On 4/17/14, 4:44 PM, Joshua D. Drake wrote: That we should also release the GD? In some cases, SD or GD are used to cache things. Having the connection pooler blow that away would defeat the

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Francois Tigeot
On Sun, Apr 20, 2014 at 04:07:25PM +0200, Palle Girgensohn wrote: If mmap needs to perform well in the kernel, I'd like to know of someone with FreeBSD kernel knowledge who is interested in working with mmap perfocmance. If mmap is indeed the cuplrit, I've just tested 9.2.8 vs 9.3.4,

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Palle Girgensohn
21 apr 2014 kl. 11:26 skrev Francois Tigeot ftig...@wolfpond.org: On Sun, Apr 20, 2014 at 04:07:25PM +0200, Palle Girgensohn wrote: If mmap needs to perform well in the kernel, I'd like to know of someone with FreeBSD kernel knowledge who is interested in working with mmap

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
Hi, On 2014-04-20 11:24:38 +0200, Palle Girgensohn wrote: I see performance degradation with PostgreSQL 9.3 vs 9.2 on FreeBSD, and I'm wondering who to poke to mitigate the problem. In reference to this thread [1], who where the FreeBSD people that Francois mentioned? If mmap needs to

Re: [HACKERS] New functions for sslinfo extension

2014-04-21 Thread Dmitry Voronin
Hello, I make an a patch, which adds 4 functions to sslinfo extension module.  You can view some information of certificate's extensions with those functions. The descriptions of functions I posted in my first letter and in source code. What do you think about it? -- Best regards, Dmitry Voronin 

Re: [HACKERS] Re: [DOCS] Docs incorrectly claiming equivalence between show and pg_settings

2014-04-21 Thread Stefan Seifert
On Saturday 19 April 2014 13:38:16 Tom Lane wrote: 3 document this restriction As for (3), I might be wrong, but I don't think the documentation mentions the possibility of abusing SET this way at all. Restrictions in undocumented quasi-features are likewise undocumented.

[HACKERS] New functions for sslinfo extension

2014-04-21 Thread Воронин Дмитрий
Hello, I make an a patch, which adds 4 functions to sslinfo extension module:1) ssl_get_count_of_extensions() --- get count of X509v3 extensions from client certificate;2) ssl_get_extension_names() --- get short names of X509v3 extensions from client certificate;3) ssl_get_extension_value(text)

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 4:10 AM, Andres Freund wrote: Hi, On 2014-04-20 11:24:38 +0200, Palle Girgensohn wrote: I see performance degradation with PostgreSQL 9.3 vs 9.2 on FreeBSD, and I'm wondering who to poke to mitigate the problem. In reference to this thread [1], who where the FreeBSD people that

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: If there are indeed such large regressions on FreeBSD we need to treat them as postgres regressions. It's nicer not to add config options for things that don't need it, but apparently that's not the case here. Imo this means we need to add GUC to

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
On 2014-04-21 10:45:24 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: If there are indeed such large regressions on FreeBSD we need to treat them as postgres regressions. It's nicer not to add config options for things that don't need it, but apparently that's not the

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Tom Lane
Noah Misch n...@leadboat.com writes: I wonder how it would work out to instead delay this new detoast effort until toast_insert_or_update(). That would require toast_insert_or_update() to know about every container datatype. I doubt it could lead to an extensible or maintainable solution. I'm

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Andres Freund
Hi, On 2014-04-20 15:38:23 -0400, Tom Lane wrote: Some poking around with oprofile says that much of the added time is coming from added syscache and typcache lookups, as I suspected. I did some further work on the patch to make it possible to cache the element tuple descriptor across calls

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I unfortunately haven't followed this in detail, but shouldn't it be relatively easily to make this even cheaper by checking for HEAP_HASEXTERNAL? If it's not set we don't need to iterate over the composite's columns, right? That's the point I

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Magnus Hagander
On Mon, Apr 21, 2014 at 4:51 PM, Andres Freund and...@2ndquadrant.comwrote: On 2014-04-21 10:45:24 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: If there are indeed such large regressions on FreeBSD we need to treat them as postgres regressions. It's nicer not to

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Andres Freund
On 2014-04-21 11:30:57 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I unfortunately haven't followed this in detail, but shouldn't it be relatively easily to make this even cheaper by checking for HEAP_HASEXTERNAL? If it's not set we don't need to iterate over the

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
Hi, On 2014-04-21 17:39:39 +0200, Magnus Hagander wrote: But do we really want a *guc* for it though? Isn't it enough (and in fact better) with a configure switch to pick the implementation when multiple are available, that could then be set by default for example by the freebsd ports build?

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 11:39 AM, Magnus Hagander wrote: On Mon, Apr 21, 2014 at 4:51 PM, Andres Freund and...@2ndquadrant.com mailto:and...@2ndquadrant.com wrote: On 2014-04-21 10:45:24 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com mailto:and...@2ndquadrant.com writes:

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
Hi, On 2014-04-21 11:45:49 -0400, Andrew Dunstan wrote: That seems to make more sense. I can't imagine why this would be a runtime parameter as opposed to build time. Because that implies that packagers and porters need to make that decision. If it's a GUC people can benchmark it and decide.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 8:45 AM, Andrew Dunstan wrote: On 04/21/2014 11:39 AM, Magnus Hagander wrote: On Mon, Apr 21, 2014 at 4:51 PM, Andres Freund and...@2ndquadrant.com mailto:and...@2ndquadrant.com wrote: On 2014-04-21 10:45:24 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 8:58 AM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-21 11:45:49 -0400, Andrew Dunstan wrote: That seems to make more sense. I can't imagine why this would be a runtime parameter as opposed to build time. Because that implies that packagers and porters

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-04-21 11:45:49 -0400, Andrew Dunstan wrote: That seems to make more sense. I can't imagine why this would be a runtime parameter as opposed to build time. Because that implies that packagers and porters need to make that decision. If it's

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
* Alfred Perlstein (alf...@freebsd.org) wrote: Can the package builder not set the default for the runtime tunable? Yeah, I was thinking about that also, but at least in this case it seems pretty clear that the 'right' answer is known at build time. Honestly we're about to select a db platform

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
On 2014-04-21 11:58:10 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-21 11:45:49 -0400, Andrew Dunstan wrote: That seems to make more sense. I can't imagine why this would be a runtime parameter as opposed to build time. Because that implies that

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Merlin Moncure
On Mon, Apr 21, 2014 at 10:40 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-04-21 11:30:57 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I unfortunately haven't followed this in detail, but shouldn't it be relatively easily to make this even cheaper by

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Josh Berkus
All, More on this: 1) I've confirmed at the 2nd site that the issue doesn't happen if pg_stat_statements.so is not loaded. So this seems to be confirmation that either auto_explain, pg_stat_statements, or both need to be loaded (but not necessarily created as extensions) in order to have the

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Stephen Frost
* Josh Berkus (j...@agliodbs.com) wrote: 1) I've confirmed at the 2nd site that the issue doesn't happen if pg_stat_statements.so is not loaded. So this seems to be confirmation that either auto_explain, pg_stat_statements, or both need to be loaded (but not necessarily created as extensions)

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 9:13 AM, Stephen Frost wrote: * Alfred Perlstein (alf...@freebsd.org) wrote: Can the package builder not set the default for the runtime tunable? Yeah, I was thinking about that also, but at least in this case it seems pretty clear that the 'right' answer is known at build time.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 11:59 AM, Alfred Perlstein wrote: On 4/21/14 8:45 AM, Andrew Dunstan wrote: On 04/21/2014 11:39 AM, Magnus Hagander wrote: On Mon, Apr 21, 2014 at 4:51 PM, Andres Freund and...@2ndquadrant.com mailto:and...@2ndquadrant.com wrote: On 2014-04-21 10:45:24 -0400, Tom Lane

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Francois Tigeot
On Mon, Apr 21, 2014 at 05:43:46PM +0200, Andres Freund wrote: On 2014-04-21 17:39:39 +0200, Magnus Hagander wrote: But do we really want a *guc* for it though? Isn't it enough (and in fact better) with a configure switch to pick the implementation when multiple are available, that could

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 9:24 AM, Andrew Dunstan wrote: On 04/21/2014 11:59 AM, Alfred Perlstein wrote: On 4/21/14 8:45 AM, Andrew Dunstan wrote: On 04/21/2014 11:39 AM, Magnus Hagander wrote: On Mon, Apr 21, 2014 at 4:51 PM, Andres Freund and...@2ndquadrant.com mailto:and...@2ndquadrant.com wrote:

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
* Alfred Perlstein (alf...@freebsd.org) wrote: There is definitely hope, however changes to the FreeBSD vm are taken as seriously as changes to core changes to Postresql's store. In addition changes to vm is somewhat in the realm of complexity of Postgresql store as well so it may not be

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 12:25 PM, Alfred Perlstein wrote: 1. OS developers are not the target audience for GUCs. If the OS developers want to test and can't be botherrd with building with a couple of different parameters then I'm not very impressed. 2. We should be trying to get rid of GUCs

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Josh Berkus
Can you get the infomask bits..? What's does pg_controldata report wrt the MultiXid's? Can't get the infomask bits. pg_controldata attached, with some redactions. Unfortunately, it appears that they've continued to do tests on this system, so the XID counter has advanced somewhat.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 9:34 AM, Stephen Frost wrote: * Alfred Perlstein (alf...@freebsd.org) wrote: There is definitely hope, however changes to the FreeBSD vm are taken as seriously as changes to core changes to Postresql's store. In addition changes to vm is somewhat in the realm of complexity of

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Alvaro Herrera
Merlin Moncure wrote: On Mon, Apr 21, 2014 at 10:40 AM, Andres Freund and...@2ndquadrant.com wrote: And too bad that infomask bits are so scarce :(. We really need to reclaim HEAP_MOVED_OFF and HEAP_MOVED_IN. The only consequence of that is losing support for in-place update for

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 9:38 AM, Andrew Dunstan wrote: On 04/21/2014 12:25 PM, Alfred Perlstein wrote: 1. OS developers are not the target audience for GUCs. If the OS developers want to test and can't be botherrd with building with a couple of different parameters then I'm not very impressed. 2.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 12:44 PM, Alfred Perlstein wrote: On 4/21/14 9:38 AM, Andrew Dunstan wrote: On 04/21/2014 12:25 PM, Alfred Perlstein wrote: 1. OS developers are not the target audience for GUCs. If the OS developers want to test and can't be botherrd with building with a couple of

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
On 2014-04-21 09:42:06 -0700, Alfred Perlstein wrote: Sure, to be fair, we are under the gun here for a product, it may just mean that the end result of that conversation is mysql. Personally arguments in that vain are removing just about any incentive I have to work on the problem. Greetings,

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alvaro Herrera
Alfred Perlstein wrote: I am unsure of the true overhead of making this a runtime tunable so pardon if I'm asking for a lot. From the perspective of both an OS developer and postgresql user (I am both) it really makes more sense to have it a runtime tunable for the following reasons: From

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: 1) I've confirmed at the 2nd site that the issue doesn't happen if pg_stat_statements.so is not loaded. So this seems to be confirmation that either auto_explain, pg_stat_statements, or both need to be loaded (but not necessarily created as extensions) in

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14 4:10 AM, Andres Freund wrote: Hi, On 2014-04-20 11:24:38 +0200, Palle Girgensohn wrote: I see performance degradation with PostgreSQL 9.3 vs 9.2 on FreeBSD, and I'm wondering who to poke to mitigate the problem. In reference to this thread [1], who where the FreeBSD people that

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14, 9:51 AM, Andrew Dunstan wrote: On 04/21/2014 12:44 PM, Alfred Perlstein wrote: On 4/21/14 9:38 AM, Andrew Dunstan wrote: On 04/21/2014 12:25 PM, Alfred Perlstein wrote: 1. OS developers are not the target audience for GUCs. If the OS developers want to test and can't be

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alvaro Herrera
Alfred Perlstein wrote: How high on the hierarchy of PostgreSQL's needs is making a single option a tunable versus compile time thing? I mean seriously you mean to stick on this one point when one of your users are asking you about this? That is pretty concerning to me. I think the

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14, 9:51 AM, Andres Freund wrote: On 2014-04-21 09:42:06 -0700, Alfred Perlstein wrote: Sure, to be fair, we are under the gun here for a product, it may just mean that the end result of that conversation is mysql. Personally arguments in that vain are removing just about any

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
* Alfred Perlstein (alf...@freebsd.org) wrote: How high on the hierarchy of PostgreSQL's needs is making a single option a tunable versus compile time thing? I mean seriously you mean to stick on this one point when one of your users are asking you about this? That is pretty concerning to

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14, 9:52 AM, Alvaro Herrera wrote: Alfred Perlstein wrote: I am unsure of the true overhead of making this a runtime tunable so pardon if I'm asking for a lot. From the perspective of both an OS developer and postgresql user (I am both) it really makes more sense to have it a runtime

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
Alfred, * Alfred Perlstein (alf...@freebsd.org) wrote: On 4/21/14, 9:51 AM, Andres Freund wrote: On 2014-04-21 09:42:06 -0700, Alfred Perlstein wrote: Sure, to be fair, we are under the gun here for a product, it may just mean that the end result of that conversation is mysql. Personally

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Andres Freund
Hi, I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much faster and find a possible root cause. Since the symptom of the problem seem to be multixacts with more than one updating xid, I added a

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much faster and find a possible root cause. Hmm ... is this the same thing Josh is reporting? If so,

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14, 11:14 AM, Stephen Frost wrote: Alfred, * Alfred Perlstein (alf...@freebsd.org) wrote: On 4/21/14, 9:51 AM, Andres Freund wrote: On 2014-04-21 09:42:06 -0700, Alfred Perlstein wrote: Sure, to be fair, we are under the gun here for a product, it may just mean that the end result

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Josh Berkus
On 04/21/2014 12:26 PM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much faster and find a possible root cause. Hmm ... is this

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 03:26 PM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much faster and find a possible root cause. Hmm ... is this the

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Andres Freund
On 2014-04-21 15:26:03 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much faster and find a possible root cause. Hmm ...

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
Alfred, * Alfred Perlstein (alf...@freebsd.org) wrote: Stephen, please calm down on the hyperbole, seriously, picking another db is not an attack. Perhaps it wasn't intended by you, but to those of us reading it, your comments came across clearly as if you don't fix this, then we won't use PG.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
On 2014-04-21 15:47:31 -0400, Stephen Frost wrote: That's certainly unfortunate. For my 2c, I'd recommend that you write a minimal implementation that allows you to test just the sysv-vs-mmap case (which could certainly take an option, to avoid having to recompile during testing), or even ask

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2014-04-21 15:47:31 -0400, Stephen Frost wrote: That's certainly unfortunate. For my 2c, I'd recommend that you write a minimal implementation that allows you to test just the sysv-vs-mmap case (which could certainly take an option, to

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Andres Freund
On 2014-04-21 12:31:09 -0700, Josh Berkus wrote: On 04/21/2014 12:26 PM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14, 12:47 PM, Stephen Frost wrote: Asking for help to address the FreeBSD performance would have been much better received. Thanks, Stephen That is exactly what I did, I asked for a version of postgresql that was easy to switch at runtime between two behaviors. That would make

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Bruce Momjian
On Mon, Apr 21, 2014 at 01:52:48PM -0700, Alfred Perlstein wrote: On 4/21/14, 12:47 PM, Stephen Frost wrote: Asking for help to address the FreeBSD performance would have been much better received. Thanks, Stephen That is exactly what I did, I asked for a version of postgresql that was

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Joshua D. Drake
On 04/21/2014 10:39 AM, Alfred Perlstein wrote: What I am seeing here is unfortunately a very strong departure from FreeBSD support by the community from several of the developers. In fact over drinks at pgcon last year there were a TON of jokes making fun of FreeBSD users and developers

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Josh Berkus
Josh, how long does it take you to reproduce the issue? A couple hours. And can you reproduce it outside of a production environment? Not yet, still working on that. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
Alfred, * Alfred Perlstein (alf...@freebsd.org) wrote: On 4/21/14, 12:47 PM, Stephen Frost wrote: Asking for help to address the FreeBSD performance would have been much better received. Thanks, Stephen That is exactly what I did, I asked for a version of postgresql that was easy to

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Bruce Momjian
On Mon, Apr 21, 2014 at 02:08:51PM -0700, Joshua Drake wrote: If the community had more *BSD presence I think it would be great but it isn't all that viable at this point. I do know however that no-one in this community would turn down a team of FreeBSD advocates helping us make PostgreSQL

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Alfred Perlstein
On 4/21/14, 2:23 PM, Stephen Frost wrote: Alfred, * Alfred Perlstein (alf...@freebsd.org) wrote: On 4/21/14, 12:47 PM, Stephen Frost wrote: Asking for help to address the FreeBSD performance would have been much better received. Thanks, Stephen That is exactly what I did, I asked for a

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andres Freund
On 2014-04-21 17:21:20 -0400, Bruce Momjian wrote: On Mon, Apr 21, 2014 at 02:08:51PM -0700, Joshua Drake wrote: If the community had more *BSD presence I think it would be great but it isn't all that viable at this point. I do know however that no-one in this community would turn down a

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Jim Nasby
On 4/21/14, 4:08 PM, Joshua D. Drake wrote: If the community had more *BSD presence I think it would be great but it isn't all that viable at this point. I do know however that no-one in this community would turn down a team of FreeBSD advocates helping us make PostgreSQL awesome for

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Jim Nasby
On 4/15/14, 1:15 PM, Peter Geoghegan wrote: On Tue, Apr 15, 2014 at 9:30 AM, Merlin Moncuremmonc...@gmail.com wrote: There are many reports of improvement from lowering shared_buffers. The problem is that it tends to show up on complex production workloads and that there is no clear evidence

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Jim Nasby
On 4/18/14, 2:51 PM, Atri Sharma wrote: I feel that if there is no memory pressure, frankly it doesnt matter much about what gets out and what not. The case I am specifically targeting is when the clocksweep gets to move about a lot i.e. high memory pressure workloads. Of course, I may be

[HACKERS] AXLE Plans for 9.5 and 9.6

2014-04-21 Thread Simon Riggs
I've discussed 2ndQuadrant's involvement in the AXLE project a few times publicly, but never on this mailing list. The project relates to innovation and improvement in Business Intelligence for systems based upon PostgreSQL in the range of 10-100TB. Our work will span the 9.5 and 9.6 cycles.

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Jim Nasby
On 4/16/14, 10:28 AM, Robert Haas wrote: Also, I think the scalability problems around buffer eviction are eminently solvable, and in particular I'm hopeful that Amit is going to succeed in solving them. Suppose we have a background process (whether the background writer or some other) that

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread David G Johnston
Jim Nasby-2 wrote I feel that if there is no memory pressure, frankly it doesnt matter much about what gets out and what not. The case I am specifically targeting is when the clocksweep gets to move about a lot i.e. high memory pressure workloads. Of course, I may be totally wrong here.

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Claudio Freire
On Mon, Apr 21, 2014 at 8:07 PM, David G Johnston david.g.johns...@gmail.com wrote: Jim Nasby-2 wrote I feel that if there is no memory pressure, frankly it doesnt matter much about what gets out and what not. The case I am specifically targeting is when the clocksweep gets to move about a lot

Re: [HACKERS] Removing dependency to wsock32.lib when compiling code on WIndows

2014-04-21 Thread Michael Paquier
On Mon, Apr 21, 2014 at 12:54 PM, Michael Paquier michael.paqu...@gmail.com wrote: When doing some work on Windows, I noticed that the mkvc specs in src/tools/msvc use wsock32.lib, which is as far as I understand an old, obsolete version of the Windows socket library. Wouldn't it make sense

Re: [HACKERS] AXLE Plans for 9.5 and 9.6

2014-04-21 Thread Josh Berkus
On 04/21/2014 03:41 PM, Simon Riggs wrote: Storage Efficiency * Compression * Column Orientation You might look at turning this: http://citusdata.github.io/cstore_fdw/ ... into a more integrated part of Postgres. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 02:54 PM, Andres Freund wrote: Hi, I spent the last two hours poking arounds in the environment Andrew provided and I was able to reproduce the issue, find a assert to reproduce it much faster and find a possible root cause. What's the assert that makes it happen faster? That

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread David G Johnston
Stephen Frost wrote * Alfred Perlstein ( alfred@ ) wrote: On 4/21/14, 12:47 PM, Stephen Frost wrote: Asking for help to address the FreeBSD performance would have been much better received. Thanks, Stephen That is exactly what I did, I asked for a version of postgresql that was easy

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Tom Lane
Jim Nasby j...@nasby.net writes: How *certain* are we that a single freelist lock (that actually ONLY protects the freelist) would be that big a deal? We used to have one. It was a big bottleneck --- and this was years ago, when the buffer manager was much less scalable than it is today.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Tatsuo Ishii
I see performance degradation with PostgreSQL 9.3 vs 9.2 on FreeBSD, and I'm wondering who to poke to mitigate the problem. In reference to this thread [1], who where the FreeBSD people that Francois mentioned? If mmap needs to perform well in the kernel, I'd like to know of someone with

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Peter Geoghegan
On Mon, Apr 21, 2014 at 4:10 AM, Andres Freund and...@2ndquadrant.com wrote: If there are indeed such large regressions on FreeBSD we need to treat them as postgres regressions. It's nicer not to add config options for things that don't need it, but apparently that's not the case here. +1, but

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Peter Geoghegan
On Mon, Apr 21, 2014 at 5:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: We used to have one. It was a big bottleneck --- and this was years ago, when the buffer manager was much less scalable than it is today. (IIRC, getting rid of a central lock was one of the main advantages of the current

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Joshua D. Drake
On 04/21/2014 03:08 PM, Jim Nasby wrote: On 4/21/14, 4:08 PM, Joshua D. Drake wrote: If the community had more *BSD presence I think it would be great but it isn't all that viable at this point. I do know however that no-one in this community would turn down a team of FreeBSD advocates

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Stephen Frost
* Tatsuo Ishii (is...@postgresql.org) wrote: I observe performance degradation with PostgreSQL 9.3 vs 9.2 on Linux as well. The hardware is HP DL980G7, 80 cores, 2TB mem, RHEL 6, pgbench is used (read only query), scale factor is 1,000 (DB size 15GB). Can you isolate the sysv-vs-mmap patch

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: On Mon, Apr 21, 2014 at 5:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: We used to have one. It was a big bottleneck --- and this was years ago, when the buffer manager was much less scalable than it is today. (IIRC, getting rid of a central lock was one of

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Tatsuo Ishii
* Tatsuo Ishii (is...@postgresql.org) wrote: I observe performance degradation with PostgreSQL 9.3 vs 9.2 on Linux as well. The hardware is HP DL980G7, 80 cores, 2TB mem, RHEL 6, pgbench is used (read only query), scale factor is 1,000 (DB size 15GB). Can you isolate the sysv-vs-mmap

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Peter Geoghegan
On Mon, Apr 21, 2014 at 5:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: ARC *was* the predecessor algorithm. See commit 5d5087363. I believe that the main impetus for replacing ARC with clock sweep came from patent issues, though. It was a happy coincidence that clock sweep happened to be better

Re: [HACKERS] assertion failure 9.3.4

2014-04-21 Thread Josh Berkus
All, I've taken a stab at creating a reproduceable test case based on the characterisitics of the production issues I'm seeing. But clearly there's an element I'm missing, because I'm not able to produce the bug with a pgbench-based test case. My current test has FKs, updating both FK'd tables,

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 08:49 PM, Stephen Frost wrote: * Tatsuo Ishii (is...@postgresql.org) wrote: I observe performance degradation with PostgreSQL 9.3 vs 9.2 on Linux as well. The hardware is HP DL980G7, 80 cores, 2TB mem, RHEL 6, pgbench is used (read only query), scale factor is 1,000 (DB size

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Tatsuo Ishii
This is exactly why we need a benchfarm. I actually have a client working based on Greg Smith's pgbench tools. What we would need is a way to graph the results - that's something beyond my very rudimentary expertise in web programming. If anyone feels like collaborating I'd be glad to

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: On Mon, Apr 21, 2014 at 5:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: ARC *was* the predecessor algorithm. See commit 5d5087363. I believe that the main impetus for replacing ARC with clock sweep came from patent issues, though. That was one issue, but

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Peter Geoghegan
On Mon, Apr 21, 2014 at 6:08 PM, Tatsuo Ishii is...@postgresql.org wrote: What we would need is a way to graph the results - that's something beyond my very rudimentary expertise in web programming. If anyone feels like collaborating I'd be glad to hear from them (The web site is programmed in

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Andrew Dunstan
On 04/21/2014 09:16 PM, Peter Geoghegan wrote: On Mon, Apr 21, 2014 at 6:08 PM, Tatsuo Ishii is...@postgresql.org wrote: What we would need is a way to graph the results - that's something beyond my very rudimentary expertise in web programming. If anyone feels like collaborating I'd be glad

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Peter Geoghegan
On Mon, Apr 21, 2014 at 6:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Did you read the commit message I pointed to? Yes. (See also 4e8af8d27.) Oh, I wasn't actually aware of the fact that 2Q made it into the tree. I thought that the first commit message you referred to just referenced on-list

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-21 Thread Peter Geoghegan
On Mon, Apr 21, 2014 at 5:59 PM, Peter Geoghegan p...@heroku.com wrote: LRU-K, and 2Q have roughly the same advantages. I'm reasonably confident you can have the best of both worlds, or something closer to it. Having said that, a big part of what I'd like to accomplish here is to address the

Re: [HACKERS] DISCARD ALL (Again)

2014-04-21 Thread Fabrízio de Royes Mello
On Mon, Apr 21, 2014 at 12:00 AM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Fri, Apr 18, 2014 at 2:24 AM, Tom Lane t...@sss.pgh.pa.us wrote: 2. While I'm no Python expert, I believe GD is just a specific instance of a general capability for

Re: [HACKERS] Composite Datums containing toasted fields are a bad idea(?)

2014-04-21 Thread Noah Misch
On Mon, Apr 21, 2014 at 10:57:34AM -0400, Tom Lane wrote: Noah Misch n...@leadboat.com writes: I wonder how it would work out to instead delay this new detoast effort until toast_insert_or_update(). That would require toast_insert_or_update() to know about every container datatype. I

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Joshua D. Drake
On 04/21/2014 06:19 PM, Andrew Dunstan wrote: If we never start we'll never get there. I can think of several organizations that might be approached to donate hardware. Like .Org? We have a hardware farm, a rack full of hardware and spindles. It isn't the most current but it is there.

Re: [HACKERS] Perfomance degradation 9.3 (vs 9.2) for FreeBSD

2014-04-21 Thread Tatsuo Ishii
Can you isolate the sysv-vs-mmap patch and see what happens with just that change..? Unfortunately I don't have an access to the machine at this moment. Where is the patch? I would like to test it on a smaller machine for now. Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: