Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 6:54 AM, Tom Lane t...@sss.pgh.pa.us wrote: Greg Stark st...@mit.edu writes: Alternately, has anyone tested whether Timsort would work well? I think that was proposed a few years ago and did not look so good in simple testing. I tested it in 2012. I got as far as

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Michael Paquier
On Thu, Aug 20, 2015 at 11:16 AM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: On 2015-08-19 PM 09:23, Simon Riggs wrote: We'll need regression tests that cover each restriction and docs that match. This is not something we should leave until last. People read the docs to understand

Re: [HACKERS] proposal: function parse_ident

2015-08-20 Thread Pavel Stehule
2015-08-20 2:22 GMT+02:00 Tom Lane t...@sss.pgh.pa.us: Jim Nasby jim.na...@bluetreble.com writes: Don't say parse names for things other than tables. Only a minority of the types of objects used in the database have names that meet this specification. Really? My impression is that

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Simon Riggs
On 20 August 2015 at 10:10, Amit Langote langote_amit...@lab.ntt.co.jp wrote: On 2015-08-20 AM 05:10, Josh Berkus wrote: On 08/19/2015 04:59 AM, Simon Riggs wrote: I like the idea of a regular partitioning step because it is how you design such tables - lets use monthly partitions.

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-19 AM 02:59, Corey Huinker wrote: Quick thoughts borne of years of slugging it out with partitions on Oracle: - Finally!!! - Your range partitioning will need to express exclusive/inclusive bounds, or go to the Oracle model where every partition is a cascading values

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-20 AM 05:10, Josh Berkus wrote: On 08/19/2015 04:59 AM, Simon Riggs wrote: I like the idea of a regular partitioning step because it is how you design such tables - lets use monthly partitions. This gives sanely terse syntax, rather than specifying pages and pages of exact values

Re: [HACKERS] Extension upgrade and GUCs

2015-08-20 Thread Simon Riggs
On 18 August 2015 at 21:03, Paul Ramsey pram...@cleverelephant.ca wrote: So I need a way to either (a) notice when I already have a (old) copy of the library loaded and avoid trying to setup the GUC in that case or (b) set-up the GUC in a somewhat less brittle way than

Re: [HACKERS] Use pg_rewind when target timeline was switched

2015-08-20 Thread Michael Paquier
On Wed, Jul 22, 2015 at 4:28 PM, Alexander Korotkov a.korot...@postgrespro.ru wrote: On Wed, Jul 22, 2015 at 8:48 AM, Michael Paquier michael.paqu...@gmail.com wrote On Mon, Jul 20, 2015 at 9:18 PM, Alexander Korotkov a.korot...@postgrespro.ru wrote: attached patch allows pg_rewind to

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-20 PM 06:34, Simon Riggs wrote: On 20 August 2015 at 10:10, Amit Langote langote_amit...@lab.ntt.co.jp wrote: On 2015-08-20 AM 05:10, Josh Berkus wrote: PARTITION BY RANGE ON (columns) INCREMENT BY (INTERVAL '1 month' ) START WITH value; Oh, I like that syntax! How would

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-19 PM 09:52, David Fetter wrote: On Wed, Aug 19, 2015 at 04:30:39PM +0900, Amit Langote wrote: One small change to make this part more efficient: 1. Take the access exclusive lock on table_name. 2. Check for a matching constraint on it. 3. If it's there, mark it as a valid

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Simon Riggs
On 20 August 2015 at 03:16, Amit Langote langote_amit...@lab.ntt.co.jp wrote: Sorry, should have added tests and docs already. I will add them in the next version of the patch. Thanks for willing to review. Thanks for picking up this challenge. It's easier if you have someone interested all

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Pavan Deolasee
On Tue, Aug 18, 2015 at 4:00 PM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: Hi, I would like propose $SUBJECT for this development cycle. Attached is a WIP patch that implements most if not all of what's described below. Some yet unaddressed parts are mentioned below, too. I'll add

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-20 PM 06:27, Pavan Deolasee wrote: On Tue, Aug 18, 2015 at 4:00 PM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: PARTITION BY LIST ON (name) PARTITION BY RANGE ON (year, month) PARTITION BY LIST ON ((lower(left(name, 2))) PARTITION BY RANGE ON ((extract(year from d)),

Re: [HACKERS] proposal: function parse_ident

2015-08-20 Thread Pavel Stehule
2015-08-20 21:16 GMT+02:00 Jim Nasby jim.na...@bluetreble.com: On 8/19/15 7:22 PM, Tom Lane wrote: Jim Nasby jim.na...@bluetreble.com writes: Don't say parse names for things other than tables. Only a minority of the types of objects used in the database have names that meet this

Re: [HACKERS] proposal: contrib module - generic command scheduler

2015-08-20 Thread Pavel Stehule
Hi 2015-08-20 16:42 GMT+02:00 Alvaro Herrera alvhe...@2ndquadrant.com: Pavel Stehule wrote: Hi, Job schedulers are important and sometimes very complex part of any software. PostgreSQL miss it. I propose new contrib module, that can be used simply for some tasks, and that can be used

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-20 PM 10:19, David Fetter wrote: On Thu, Aug 20, 2015 at 06:58:24PM +0900, Amit Langote wrote: Ah, I understand the point of parameterization (TRUST). Seems like it would be good to have with appropriate documentation of the same. Perhaps, it might as well a parameter to the step 1

Re: [HACKERS] 9.5 release notes

2015-08-20 Thread Peter Geoghegan
On Sat, Jun 13, 2015 at 3:53 PM, Peter Geoghegan p...@heroku.com wrote: I think we should really address this. Attached patch adds a new release note item for it. It also adds to the documentation that explains why users should prefer varchar(n)/text to character(n); the lack of abbreviated

Re: [HACKERS] jsonb array-style subscripting

2015-08-20 Thread Jim Nasby
On 8/20/15 3:44 PM, Josh Berkus wrote: What could be added as an extension? A method for preventing duplicate object keys. Since I'm in the minority here lets just drop it. :) -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Data in Trouble? Get it in Treble!

Re: [HACKERS] statistics for array types

2015-08-20 Thread Tomas Vondra
Hi, On 08/11/2015 04:38 PM, Jeff Janes wrote: When reviewing some recent patches, I decided the statistics gathered for arrays had some pre-existing shortcomings. The main one is that when the arrays contain rare elements there is no histogram to fall back upon when the MCE array is empty,

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-20 Thread Amit Kapila
On Thu, Aug 20, 2015 at 3:38 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Aug 19, 2015 at 9:09 PM, Andres Freund and...@anarazel.de wrote: How hard did you try checking whether this causes regressions? This increases the number of atomics in the commit path a fair bit. I doubt

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-20 Thread Andres Freund
On 2015-08-20 15:38:36 +0530, Amit Kapila wrote: On Wed, Aug 19, 2015 at 9:09 PM, Andres Freund and...@anarazel.de wrote: I spent some time today reviewing the commited patch. So far my only major complaint is that I think the comments are only insufficiently documenting the approach taken:

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Magnus Hagander
On Thu, Aug 20, 2015 at 12:55 PM, Marko Tiikkaja ma...@joh.to wrote: On 8/20/15 12:53 PM, Magnus Hagander wrote: We could change it to radius_servers and radius_ports, and deprecate but keep accepting the old parameters for a release or two. That's one option.. To make it easy, we make

Re: [HACKERS] Reduce ProcArrayLock contention

2015-08-20 Thread Amit Kapila
On Wed, Aug 19, 2015 at 9:09 PM, Andres Freund and...@anarazel.de wrote: Hi, On Wed, Aug 5, 2015 at 10:59 AM, Amit Kapila amit.kapil...@gmail.com wrote: OK, committed. I spent some time today reviewing the commited patch. So far my only major complaint is that I think the comments are

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Marko Tiikkaja
On 8/20/15 12:53 PM, Magnus Hagander wrote: We could change it to radius_servers and radius_ports, and deprecate but keep accepting the old parameters for a release or two. That's one option.. To make it easy, we make sure that both parameter names accepts the same format parameter, so it's

Re: [HACKERS] Extension upgrade and GUCs

2015-08-20 Thread Paul Ramsey
On August 20, 2015 at 2:17:31 AM, Simon Riggs (si...@2ndquadrant.com(mailto:si...@2ndquadrant.com)) wrote: On 18 August 2015 at 21:03, Paul Ramsey wrote: So I need a way to either (a) notice when I already have a (old) copy of the library loaded and avoid trying to setup the GUC in that

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Magnus Hagander
On Thu, Aug 20, 2015 at 2:36 AM, Marko Tiikkaja ma...@joh.to wrote: On 2015-08-20 02:29, Tom Lane wrote: Marko Tiikkaja ma...@joh.to writes: So I'm developing a patch to fix this issue, but I'm not exactly sure what the configuration should look like. I see multiple options, but the one I

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Marko Tiikkaja
On 8/20/15 12:57 PM, Magnus Hagander wrote: I mean that you could write radius_server=foo or radius_servers=foo as well as radius_server=foo,bar and radius_servers=foo,bar. As long as you don't specify both radius_server and radius_servers, either one of them should accept either one server or

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 5:02 PM, Greg Stark st...@mit.edu wrote: I haven't thought through the exponential growth carefully enough to tell if doubling the run size should decrease the number of passes linearly or by a constant number. It seems that with 5 times the data that previously

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Greg Stark
On Thu, Aug 20, 2015 at 11:16 PM, Peter Geoghegan p...@heroku.com wrote: It could reduce seek time, which might be the dominant cost (but not I/O as such). No I didn't quite follow the argument to completion. Increasing the run size is a win if it reduces the number of passes. In the

Re: [HACKERS] Autonomous Transaction is back

2015-08-20 Thread Rajeev rastogi
On 18 August 2015 21:18, Robert Haas Wrote: This footnote goes to my point. It seems clear to me that having the autonomous transaction see the effects of the outer uncommitted transaction is a recipe for trouble. If the autonomous transaction updates a row and commits, and the outer transaction

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-20 Thread Kouhei Kaigai
On 08/19/2015 03:53 PM, Tom Lane wrote: I don't see anything very wrong with constraining the initial allocation to 1GB, or even less. That will prevent consuming insane amounts of work_mem when the planner's rows estimate is too high rather than too low. And we do have the ability to

Re: [HACKERS] 9.5 release notes

2015-08-20 Thread Tom Lane
Arthur Silva arthur...@gmail.com writes: Are we landing pg_tgrm 1.2 in pg 9.5? No, we aren't. And please don't quote 70 lines of unrelated stuff before making your point. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] 9.5 release notes

2015-08-20 Thread Arthur Silva
On Sat, Aug 8, 2015 at 11:04 PM, Bruce Momjian br...@momjian.us wrote: On Sun, Aug 9, 2015 at 01:24:33AM +1200, David Rowley wrote: On 7 August 2015 at 14:24, Bruce Momjian br...@momjian.us wrote: On Tue, Jun 30, 2015 at 09:00:44PM +0200, Andres Freund wrote: * 2014-12-08

Re: [HACKERS] [PATCH] postgres_fdw extension support

2015-08-20 Thread Michael Paquier
On Thu, Jul 23, 2015 at 11:48 PM, Paul Ramsey pram...@cleverelephant.ca wrote: On Wed, Jul 22, 2015 at 12:19 PM, Paul Ramsey pram...@cleverelephant.ca wrote: I’ll have a look at doing invalidation for the case of changes to the FDW wrappers and servers. Here's an updated patch that clears

Re: [HACKERS] TAP tests are badly named

2015-08-20 Thread Andrew Dunstan
On 08/16/2015 08:30 PM, Michael Paquier wrote: On Mon, Aug 17, 2015 at 7:15 AM, Noah Misch n...@leadboat.com wrote: On Sun, Aug 16, 2015 at 05:08:56PM -0400, Andrew Dunstan wrote: On 08/16/2015 02:23 PM, Noah Misch wrote: -sub tapcheck +sub tap_check { - InstallTemp(); + die Tap tests

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Josh Berkus
On 08/20/2015 06:19 AM, David Fetter wrote: On Thu, Aug 20, 2015 at 06:58:24PM +0900, Amit Langote wrote: Do you mean ATTACH and DETACH, if they require access exclusive lock on the parent, should not be in the first cut? Or am I misreading? Sorry I was unclear. ATTACH and DETACH should

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 8:15 AM, Simon Riggs si...@2ndquadrant.com wrote: On 20 August 2015 at 03:24, Peter Geoghegan p...@heroku.com wrote: The patch is ~3.25x faster than master I've tried to read this post twice and both times my work_mem overflowed. ;-) Can you summarize what this

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Tom Lane
Greg Stark st...@mit.edu writes: Alternately, has anyone tested whether Timsort would work well? I think that was proposed a few years ago and did not look so good in simple testing. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] PostgreSQL for VAX on NetBSD/OpenBSD

2015-08-20 Thread Greg Stark
On Wed, Jun 25, 2014 at 6:05 PM, John Klos j...@ziaspace.com wrote: While I wouldn't be surprised if you remove the VAX code because not many people are going to be running PostgreSQL, I'd disagree with the assessment that this port is broken. It compiles, it initializes databases, it runs, et

Re: [HACKERS] Extension upgrade and GUCs

2015-08-20 Thread Tom Lane
Paul Ramsey pram...@cleverelephant.ca writes: On August 20, 2015 at 2:17:31 AM, Simon Riggs (si...@2ndquadrant.com(mailto:si...@2ndquadrant.com)) wrote: Sounds like we need RedefineCustomStringVariable()  Yes, if that had existed we would not have had any problems (as long as it delegated

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Marko Tiikkaja
On 8/20/15 4:25 PM, Magnus Hagander wrote: On Aug 20, 2015 4:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: Our expectations about forward compatibility for postgresql.conf entries have always been pretty low; even more so for not-widely-used settings. In any case, wouldn't what you describe simply

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Magnus Hagander
On Aug 20, 2015 4:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Aug 20, 2015 at 2:36 AM, Marko Tiikkaja ma...@joh.to wrote: On 2015-08-20 02:29, Tom Lane wrote: Why add new GUCs for that? Can't we just redefine radiusserver as a list of

Re: [HACKERS] PostgreSQL for VAX on NetBSD/OpenBSD

2015-08-20 Thread Tom Lane
Greg Stark st...@mit.edu writes: So I've been playing with this a bit. I have simh running on my home server as a Vax 3900 with NetBSD 6.1.5. My home server was mainly intended to be a SAN and its cpu is woefully underpowered so the resulting VAX is actually very very slow. So slow I wonder

Re: [HACKERS] proposal: contrib module - generic command scheduler

2015-08-20 Thread Alvaro Herrera
Pavel Stehule wrote: Hi, Job schedulers are important and sometimes very complex part of any software. PostgreSQL miss it. I propose new contrib module, that can be used simply for some tasks, and that can be used as base for other more richer schedulers. I prefer minimalist design - but

Re: [HACKERS] Freeze avoidance of very large table.

2015-08-20 Thread Alvaro Herrera
Jim Nasby wrote: I think things like pageinspect are very different; I really can't see any use for those beyond debugging (and debugging by an expert at that). I don't think that necessarily means it must continue to be in contrib. Quite the contrary, I think it is a tool critical enough that

[HACKERS] exposing pg_controldata and pg_config as functions

2015-08-20 Thread Andrew Dunstan
Is there any significant interest in either of these? Josh Berkus tells me that he would like pg_controldata information, and I was a bit interested in pg_config information, for this reason: I had a report of someone who had configured using --with-libxml but the xml tests actually returned

Re: [HACKERS] Supporting fallback RADIUS server(s)

2015-08-20 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Thu, Aug 20, 2015 at 2:36 AM, Marko Tiikkaja ma...@joh.to wrote: On 2015-08-20 02:29, Tom Lane wrote: Why add new GUCs for that? Can't we just redefine radiusserver as a list of servers to try in sequence, and similarly split radiusport into a

Re: [HACKERS] Extension upgrade and GUCs

2015-08-20 Thread Paul Ramsey
On August 20, 2015 at 7:21:10 AM, Tom Lane (t...@sss.pgh.pa.us(mailto:t...@sss.pgh.pa.us)) wrote: I'm not sure that the situation you describe can be expected to work reliably; the problems are far wider than just GUC variables. If two different .so's are exposing broadly the same set of C

Re: [HACKERS] Extension upgrade and GUCs

2015-08-20 Thread Simon Riggs
On 20 August 2015 at 13:21, Paul Ramsey pram...@cleverelephant.ca wrote: On August 20, 2015 at 2:17:31 AM, Simon Riggs (si...@2ndquadrant.com (mailto:si...@2ndquadrant.com)) wrote: On 18 August 2015 at 21:03, Paul Ramsey wrote: So I need a way to either (a) notice when I already have a

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Greg Stark
On Thu, Aug 20, 2015 at 3:24 AM, Peter Geoghegan p...@heroku.com wrote: I believe, in general, that we should consider a multi-pass sort to be a kind of inherently suspect thing these days, in the same way that checkpoints occurring 5 seconds apart are: not actually abnormal, but something

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread David Fetter
On Thu, Aug 20, 2015 at 06:58:24PM +0900, Amit Langote wrote: On 2015-08-19 PM 09:52, David Fetter wrote: On Wed, Aug 19, 2015 at 04:30:39PM +0900, Amit Langote wrote: There are use cases where we need to warn people that their assertions need to be true, and if those assertions are not

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread David Steele
On 8/20/15 5:45 AM, Amit Langote wrote: On 2015-08-20 PM 06:27, Pavan Deolasee wrote: On Tue, Aug 18, 2015 at 4:00 PM, Amit Langote langote_amit...@lab.ntt.co.jp wrote: PARTITION BY LIST ON (name) PARTITION BY RANGE ON (year, month) PARTITION BY LIST ON ((lower(left(name, 2))) PARTITION

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Amit Langote
On 2015-08-21 AM 06:27, David Fetter wrote: By the last sentence, do you mean only UPDATEs to the partition key that cause rows to jump partitions or simply any UPDATEs to the partition key? I don't know what Simon had in mind, but it seems to me that we have the following in descending

Re: [HACKERS] jsonb array-style subscripting

2015-08-20 Thread Josh Berkus
On 08/20/2015 12:24 PM, Jim Nasby wrote: On 8/17/15 4:25 PM, Josh Berkus wrote: On 08/17/2015 02:18 PM, Jim Nasby wrote: On 8/17/15 3:33 PM, Josh Berkus wrote: Again, how do we handle missing keys? Just return NULL? or ERROR? I'd prefer the former, but there will be arguments the other

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread David Fetter
On Thu, Aug 20, 2015 at 11:16:37AM +0900, Amit Langote wrote: On 2015-08-19 PM 09:23, Simon Riggs wrote: On 18 August 2015 at 11:30, Amit Langote langote_amit...@lab.ntt.co.jp wrote: You haven't specified what would happen if an UPDATE would change a row's partition. I'm happy to add

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Stefan Kaltenbrunner
On 08/20/2015 06:09 PM, Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I wonder if we should have a default of capping the dump to say 1k lines or such and only optionally do a full one. -1. It's worked like this for the last fifteen years or thereabouts, and you're

Re: [HACKERS] Annotating pg_dump/pg_restore

2015-08-20 Thread Guillaume Lelarge
2015-08-20 18:43 GMT+02:00 Kevin Burke bu...@shyp.com: Hi, Normally I try to annotate incoming queries, to make it easier to diagnose slow ones. For example: -- Users.findByPhoneNumber SELECT * FROM The pg_dump and pg_restore commands issue a COPY with no possibility of adding a

[HACKERS] remove unused ExecGetScanType

2015-08-20 Thread Alvaro Herrera
I don't see any reason not to remove this. It's been unused since a191a169d6d0b9558da4519e66510c4540204a51, dated Jan 10 2007. -- Álvaro Herrera Peñalolén, Chile La realidad se compone de muchos sueños, todos ellos diferentes, pero en cierto aspecto, parecidos... (Yo, hablando de sueños

Re: [HACKERS] allowing wal_level change at run time

2015-08-20 Thread Peter Eisentraut
On 8/19/15 9:32 AM, Andres Freund wrote: I agree that we want both. But requiring a restart is a hard stop, whereas making configuration easier is a soft feature. I don't think it makes that much of a difference for people new to postgres. People new to postgres are not the only audience

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 12:42 PM, Feng Tian ft...@vitessedata.com wrote: Just a quick anecdotal evidence. I did similar experiment about three years ago. The conclusion was that if you have SSD, just do quick sort and forget the longer runs, but if you are using hard drives, longer runs is

Re: [HACKERS] exposing pg_controldata and pg_config as functions

2015-08-20 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/20/2015 06:59 AM, Andrew Dunstan wrote: Is there any significant interest in either of these? Josh Berkus tells me that he would like pg_controldata information, and I was a bit interested in pg_config information, for this reason: I had a

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Corey Huinker
It seems the way of specifying per-partition definition/constraint, especially for range partitioning, would have a number of interesting alternatives. By the way, the [USING opclass_name] bit is just a way of telling that a particular key column has user-defined notion of ordering in case

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Simon Riggs
On 20 August 2015 at 03:24, Peter Geoghegan p...@heroku.com wrote: The patch is ~3.25x faster than master I've tried to read this post twice and both times my work_mem overflowed. ;-) Can you summarize what this patch does? I understand clearly what it doesn't do... -- Simon Riggs

Re: [HACKERS] PostgreSQL for VAX on NetBSD/OpenBSD

2015-08-20 Thread Greg Stark
On Thu, Aug 20, 2015 at 4:13 PM, David Brownlee a...@absd.org wrote: 2) The initdb problem is actually not our fault. It looks like a NetBSD kernel bug when allocating large shared memory blocks on a machine without lots of memory. There's not much initdb can do with a kernel panic... That

Re: [HACKERS] PostgreSQL for VAX on NetBSD/OpenBSD

2015-08-20 Thread Greg Stark
On Thu, Aug 20, 2015 at 3:29 PM, Tom Lane t...@sss.pgh.pa.us wrote: 4) One of the tablesample tests seems to freeze indefinitely. I haven't looked into why yet. That might indeed indicate that the spinlock code isn't working? The tablesample tests seem like a not-very-likely first place for

[HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Stefan Kaltenbrunner
Hi all! We just had a case of a very long running process of ours that creates does a lot of prepared statements through Perls DBD:Pg running into: https://rt.cpan.org/Public/Bug/Display.html?id=88827 This resulted in millions of prepared statements created, but not removed in the affected

Re: [HACKERS] allowing wal_level change at run time

2015-08-20 Thread Andres Freund
On 2015-08-20 15:11:02 -0400, Peter Eisentraut wrote: It seems to me that this would effectively replace the wal_level parameter with the presence or absence of a magic replication slot. That doesn't seem like a net improvement. It just replaces one well-known configuration mechanism with a

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 1:28 PM, Feng Tian ft...@vitessedata.com wrote: Agree everything in principal,except one thing -- no, random IO on HDD in 2010s (relative to CPU/Memory/SSD), is not any faster than tape in 1970s. :-) Sure. The advantage of replacement selection could be a deciding

Re: [HACKERS] jsonb array-style subscripting

2015-08-20 Thread Andrew Dunstan
On 08/20/2015 03:24 PM, Jim Nasby wrote: On 8/17/15 4:25 PM, Josh Berkus wrote: On 08/17/2015 02:18 PM, Jim Nasby wrote: On 8/17/15 3:33 PM, Josh Berkus wrote: Again, how do we handle missing keys? Just return NULL? or ERROR? I'd prefer the former, but there will be arguments the other

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Feng Tian
On Thu, Aug 20, 2015 at 1:16 PM, Peter Geoghegan p...@heroku.com wrote: On Thu, Aug 20, 2015 at 12:42 PM, Feng Tian ft...@vitessedata.com wrote: Just a quick anecdotal evidence. I did similar experiment about three years ago. The conclusion was that if you have SSD, just do quick sort

Re: [HACKERS] proposal: function parse_ident

2015-08-20 Thread Jim Nasby
On 8/19/15 7:22 PM, Tom Lane wrote: Jim Nasby jim.na...@bluetreble.com writes: Don't say parse names for things other than tables. Only a minority of the types of objects used in the database have names that meet this specification. Really? My impression is that almost everything that's not

Re: [HACKERS] jsonb array-style subscripting

2015-08-20 Thread Jim Nasby
On 8/17/15 4:25 PM, Josh Berkus wrote: On 08/17/2015 02:18 PM, Jim Nasby wrote: On 8/17/15 3:33 PM, Josh Berkus wrote: Again, how do we handle missing keys? Just return NULL? or ERROR? I'd prefer the former, but there will be arguments the other way. I've been wondering if we should add

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Feng Tian
On Thu, Aug 20, 2015 at 10:41 AM, Peter Geoghegan p...@heroku.com wrote: On Thu, Aug 20, 2015 at 8:15 AM, Simon Riggs si...@2ndquadrant.com wrote: On 20 August 2015 at 03:24, Peter Geoghegan p...@heroku.com wrote: The patch is ~3.25x faster than master I've tried to read this post

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-20 Thread Tomas Vondra
Hi, On 08/19/2015 03:53 PM, Tom Lane wrote: I don't see anything very wrong with constraining the initial allocation to 1GB, or even less. That will prevent consuming insane amounts of work_mem when the planner's rows estimate is too high rather than too low. And we do have the ability to

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 6:05 AM, Greg Stark st...@mit.edu wrote: On Thu, Aug 20, 2015 at 3:24 AM, Peter Geoghegan p...@heroku.com wrote: I believe, in general, that we should consider a multi-pass sort to be a kind of inherently suspect thing these days, in the same way that checkpoints

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Tomas Vondra
Hi, On 08/20/2015 11:04 PM, Stefan Kaltenbrunner wrote: On 08/20/2015 06:09 PM, Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I wonder if we should have a default of capping the dump to say 1k lines or such and only optionally do a full one. -1. It's worked like this

Re: [HACKERS] PostgreSQL for VAX on NetBSD/OpenBSD

2015-08-20 Thread David Brownlee
On 20 August 2015 at 14:54, Greg Stark st...@mit.edu wrote: On Wed, Jun 25, 2014 at 6:05 PM, John Klos j...@ziaspace.com wrote: While I wouldn't be surprised if you remove the VAX code because not many people are going to be running PostgreSQL, I'd disagree with the assessment that this port

Re: [HACKERS] Declarative partitioning

2015-08-20 Thread Corey Huinker
My experiences with Oracle's hash function were generally not good - there's a reason many hash algorithms exist. If/when we do hash partitioning in Postgres I'd like to see the hash function be user-definable. +1 In my experience, hash partitioning had one use: When you had run out of

Re: [HACKERS] deparsing utility commands

2015-08-20 Thread Alvaro Herrera
Shulgin, Oleksandr wrote: Another quirk of ALTER TABLE is that due to multi-pass processing in ATRewriteCatalogs, the same command might be collected a number of times. For example, in src/test/regress/sql/inherit.sql: alter table a alter column aa type integer using bit_length(aa); the

[HACKERS] Annotating pg_dump/pg_restore

2015-08-20 Thread Kevin Burke
Hi, Normally I try to annotate incoming queries, to make it easier to diagnose slow ones. For example: -- Users.findByPhoneNumber SELECT * FROM The pg_dump and pg_restore commands issue a COPY with no possibility of adding a comment. It would be useful to know who or what exactly is

Re: [HACKERS] deparsing utility commands

2015-08-20 Thread Alvaro Herrera
Shulgin, Oleksandr wrote: A particularly nasty one is: ERROR: index cwi_replaced_pkey does not exist The test statement that's causing it is this one: ALTER TABLE cwi_test DROP CONSTRAINT cwi_uniq_idx, ADD CONSTRAINT cwi_replaced_pkey PRIMARY KEY USING INDEX cwi_uniq2_idx; Which

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Joshua D. Drake
On 08/20/2015 08:51 AM, Stefan Kaltenbrunner wrote: This is 9.1.14 on Debian Wheezy/amd64 fwiw - but I dont think we have made relevant changes in more recent versions. It seems we may also want to consider a way to drop those prepared queries after a period time of non use. JD

Re: [HACKERS] PostgreSQL for VAX on NetBSD/OpenBSD

2015-08-20 Thread Andres Freund
On 2015-08-20 16:42:21 +0100, Greg Stark wrote: Ah, I was wrong. It's not the tablesample test -- I think that was the last one to complete. Annoyingly we don't seem to print test names until they finish. It was groupingsets. And it's stuck again on the same query: regression=# select

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Tom Lane
Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I wonder if we should have a default of capping the dump to say 1k lines or such and only optionally do a full one. -1. It's worked like this for the last fifteen years or thereabouts, and you're the first one to complain. I suspect some

Re: [HACKERS] deparsing utility commands

2015-08-20 Thread Shulgin, Oleksandr
On Thu, Aug 20, 2015 at 4:28 PM, Shulgin, Oleksandr oleksandr.shul...@zalando.de wrote: Which gets deparsed as: ALTER TABLE cwi_test DROP CONSTRAINT cwi_uniq_idx, ADD CONSTRAINT cwi_replaced_pkey PRIMARY KEY USING INDEX cwi_replaced_pkey; The problem is that during constraint