Re: [HACKERS] WITH CHECK OPTION for auto-updatable views

2013-06-22 Thread Stephen Frost
Dean, * Dean Rasheed (dean.a.rash...@gmail.com) wrote: Here's an updated version --- I missed the necessary update to the check_option column of information_schema.views. Thanks! This is really looking quite good, but it's a bit late and I'm going on vacation tomorrow, so I didn't quite want

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Stephen Frost
Noah, * Noah Misch (n...@leadboat.com) wrote: This patch introduces MemoryContextAllocHuge() and repalloc_huge() that check a higher MaxAllocHugeSize limit of SIZE_MAX/2. Nice! I've complained about this limit a few different times and just never got around to addressing it. This was made

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Simon Riggs
On 13 May 2013 15:26, Noah Misch n...@leadboat.com wrote: A memory chunk allocated through the existing palloc.h interfaces is limited to MaxAllocSize (~1 GiB). This is best for most callers; SET_VARSIZE() need not check its own 1 GiB limit, and algorithms that grow a buffer by doubling need

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Simon Riggs
On 22 June 2013 08:46, Stephen Frost sfr...@snowman.net wrote: The next limit faced by sorts is INT_MAX concurrent tuples in memory, which limits helpful work_mem to about 150 GiB when sorting int4. That's frustratingly small. :( But that has nothing to do with this patch, right? And is

Re: [HACKERS] XLogInsert scaling, revisited

2013-06-22 Thread Heikki Linnakangas
On 21.06.2013 21:55, Jeff Janes wrote: I think I'm getting an undetected deadlock between the checkpointer and a user process running a TRUNCATE command. This is the checkpointer: #0 0x003a73eeaf37 in semop () from /lib64/libc.so.6 #1 0x005ff847 in PGSemaphoreLock

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: On 22 June 2013 08:46, Stephen Frost sfr...@snowman.net wrote: The next limit faced by sorts is INT_MAX concurrent tuples in memory, which limits helpful work_mem to about 150 GiB when sorting int4. That's frustratingly small. :( But that

[HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
Previous discussions of Hash Joins have noted that the performance decreases when the average number of tuples per bucket increases. O(N^2) effects are seen. We've argued this about many different ways, yet all of those discussions have centred around the constant NTUP_PER_BUCKET. I believe that

[Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Cédric Villemain
Le lundi 17 juin 2013 00:02:21, Fabien COELHO a écrit : What activity would you expect? A patch which applies cleanly to git HEAD. This one doesn't for me, although I'm not really sure why, I don't see any obvious conflicts. Please find attached a freshly generated patch against

Re: [HACKERS] Hardware donation

2013-06-22 Thread Simon Riggs
On 21 June 2013 20:03, Jim Nasby j...@nasby.net wrote: Who can be point of contact from the community to arrange shipping, etc? Do they need to be shipped? Can we just leave them where they are and arrange access and power charges to be passed to SPI? Sounds like it would be cheaper and easier

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Andres Freund
On 2013-06-22 12:50:52 +0900, Michael Paquier wrote: On Fri, Jun 21, 2013 at 10:47 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. Looking at how this is currently used - I am afraid it's not correct... the reason RelationGetIndexList() returns a copy is that cache invalidations will

Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Andres Freund
On 2013-06-21 23:19:27 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Jun 21, 2013 at 5:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: The traditional theory has been that that would be less robust, not more so. Child backends are (mostly) able to carry out queries

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Michael Paquier
On Sat, Jun 22, 2013 at 10:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-22 12:50:52 +0900, Michael Paquier wrote: On Fri, Jun 21, 2013 at 10:47 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. Looking at how this is currently used - I am afraid it's not correct... the

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Andres Freund
On 2013-06-22 22:45:26 +0900, Michael Paquier wrote: On Sat, Jun 22, 2013 at 10:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-22 12:50:52 +0900, Michael Paquier wrote: By looking at the comments of RelationGetIndexList:relcache.c, actually the method of the patch is

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: Previous discussions of Hash Joins have noted that the performance decreases when the average number of tuples per bucket increases. Having the hash table so small that we have hash bucket collisions with different 32bit hash values is

Re: [HACKERS] Unaccent performance

2013-06-22 Thread Andres Freund
On 2013-06-21 22:52:04 +0100, Thom Brown wrote: CREATE OR REPLACE FUNCTION public.myunaccent(sometext text) RETURNS text LANGUAGE sql IMMUTABLE AS $function$ SELECT

Re: [HACKERS] Implementing incremental backup

2013-06-22 Thread Cédric Villemain
Le samedi 22 juin 2013 01:09:20, Jehan-Guillaume (ioguix) de Rorthais a écrit : On 20/06/2013 03:25, Tatsuo Ishii wrote: On Wed, Jun 19, 2013 at 8:40 PM, Tatsuo Ishii is...@postgresql.org wrote: On Wed, Jun 19, 2013 at 6:20 PM, Stephen Frost sfr...@snowman.net wrote: * Claudio Freire

Re: [HACKERS] Implementing incremental backup

2013-06-22 Thread Andres Freund
On 2013-06-22 15:58:35 +0200, Cédric Villemain wrote: A differential backup resulting from a bunch of WAL between W1 and Wn would help to recover much faster to the time of Wn than replaying all the WALs between W1 and Wn and saves a lot of space. I was hoping to find some time to dig

Re: [HACKERS] Possible bug in CASE evaluation

2013-06-22 Thread Andres Freund
On 2013-06-21 16:45:28 +0200, Andres Freund wrote: On 2013-06-21 09:51:05 -0400, Noah Misch wrote: That being said, if we discover a simple-enough fix that performs well, we may as well incorporate it. What about passing another parameter down eval_const_expressions_mutator (which is

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-22 Thread Bruce Momjian
On Mon, Jun 10, 2013 at 07:28:24AM +0800, Craig Ringer wrote: (I'm still learning the details of Pg's WAL, WAL replay and recovery, so the below's just my understanding): The problem is that WAL for all tablespaces is mixed together in the archives. If you lose your tablespace then you have

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
On 22 June 2013 14:48, Stephen Frost sfr...@snowman.net wrote: Based on your argument that we want to have a bucket load which is, on average, the size of NTUP_PER_BUCKET, I have to '-1' on this. That is not my argument. I am pointing out that the comments claim the code does that, and yet the

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-22 Thread Fabien COELHO
Dear Robert and Greg, I think so. If it doesn't get fixed now, it's not likely to get fixed later. And the fact that nobody understands why it's happening is kinda worrisome... Possibly, but I thing that it is not my fault:-) So, I spent some time at performance debugging... First, I

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Alvaro Herrera
Andres Freund escribió: On 2013-06-22 22:45:26 +0900, Michael Paquier wrote: And I imagine that you have the same problem even with RelationGetIndexList, not only RelationGetIndexListIfInvalid, because this would appear as long as you try to open more than 1 index with an index list.

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Alvaro Herrera
MauMau escribió: Are you suggesting simplifying the following part in ServerLoop()? I welcome the idea if this condition becomes simpler. However, I cannot imagine how. if (AbortStartTime 0 /* SIGKILL only once */ (Shutdown == ImmediateShutdown || (FatalError !SendStop)) now -

[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Andres Freund
On 2013-06-21 16:56:57 -0400, Alvaro Herrera wrote: What we could do to improve the robustness a bit - at least on linux - is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be killed if the postmaster goes away... This is an interesting idea (even if it has no

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-22 Thread Fabien COELHO
Please find attached a v12, which under timer_exceeded interrupts clients which are being throttled instead of waiting for the end of the transaction, as the transaction is not started yet. I've also changed the log format that I used for debugging the apparent latency issue: x y z

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-22 Thread Fabien COELHO
Please find attached a v12, which under timer_exceeded interrupts clients which are being throttled instead of waiting for the end of the transaction, as the transaction is not started yet. Oops, I forgot the attachment. Here it is! -- Fabien.diff --git a/contrib/pgbench/pgbench.c

Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Robert Haas
On Fri, Jun 21, 2013 at 11:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think that's the Tom Lane theory. The Robert Haas theory is that if the postmaster has died, there's no reason to suppose that it hasn't corrupted shared memory on the way down, or that the system isn't otherwise heavily

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 9:15 AM, Simon Riggs si...@2ndquadrant.com wrote: Previous discussions of Hash Joins have noted that the performance decreases when the average number of tuples per bucket increases. O(N^2) effects are seen. We've argued this about many different ways, yet all of those

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 9:48 AM, Stephen Frost sfr...@snowman.net wrote: The correct calculation that would match the objective set out in the comment would be dbuckets = (hash_table_bytes / tupsize) / NTUP_PER_BUCKET; This looks to be driving the size of the hash table size off of how

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 3:46 AM, Stephen Frost sfr...@snowman.net wrote: I'm not a huge fan of moving directly to INT_MAX. Are we confident that everything can handle that cleanly..? I feel like it might be a bit safer to shy a bit short of INT_MAX (say, by 1K). Maybe it would be better to

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 9:16 AM, Cédric Villemain ced...@2ndquadrant.com wrote: patch is in unified format and apply on HEAD. patch contains documentation, however I believe 'AS IMPLICIT' is a PostgreSQL extension with special behavior and 'AS EXPLICIT' respect the standard except that

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Robert Haas
On Fri, Jun 21, 2013 at 10:02 PM, MauMau maumau...@gmail.com wrote: I'm comfortable with 5 seconds. We are talking about the interval between sending SIGQUIT to the children and then sending SIGKILL to them. In most situations, the backends should terminate immediately. However, as I said a

Re: [HACKERS] dump difference between 9.3 and master after upgrade

2013-06-22 Thread Andrew Dunstan
On 06/20/2013 11:16 AM, I wrote: On 06/20/2013 10:43 AM, Robert Haas wrote: On Tue, Jun 18, 2013 at 12:18 PM, Andrew Dunstan and...@dunslane.net wrote: As I was updating my cross version upgrade tester to include support for the 9.3 branch, I noted this dump difference between the dump of

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Fabien COELHO
Hello Robert, I object to this patch. This patch a new keyword, EXPLICIT, for reasons that are strictly cosmetic. Everything that you can do with this patch can also be done without this patch. It is not a good idea to slow down parsing of every SQL statement we have just so that someone

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Fabien COELHO
Hello Cédric, So maybe it is possible to rephrase this piece: - literalAS IMPLICIT/ is a productnamePostgreSQL/productname - extension, too. + literalAS IMPLICIT/ and literalAS EXPLICIT/ are + a productnamePostgreSQL/productname extension, too. Ok. Back in 2012 Tom exposed

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Fabien COELHO
I flag it 'return with feedback', please update the patch so it builds. Everything else is ok. Here it is. -- Fabien.diff --git a/doc/src/sgml/ref/create_cast.sgml b/doc/src/sgml/ref/create_cast.sgml index 29ea298..0ace996 100644 --- a/doc/src/sgml/ref/create_cast.sgml +++

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
On Saturday, June 22, 2013, Simon Riggs wrote: On 22 June 2013 14:48, Stephen Frost sfr...@snowman.net javascript:; wrote: Based on your argument that we want to have a bucket load which is, on average, the size of NTUP_PER_BUCKET, I have to '-1' on this. That is not my argument. I am

Re: [HACKERS] [PATCH] add --progress option to pgbench (submission 3)

2013-06-22 Thread Fabien COELHO
Hello Mitsumasa, Thanks for the review. * 2. Output format in result for more readable. 5.0 s[thread 1]: tps = 1015.576032, AverageLatency(ms) = 0.000984663 5.0 s[thread 0]: tps = 1032.580794, AverageLatency(ms) = 0.000968447 10.0 s [thread 0]: tps = 1129.591189, AverageLatency(ms) =

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Heikki Linnakangas
On 22.06.2013 19:19, Simon Riggs wrote: So I think that (2) is the best route: Given that we know with much better certainty the number of rows in the scanned-relation, we should be able to examine our hash table after it has been built and decide whether it would be cheaper to rebuild the hash

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
On 22 June 2013 21:40, Stephen Frost sfr...@snowman.net wrote: I'm actually not a huge fan of this as it's certainly not cheap to do. If it can be shown to be better than an improved heuristic then perhaps it would work but I'm not convinced. We need two heuristics, it would seem: * an

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
On Saturday, June 22, 2013, Heikki Linnakangas wrote: On 22.06.2013 19:19, Simon Riggs wrote: So I think that (2) is the best route: Given that we know with much better certainty the number of rows in the scanned-relation, we should be able to examine our hash table after it has been built

Re: [HACKERS] Support for RANGE ... PRECEDING windows in OVER

2013-06-22 Thread ian link
Thanks Craig! That definitely does help. I probably still have some questions but I think I will read through the rest of the code before asking. Thanks again! Ian Craig Ringer Friday, June 21, 2013 8:41 PM On 06/22/2013 03:30 AM, ian link wrote: Forgive my ignorance, but I don't entirely

Re: [HACKERS] Patch for fast gin cache performance improvement

2013-06-22 Thread ian link
Looks like my community login is still not working. No rush, just wanted to let you know. Thanks! Ian On Tue, Jun 18, 2013 at 11:41 AM, Ian Link i...@ilink.io wrote: No worries! I'll just wait until it's up again. Thanks Ian Kevin Grittner kgri...@ymail.com Tuesday, June 18, 2013

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
On Saturday, June 22, 2013, Simon Riggs wrote: On 22 June 2013 21:40, Stephen Frost sfr...@snowman.net javascript:; wrote: I'm actually not a huge fan of this as it's certainly not cheap to do. If it can be shown to be better than an improved heuristic then perhaps it would work but

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread MauMau
From: Alvaro Herrera alvhe...@2ndquadrant.com MauMau escribió: I thought of adding some new state of pmState for some reason (that might be the same as your idea). But I refrained from doing that, because pmState has already many states. I was afraid adding a new pmState value for this bug fix

[HACKERS] [PATCH] Revive line type

2013-06-22 Thread rui hua
Hi, Test results are as follows: Contents Purpose This patch is for finishing the line type and related functions that not done yet but listed in catalogs and documentation. There are no other new features added in this patch. The regression test cases which included in this patch,

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread MauMau
From: Robert Haas robertmh...@gmail.com On Fri, Jun 21, 2013 at 10:02 PM, MauMau maumau...@gmail.com wrote: I'm comfortable with 5 seconds. We are talking about the interval between sending SIGQUIT to the children and then sending SIGKILL to them. In most situations, the backends should