Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
On 23 June 2013 03:16, Stephen Frost wrote: > Still doesn't really address the issue of dups though. Checking for duplicates in all cases would be wasteful, since often we are joining to the PK of a smaller table. If duplicates are possible at all for a join, then it would make sense to build

[HACKERS] wrong state of patch in CF

2013-06-22 Thread Amit kapila
Patch (https://commitfest.postgresql.org/action/patch_view?id=1161) is already committed by Commit http://git.postgresql.org/pg/commitdiff/b23160889c963dfe23d8cf1f9be64fb3c535a2d6 It should be marked as Committed in CF. With Regards, Amit Kapila. -- Sent via pgsql-hackers mailing list (pgsql-

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread MauMau
From: "Robert Haas" On Fri, Jun 21, 2013 at 10:02 PM, MauMau wrote: I'm comfortable with 5 seconds. We are talking about the interval between sending SIGQUIT to the children and then sending SIGKILL to them. In most situations, the backends should terminate immediately. However, as I said

[HACKERS] [PATCH] Revive line type

2013-06-22 Thread rui hua
Hi, Test results are as follows: Contents & Purpose This patch is for finishing the line type and related functions that not done yet but listed in catalogs and documentation. There are no other new features added in this patch. The regression test cases which included in this patch, Documentat

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread MauMau
From: "Alvaro Herrera" MauMau escribió: I thought of adding some new state of pmState for some reason (that might be the same as your idea). But I refrained from doing that, because pmState has already many states. I was afraid adding a new pmState value for this bug fix would complicate the s

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
On Saturday, June 22, 2013, Simon Riggs wrote: > On 22 June 2013 21:40, Stephen Frost > > wrote: > > > I'm actually not a huge fan of this as it's certainly not cheap to do. > If it > > can be shown to be better than an improved heuristic then perhaps it > would > > work but I'm not convinced. > >

Re: [HACKERS] Patch for fast gin cache performance improvement

2013-06-22 Thread ian link
Looks like my community login is still not working. No rush, just wanted to let you know. Thanks! Ian On Tue, Jun 18, 2013 at 11:41 AM, Ian Link wrote: > > No worries! I'll just wait until it's up again. > Thanks > Ian > > Kevin Grittner > Tuesday, June 18, 2013 9:15 AM > > Oops -- we seem

Re: [HACKERS] Support for RANGE ... PRECEDING windows in OVER

2013-06-22 Thread ian link
Thanks Craig! That definitely does help. I probably still have some questions but I think I will read through the rest of the code before asking. Thanks again! Ian > Craig Ringer > Friday, June 21, 2013 8:41 PM > > On 06/22/2013 03:30 AM, ian link wrote: >> >> Forgive my ignorance, but I don't en

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
On Saturday, June 22, 2013, Heikki Linnakangas wrote: > On 22.06.2013 19:19, Simon Riggs wrote: > >> So I think that (2) is the best route: Given that we know with much >> better certainty the number of rows in the scanned-relation, we should >> be able to examine our hash table after it has been

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
On 22 June 2013 21:40, Stephen Frost wrote: > I'm actually not a huge fan of this as it's certainly not cheap to do. If it > can be shown to be better than an improved heuristic then perhaps it would > work but I'm not convinced. We need two heuristics, it would seem: * an initial heuristic to

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Heikki Linnakangas
On 22.06.2013 19:19, Simon Riggs wrote: So I think that (2) is the best route: Given that we know with much better certainty the number of rows in the scanned-relation, we should be able to examine our hash table after it has been built and decide whether it would be cheaper to rebuild the hash t

Re: [HACKERS] [PATCH] add --progress option to pgbench (submission 3)

2013-06-22 Thread Fabien COELHO
Hello Mitsumasa, Thanks for the review. * 2. Output format in result for more readable. 5.0 s[thread 1]: tps = 1015.576032, AverageLatency(ms) = 0.000984663 5.0 s[thread 0]: tps = 1032.580794, AverageLatency(ms) = 0.000968447 10.0 s [thread 0]: tps = 1129.591189, AverageLatency(ms) =

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
On Saturday, June 22, 2013, Simon Riggs wrote: > On 22 June 2013 14:48, Stephen Frost > > wrote: > > > Based on your argument that we want to have a bucket load which is, on > > average, the size of NTUP_PER_BUCKET, I have to '-1' on this. > > That is not my argument. I am pointing out that the co

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Fabien COELHO
I flag it 'return with feedback', please update the patch so it builds. Everything else is ok. Here it is. -- Fabien.diff --git a/doc/src/sgml/ref/create_cast.sgml b/doc/src/sgml/ref/create_cast.sgml index 29ea298..0ace996 100644 --- a/doc/src/sgml/ref/create_cast.sgml +++ b/doc/src/sgml/ref/

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Fabien COELHO
Hello Cédric, So maybe it is possible to rephrase this piece: - AS IMPLICIT is a PostgreSQL - extension, too. + AS IMPLICIT and AS EXPLICIT are + a PostgreSQL extension, too. Ok. Back in 2012 Tom exposed arguments against it, or at least not a clear +1. The patch add nothing but mor

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Fabien COELHO
Hello Robert, I object to this patch. This patch a new keyword, EXPLICIT, for reasons that are strictly cosmetic. Everything that you can do with this patch can also be done without this patch. It is not a good idea to slow down parsing of every SQL statement we have just so that someone can

Re: [HACKERS] dump difference between 9.3 and master after upgrade

2013-06-22 Thread Andrew Dunstan
On 06/20/2013 11:16 AM, I wrote: On 06/20/2013 10:43 AM, Robert Haas wrote: On Tue, Jun 18, 2013 at 12:18 PM, Andrew Dunstan wrote: As I was updating my cross version upgrade tester to include support for the 9.3 branch, I noted this dump difference between the dump of the original 9.3 data

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Robert Haas
On Fri, Jun 21, 2013 at 10:02 PM, MauMau wrote: > I'm comfortable with 5 seconds. We are talking about the interval between > sending SIGQUIT to the children and then sending SIGKILL to them. In most > situations, the backends should terminate immediately. However, as I said a > few months ago,

Re: [Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 9:16 AM, Cédric Villemain wrote: > patch is in unified format and apply on HEAD. > patch contains documentation, however I believe 'AS IMPLICIT' is a PostgreSQL > extension with special behavior and 'AS EXPLICIT' respect the standard except > that PostgreSQL adds only the e

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 3:46 AM, Stephen Frost wrote: > I'm not a huge fan of moving directly to INT_MAX. Are we confident that > everything can handle that cleanly..? I feel like it might be a bit > safer to shy a bit short of INT_MAX (say, by 1K). Maybe it would be better to stick with INT_MA

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 9:48 AM, Stephen Frost wrote: >> The correct calculation that would match the objective set out in the >> comment would be >> >> dbuckets = (hash_table_bytes / tupsize) / NTUP_PER_BUCKET; > > This looks to be driving the size of the hash table size off of "how > many of th

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Robert Haas
On Sat, Jun 22, 2013 at 9:15 AM, Simon Riggs wrote: > Previous discussions of Hash Joins have noted that the performance > decreases when the average number of tuples per bucket increases. > O(N^2) effects are seen. > > We've argued this about many different ways, yet all of those > discussions ha

Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Robert Haas
On Fri, Jun 21, 2013 at 11:19 PM, Tom Lane wrote: >> I think that's the Tom Lane theory. The Robert Haas theory is that if >> the postmaster has died, there's no reason to suppose that it hasn't >> corrupted shared memory on the way down, or that the system isn't >> otherwise heavily fuxxored in

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-22 Thread Fabien COELHO
Please find attached a v12, which under timer_exceeded interrupts clients which are being throttled instead of waiting for the end of the transaction, as the transaction is not started yet. Oops, I forgot the attachment. Here it is! -- Fabien.diff --git a/contrib/pgbench/pgbench.c b/contrib/

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-22 Thread Fabien COELHO
Please find attached a v12, which under timer_exceeded interrupts clients which are being throttled instead of waiting for the end of the transaction, as the transaction is not started yet. I've also changed the log format that I used for debugging the apparent latency issue: x y z 12345

[HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Andres Freund
On 2013-06-21 16:56:57 -0400, Alvaro Herrera wrote: > > What we could do to improve the robustness a bit - at least on linux - > > is to prctl(PR_SET_PDEATHSIG, SIGKILL) which would cause children to be > > killed if the postmaster goes away... > > This is an interesting idea (even if it has no re

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Alvaro Herrera
MauMau escribió: > Are you suggesting simplifying the following part in ServerLoop()? > I welcome the idea if this condition becomes simpler. However, I > cannot imagine how. > if (AbortStartTime > 0 && /* SIGKILL only once */ > (Shutdown == ImmediateShutdown || (FatalError && !SendStop)) &&

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Alvaro Herrera
Andres Freund escribió: > On 2013-06-22 22:45:26 +0900, Michael Paquier wrote: > > And I imagine that you have the same problem even with > > RelationGetIndexList, not only RelationGetIndexListIfInvalid, because > > this would appear as long as you try to open more than 1 index with an > > index l

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-22 Thread Fabien COELHO
Dear Robert and Greg, I think so. If it doesn't get fixed now, it's not likely to get fixed later. And the fact that nobody understands why it's happening is kinda worrisome... Possibly, but I thing that it is not my fault:-) So, I spent some time at performance debugging... First, I fina

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
On 22 June 2013 14:48, Stephen Frost wrote: > Based on your argument that we want to have a bucket load which is, on > average, the size of NTUP_PER_BUCKET, I have to '-1' on this. That is not my argument. I am pointing out that the comments claim the code does that, and yet the code does not.

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-22 Thread Bruce Momjian
On Mon, Jun 10, 2013 at 07:28:24AM +0800, Craig Ringer wrote: > (I'm still learning the details of Pg's WAL, WAL replay and recovery, so > the below's just my understanding): > > The problem is that WAL for all tablespaces is mixed together in the > archives. If you lose your tablespace then you h

Re: [HACKERS] Possible bug in CASE evaluation

2013-06-22 Thread Andres Freund
On 2013-06-21 16:45:28 +0200, Andres Freund wrote: > On 2013-06-21 09:51:05 -0400, Noah Misch wrote: > > That being said, if we discover a simple-enough fix that performs well, we > > may > > as well incorporate it. > > What about passing another parameter down eval_const_expressions_mutator > (w

Re: [HACKERS] Implementing incremental backup

2013-06-22 Thread Andres Freund
On 2013-06-22 15:58:35 +0200, Cédric Villemain wrote: > > A differential backup resulting from a bunch of WAL between W1 and Wn > > would help to recover much faster to the time of Wn than replaying all > > the WALs between W1 and Wn and saves a lot of space. > > > > I was hoping to find some time

Re: [HACKERS] Implementing incremental backup

2013-06-22 Thread Cédric Villemain
Le samedi 22 juin 2013 01:09:20, Jehan-Guillaume (ioguix) de Rorthais a écrit : > On 20/06/2013 03:25, Tatsuo Ishii wrote: > >> On Wed, Jun 19, 2013 at 8:40 PM, Tatsuo Ishii wrote: > On Wed, Jun 19, 2013 at 6:20 PM, Stephen Frost wrote: > > * Claudio Freire (klaussfre...@gmail.com) wr

Re: [HACKERS] Unaccent performance

2013-06-22 Thread Andres Freund
On 2013-06-21 22:52:04 +0100, Thom Brown wrote: > > CREATE OR REPLACE FUNCTION public.myunaccent(sometext text) > > RETURNS text > > LANGUAGE sql > > IMMUTABLE > > AS $function$ > > SELECT > > replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace

Re: [HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Stephen Frost
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: > Previous discussions of Hash Joins have noted that the performance > decreases when the average number of tuples per bucket increases. Having the hash table so small that we have hash bucket collisions with different 32bit hash values is extre

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Andres Freund
On 2013-06-22 22:45:26 +0900, Michael Paquier wrote: > On Sat, Jun 22, 2013 at 10:34 PM, Andres Freund > wrote: > > On 2013-06-22 12:50:52 +0900, Michael Paquier wrote: > >> By looking at the comments of RelationGetIndexList:relcache.c, > >> actually the method of the patch is correct because in

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Michael Paquier
On Sat, Jun 22, 2013 at 10:34 PM, Andres Freund wrote: > On 2013-06-22 12:50:52 +0900, Michael Paquier wrote: >> On Fri, Jun 21, 2013 at 10:47 PM, Andres Freund >> wrote: >> > Hm. Looking at how this is currently used - I am afraid it's not >> > correct... the reason RelationGetIndexList() retur

Re: [HACKERS] Re: backend hangs at immediate shutdown (Re: Back-branch update releases coming in a couple weeks)

2013-06-22 Thread Andres Freund
On 2013-06-21 23:19:27 -0400, Tom Lane wrote: > Robert Haas writes: > > On Fri, Jun 21, 2013 at 5:44 PM, Tom Lane wrote: > >> The traditional theory has been that that would be less robust, not > >> more so. Child backends are (mostly) able to carry out queries whether > >> or not the postmaster

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-06-22 Thread Andres Freund
On 2013-06-22 12:50:52 +0900, Michael Paquier wrote: > On Fri, Jun 21, 2013 at 10:47 PM, Andres Freund > wrote: > > Hm. Looking at how this is currently used - I am afraid it's not > > correct... the reason RelationGetIndexList() returns a copy is that > > cache invalidations will throw away that

Re: [HACKERS] Hardware donation

2013-06-22 Thread Simon Riggs
On 21 June 2013 20:03, Jim Nasby wrote: > Who can be point of contact from the community to arrange shipping, etc? Do they need to be shipped? Can we just leave them where they are and arrange access and power charges to be passed to SPI? Sounds like it would be cheaper and easier to leave them

[Review] Re: [HACKERS] minor patch submission: CREATE CAST ... AS EXPLICIT

2013-06-22 Thread Cédric Villemain
Le lundi 17 juin 2013 00:02:21, Fabien COELHO a écrit : > >> What activity would you expect? > > > > A patch which applies cleanly to git HEAD. This one doesn't for me, > > although I'm not really sure why, I don't see any obvious conflicts. > > Please find attached a freshly generated patch aga

[HACKERS] A better way than tweaking NTUP_PER_BUCKET

2013-06-22 Thread Simon Riggs
Previous discussions of Hash Joins have noted that the performance decreases when the average number of tuples per bucket increases. O(N^2) effects are seen. We've argued this about many different ways, yet all of those discussions have centred around the constant NTUP_PER_BUCKET. I believe that w

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: > On 22 June 2013 08:46, Stephen Frost wrote: > >>The next limit faced by sorts is > >> INT_MAX concurrent tuples in memory, which limits helpful work_mem to about > >> 150 GiB when sorting int4. > > > > That's frustratingly small. :( > > But that has

Re: [HACKERS] XLogInsert scaling, revisited

2013-06-22 Thread Heikki Linnakangas
On 21.06.2013 21:55, Jeff Janes wrote: I think I'm getting an undetected deadlock between the checkpointer and a user process running a TRUNCATE command. This is the checkpointer: #0 0x003a73eeaf37 in semop () from /lib64/libc.so.6 #1 0x005ff847 in PGSemaphoreLock (sema=0x7f8c0a4e

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Simon Riggs
On 22 June 2013 08:46, Stephen Frost wrote: >>The next limit faced by sorts is >> INT_MAX concurrent tuples in memory, which limits helpful work_mem to about >> 150 GiB when sorting int4. > > That's frustratingly small. :( > But that has nothing to do with this patch, right? And is easily fixed,

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Simon Riggs
On 13 May 2013 15:26, Noah Misch wrote: > A memory chunk allocated through the existing palloc.h interfaces is limited > to MaxAllocSize (~1 GiB). This is best for most callers; SET_VARSIZE() need > not check its own 1 GiB limit, and algorithms that grow a buffer by doubling > need not check for

Re: [HACKERS] MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

2013-06-22 Thread Stephen Frost
Noah, * Noah Misch (n...@leadboat.com) wrote: > This patch introduces MemoryContextAllocHuge() and repalloc_huge() that check > a higher MaxAllocHugeSize limit of SIZE_MAX/2. Nice! I've complained about this limit a few different times and just never got around to addressing it. > This was ma