Re: [HACKERS] Performance degradation in commit ac1d794

On Tue, May 31, 2016 at 10:23 PM, Tom Lane wrote: > Noah Misch writes: >> On Tue, May 31, 2016 at 10:09:05PM -0400, Robert Haas wrote: >>> What I *think* is going on here is: >>> - ac1d794 lowered performance >>> - backend_flush_after with a non-zero

Re: [HACKERS] Performance degradation in commit ac1d794

Noah Misch writes: > On Tue, May 31, 2016 at 10:09:05PM -0400, Robert Haas wrote: >> What I *think* is going on here is: >> - ac1d794 lowered performance >> - backend_flush_after with a non-zero default lowered performance with >> a vengeance >> - 98a64d0 repaired the damage

Re: [HACKERS] Performance degradation in commit ac1d794

On Tue, May 31, 2016 at 10:09:05PM -0400, Robert Haas wrote: > On Sun, May 29, 2016 at 1:40 AM, Noah Misch wrote: > > On Fri, Dec 25, 2015 at 08:08:15PM +0300, Васильев Дмитрий wrote: > >> I suddenly found commit ac1d794 gives up to 3 times performance > >> degradation. > >> >

Re: [HACKERS] Performance degradation in commit ac1d794

On Sun, May 29, 2016 at 1:40 AM, Noah Misch wrote: > On Fri, Dec 25, 2015 at 08:08:15PM +0300, Васильев Дмитрий wrote: >> I suddenly found commit ac1d794 gives up to 3 times performance degradation. >> >> I tried to run pgbench -s 1000 -j 48 -c 48 -S -M prepared on 70 CPU-core

Re: [HACKERS] Performance degradation in commit ac1d794

On Fri, Dec 25, 2015 at 08:08:15PM +0300, Васильев Дмитрий wrote: > I suddenly found commit ac1d794 gives up to 3 times performance degradation. > > I tried to run pgbench -s 1000 -j 48 -c 48 -S -M prepared on 70 CPU-core > machine: > commit ac1d794 gives me 363,474 tps > and previous commit

Re: [HACKERS] Performance degradation in commit ac1d794

On Mon, Mar 21, 2016 at 6:09 PM, Andres Freund wrote: > On 2016-03-21 11:52:43 +1300, Thomas Munro wrote: >> * I would be interested in writing a kqueue implementation of this for >> *BSD (and MacOSX?) at some point if someone doesn't beat me to it. > > I hoped that somebody

Re: [HACKERS] Performance degradation in commit ac1d794

Hi, I've pushed the new API. We might want to use it in more places... On 2015-12-25 20:08:15 +0300, Васильев Дмитрий wrote: > I suddenly found commit ac1d794 gives up to 3 times performance degradation. > > I tried to run pgbench -s 1000 -j 48 -c 48 -S -M prepared on 70 CPU-core > machine: >

Re: [HACKERS] Performance degradation in commit ac1d794

Hi,m On 2016-03-21 11:26:43 +0530, Amit Kapila wrote: > LOG: database system was shut down at 2016-03-21 11:17:13 IST > LOG: MultiXact member wraparound protections are now enabled > LOG: database system is ready to accept connections > LOG: autovacuum launcher started > FATAL: failed to set

Re: [HACKERS] Performance degradation in commit ac1d794

On Mon, Mar 21, 2016 at 10:26 AM, Amit Kapila wrote: > > On Mon, Mar 21, 2016 at 10:21 AM, Andres Freund wrote: >> >> >> >> On March 21, 2016 5:12:38 AM GMT+01:00, Amit Kapila < amit.kapil...@gmail.com> wrote: >> >> >The article pointed by you

Re: [HACKERS] Performance degradation in commit ac1d794

Hi, On 2016-03-21 11:52:43 +1300, Thomas Munro wrote: > The contract that I invented here is that an async-aware parent node > can ask any child node "are you ready?" and get back various answers > including an fd which means please don't call ExecProcNode until this > fd is ready to read. But

Re: [HACKERS] Performance degradation in commit ac1d794

On Mon, Mar 21, 2016 at 10:21 AM, Andres Freund wrote: > > > On March 21, 2016 5:12:38 AM GMT+01:00, Amit Kapila < > amit.kapil...@gmail.com> wrote: > > >The article pointed by you justifies that the way ResetEvent is done by > >patch is correct. I am not sure, but you can

Re: [HACKERS] Performance degradation in commit ac1d794

On March 21, 2016 5:12:38 AM GMT+01:00, Amit Kapila wrote: >The article pointed by you justifies that the way ResetEvent is done by >patch is correct. I am not sure, but you can weigh, if there is a need >of >comment so that if we want enhance this part of code (or

Re: [HACKERS] Performance degradation in commit ac1d794

On Sun, Mar 20, 2016 at 7:13 AM, Andres Freund wrote: > > On 2016-03-19 15:43:27 +0530, Amit Kapila wrote: > > On Sat, Mar 19, 2016 at 12:40 PM, Andres Freund wrote: > > > > > > On March 18, 2016 11:52:08 PM PDT, Amit Kapila < amit.kapil...@gmail.com> > >

Re: [HACKERS] Performance degradation in commit ac1d794

On Mon, Mar 21, 2016 at 3:46 AM, Andres Freund wrote: > On 2016-03-21 01:31:30 +1300, Thomas Munro wrote: >> I couldn't get the second patch to apply for some reason, > > Weird? Even efter appying the first one first? > > >> but I have been trying out your "latch" branch on

Re: [HACKERS] Performance degradation in commit ac1d794

On Mon, Mar 21, 2016 at 3:46 AM, Andres Freund wrote: > On 2016-03-21 01:31:30 +1300, Thomas Munro wrote: >> I couldn't get the second patch to apply for some reason, > > Weird? Even efter appying the first one first? Ah, I was using patch -p1. I needed to use git am, which

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-18 05:56:41 -0400, Robert Haas wrote: > >> 0 at the top of the loop and skip it forthwith if so. > > > > You mean in WaitEventSetWait()? There's > > else if (rc == 0) > > { > > break; > > } > > which is the timeout

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-21 01:31:30 +1300, Thomas Munro wrote: > I couldn't get the second patch to apply for some reason, Weird? Even efter appying the first one first? > but I have been trying out your "latch" branch on some different OSs > and porting some code that does a bunch of waiting on many

Re: [HACKERS] Performance degradation in commit ac1d794

On Sun, Mar 20, 2016 at 5:14 PM, Andres Freund wrote: > On 2016-03-19 18:45:36 -0700, Andres Freund wrote: >> On 2016-03-19 16:44:49 +0530, Amit Kapila wrote: >> > On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: >> > > >> > > >> > > Attached is a

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-17 09:01:36 -0400, Robert Haas wrote: > > * Right now the caller has to allocate the WaitEvents he's waiting for > > locally (likely on the stack), but we also could allocate them as part > > of the WaitEventSet. Not sure if that'd be a benefit. > > I'm not seeing this. What do you

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-17 09:40:08 -0400, Robert Haas wrote: > On Thu, Mar 17, 2016 at 9:01 AM, Robert Haas wrote: > > I'll look at 0005 next, but thought I would send these comments along first. > > 0005: This is obviously very much WIP, but I think the overall > direction of it is

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-19 18:45:36 -0700, Andres Freund wrote: > On 2016-03-19 16:44:49 +0530, Amit Kapila wrote: > > On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: > > > > > > > > > Attached is a significantly revised version of the earlier series. Most > > > importantly I have: >

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-16 15:08:07 -0400, Robert Haas wrote: > On Wed, Mar 16, 2016 at 2:52 PM, Andres Freund wrote: > > How about the following sketch of an API > > > > typedef struct LatchEvent > > { > > uint32 events; /* interesting events */ > > uint32 revents; /*

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-19 16:44:49 +0530, Amit Kapila wrote: > On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: > > > > > > Attached is a significantly revised version of the earlier series. Most > > importantly I have: > > * Unified the window/unix latch implementation into one file

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-19 15:43:27 +0530, Amit Kapila wrote: > On Sat, Mar 19, 2016 at 12:40 PM, Andres Freund wrote: > > > > On March 18, 2016 11:52:08 PM PDT, Amit Kapila > wrote: > > >> >Won't the new code needs to ensure that ResetEvent(latchevent) > >

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-02-08 17:49:18 +0900, Kyotaro HORIGUCHI wrote: > How about allowing registration of a callback for every waiting > socket. The signature of the callback function would be like I don't think a a callback based API is going to serve us well. Most of the current latch callers would get

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Mar 17, 2016 at 9:01 AM, Robert Haas wrote: > I'll look at 0005 next, but thought I would send these comments along first. 0005: This is obviously very much WIP, but I think the overall direction of it is good. 0006: Same. I think you should use PGINVALID_SOCKET

Re: [HACKERS] Performance degradation in commit ac1d794

On Wed, Mar 16, 2016 at 3:25 PM, Andres Freund wrote: >> > - Given current users we don't need a large amount of events, so having >> > to iterate through the registered events doesn't seem bothersome. We >> > could however change the api to be something like >> > >> >

Re: [HACKERS] Performance degradation in commit ac1d794

Hi, On 2016-03-17 09:01:36 -0400, Robert Haas wrote: > 0001: Looking at this again, I'm no longer sure this is a bug. > Doesn't your patch just check the same conditions in the opposite > order? Yes, that's what's required > 0004: > > + * drain it everytime WaitLatchOrSocket() is used.

Re: [HACKERS] Performance degradation in commit ac1d794

Andres Freund wrote: > From 1d444b0855dbf65d66d73beb647b772fff3404c8 Mon Sep 17 00:00:00 2001 > From: Andres Freund > Date: Fri, 18 Mar 2016 00:52:07 -0700 > Subject: [PATCH 4/5] Combine win32 and unix latch implementations. > > Previously latches for windows and unix had

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > On Thu, Jan 14, 2016 at 12:06 PM, Andres Freund wrote: > > On 2016-01-14 11:31:03 -0500, Robert Haas wrote: > >> On Thu, Jan 14, 2016 at 10:56 AM, Andres Freund wrote: > >> I think your idea of a data

Re: [HACKERS] Performance degradation in commit ac1d794

Hi, On 2016-03-16 15:41:28 -0400, Robert Haas wrote: > On Wed, Mar 16, 2016 at 3:25 PM, Andres Freund wrote: > >> > - Given current users we don't need a large amount of events, so having > >> > to iterate through the registered events doesn't seem bothersome. We > >> >

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-18 20:14:07 +0530, Amit Kapila wrote: > + * from inside a signal handler in latch_sigusr1_handler(). > > * > > * Note: we assume that the kernel calls involved in drainSelfPipe() > > * and SetLatch() will provide adequate synchronization on machines > > * with weak memory

Re: [HACKERS] Performance degradation in commit ac1d794

On Fri, Mar 18, 2016 at 1:53 PM, Andres Freund wrote: > On 2016-03-18 05:56:41 -0400, Robert Haas wrote: >> >> 0 at the top of the loop and skip it forthwith if so. >> > >> > You mean in WaitEventSetWait()? There's >> > else if (rc == 0) >> > {

Re: [HACKERS] Performance degradation in commit ac1d794

Hi, On 2016-03-17 09:01:36 -0400, Robert Haas wrote: > 0001: Looking at this again, I'm no longer sure this is a bug. > Doesn't your patch just check the same conditions in the opposite > order? Which is important, because what's in what pfds[x] depends on wakeEvents. Folded it into a later

Re: [HACKERS] Performance degradation in commit ac1d794

On Wed, Mar 16, 2016 at 10:04 PM, Andres Freund wrote: > Questions: > * I'm kinda inclined to merge the win32 and unix latch > implementations. There's already a fair bit in common, and this is > just going to increase that amount. Don't care either way. > * Right now

Re: [HACKERS] Performance degradation in commit ac1d794

On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: > > > Attached is a significantly revised version of the earlier series. Most > importantly I have: > * Unified the window/unix latch implementation into one file (0004) > After applying patch 0004* on HEAD, using command

Re: [HACKERS] Performance degradation in commit ac1d794

On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: > > Hi, > > On 2016-03-17 09:01:36 -0400, Robert Haas wrote: > > 0001: Looking at this again, I'm no longer sure this is a bug. > > Doesn't your patch just check the same conditions in the opposite > > order? > > Which is

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Mar 17, 2016 at 10:53 AM, Andres Freund wrote: >> I wonder if there's a way to refactor this code to avoid having so >> much cut-and-paste duplication. > > I guess you mean WaitEventSetWait() and WaitEventAdjust*? I've tried, > and my attempt ended up look nearly

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-16 15:41:28 -0400, Robert Haas wrote: > On Wed, Mar 16, 2016 at 3:25 PM, Andres Freund wrote: > >> > - Given current users we don't need a large amount of events, so having > >> > to iterate through the registered events doesn't seem bothersome. We > >> > could

Re: [HACKERS] Performance degradation in commit ac1d794

On Sat, Mar 19, 2016 at 12:40 PM, Andres Freund wrote: > > On March 18, 2016 11:52:08 PM PDT, Amit Kapila wrote: > >> >Won't the new code needs to ensure that ResetEvent(latchevent) > >should > >> >get > >> >called in case WaitForMultipleObjects()

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Mar 17, 2016 at 10:57 AM, Andres Freund wrote: > On 2016-03-17 09:01:36 -0400, Robert Haas wrote: >> 0001: Looking at this again, I'm no longer sure this is a bug. >> Doesn't your patch just check the same conditions in the opposite >> order? > > Yes, that's what's

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-18 14:00:58 -0400, Robert Haas wrote: > No, I mean it should be quite common for a particular fd to have no > events reported. If we're polling on 100 fds and 1 of them is active > and the other 99 are just sitting there, we want to skip over the > other 99 as quickly as possible.

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Mar 17, 2016 at 7:34 AM, Andres Freund wrote: > Hi, > > * I can do a blind rewrite of the windows implementation, but I'm > obviously not going to get that entirely right. So I need some help > from a windows person to test this. > I can help you verifying the

Re: [HACKERS] Performance degradation in commit ac1d794

On March 18, 2016 11:52:08 PM PDT, Amit Kapila wrote: >On Sat, Mar 19, 2016 at 12:14 PM, Andres Freund >wrote: >> >> >> >> On March 18, 2016 11:32:53 PM PDT, Amit Kapila > >wrote: >> >On Sat, Mar 19, 2016 at 12:00 AM, Andres

Re: [HACKERS] Performance degradation in commit ac1d794

On Wed, Mar 16, 2016 at 2:52 PM, Andres Freund wrote: > How about the following sketch of an API > > typedef struct LatchEvent > { > uint32 events; /* interesting events */ > uint32 revents; /* returned events */ > int fd; /* fd associated with event

Re: [HACKERS] Performance degradation in commit ac1d794

On Sat, Mar 19, 2016 at 12:14 PM, Andres Freund wrote: > > > > On March 18, 2016 11:32:53 PM PDT, Amit Kapila wrote: > >On Sat, Mar 19, 2016 at 12:00 AM, Andres Freund > >wrote: > >> > >> On 2016-03-18 20:14:07 +0530, Amit Kapila

Re: [HACKERS] Performance degradation in commit ac1d794

On March 18, 2016 11:32:53 PM PDT, Amit Kapila wrote: >On Sat, Mar 19, 2016 at 12:00 AM, Andres Freund >wrote: >> >> On 2016-03-18 20:14:07 +0530, Amit Kapila wrote: >> >> > I have done some >> > tests on Windows with 0003 patch which includes

Re: [HACKERS] Performance degradation in commit ac1d794

On Sat, Mar 19, 2016 at 12:00 AM, Andres Freund wrote: > > On 2016-03-18 20:14:07 +0530, Amit Kapila wrote: > > > I have done some > > tests on Windows with 0003 patch which includes running the regressions > > (vcregress check) and it passes. Will look into it tomorrow once

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Mar 17, 2016 at 11:17 AM, Andres Freund wrote: > Right now, do use a WaitEventSet you'd do something like > WaitEvent event; > > ModifyWaitEvent(FeBeWaitSet, 0, waitfor, NULL); > > WaitEventSetWait(FeBeWaitSet, 0 /* no

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-03-18 15:49:51 -0300, Alvaro Herrera wrote: > This seems a reasonable change, but I think that the use of WIN32 vs. > LATCH_USE_WIN32 is pretty confusing. In particular, LATCH_USE_WIN32 > isn't actually used for anything ... I suppose we don't care since this > is a temporary state of

[PoC] WaitLatchOrSocketMulti (Re: [HACKERS] Performance degradation in commit ac1d794)

Hello, I don't see how ac1d794 will be dealt, but I tried an example implement of multi-socket version of WaitLatchOrSocket using callbacks on top of the current master where ac1d794 has not been removed yet. At Thu, 14 Jan 2016 13:46:44 -0500, Robert Haas wrote in

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Jan 14, 2016 at 12:28 PM, Tom Lane wrote: > Andres Freund writes: >> This brings me to something related: I'm wondering if we shouldn't merge >> unix/win32_latch.c. > > Well, it's duplicated code on the one hand versus maze-of-ifdefs on the >

Re: [HACKERS] Performance degradation in commit ac1d794

Robert Haas writes: > I think we should either get this fixed RSN or revert the problematic > commit until we get it fixed. I'd be rather disappointed about the > latter because I think this was a very good thing on the merits, but > probably not good enough to justify

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Feb 11, 2016 at 12:48 PM, Tom Lane wrote: > Robert Haas writes: >> I think we should either get this fixed RSN or revert the problematic >> commit until we get it fixed. I'd be rather disappointed about the >> latter because I think this was a

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-02-11 12:50:58 -0500, Robert Haas wrote: > On Thu, Feb 11, 2016 at 12:48 PM, Tom Lane wrote: > > Robert Haas writes: > >> I think we should either get this fixed RSN or revert the problematic > >> commit until we get it fixed. I'd be rather

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Feb 11, 2016 at 12:53 PM, Andres Freund wrote: >> One problem is that it makes for misleading results if you try to >> benchmark 9.5 against 9.6. > > You need a really beefy box to show the problem. On a large/new 2 socket > machine the performance regression in in the

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-02-11 13:09:27 -0500, Robert Haas wrote: > On Thu, Feb 11, 2016 at 12:53 PM, Andres Freund wrote: > >> One problem is that it makes for misleading results if you try to > >> benchmark 9.5 against 9.6. > > > > You need a really beefy box to show the problem. On a

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Feb 11, 2016 at 1:19 PM, Andres Freund wrote: >> > Putting it on the open items list sounds good to me. >> >> Well, OK, I've done that then. I don't really agree that it's not a >> problem; the OP said he saw a 3x regression, and some of my colleagues >> doing

Re: [HACKERS] Performance degradation in commit ac1d794

Robert Haas writes: > On Thu, Feb 11, 2016 at 1:19 PM, Andres Freund wrote: >> Well, I can't do anything about that right now. I won't have the time to >> whip up the new/more complex API we discussed upthread in the next few >> days. So either we go

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Feb 11, 2016 at 1:41 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Feb 11, 2016 at 1:19 PM, Andres Freund wrote: >>> Well, I can't do anything about that right now. I won't have the time to >>> whip up the new/more

Re: [HACKERS] Performance degradation in commit ac1d794

Hello, I am one who wants waiting on many sockets at once. At Thu, 14 Jan 2016 18:55:51 +0100, Andres Freund wrote in <20160114175551.gm10...@awork2.anarazel.de> > On 2016-01-14 18:14:21 +0100, Andres Freund wrote: > > On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > > > >

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-01-14 10:39:55 -0500, Robert Haas wrote: > Doesn't this code make it possible for the self-pipe to fill up, > self-deadlocking the process? Suppose we repeatedly enter > WaitLatchOrSocket(). Each time we do, just after waiting = true is > set, somebody sets the latch. We handle the

Re: [HACKERS] Performance degradation in commit ac1d794

On January 14, 2016 5:16:59 PM GMT+01:00, Robert Haas wrote: >Yeah. Although I think for now it would be fine to just error out if >somebody tries to add a socket and there already is one. Then we >could lift that limitation in a later commit. Of course if Andres >wants

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Jan 14, 2016 at 12:06 PM, Andres Freund wrote: > On 2016-01-14 11:31:03 -0500, Robert Haas wrote: >> On Thu, Jan 14, 2016 at 10:56 AM, Andres Freund wrote: >> I think your idea of a data structure the encapsulates a set of events >> for which to

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-01-14 18:14:21 +0100, Andres Freund wrote: > I'm thinking of something like; > > int WaitOnLatchSet(LatchEventSet *set, int wakeEvents, long timeout); > > int > WaitLatchOrSocket(volatile Latch *latch, int wakeEvents, pgsocket sock,long > timeout) > { > LatchEventSet set; > >

Re: [HACKERS] Performance degradation in commit ac1d794

Andres Freund writes: > This brings me to something related: I'm wondering if we shouldn't merge > unix/win32_latch.c. Well, it's duplicated code on the one hand versus maze-of-ifdefs on the other. Feel free to try it and see, but I'm unsure it'd be an improvement.

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-01-14 18:14:21 +0100, Andres Freund wrote: > On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > > > Do we want to provide a backward compatible API for all this? I'm fine > > > either way. > > > > How would that work? > > I'm thinking of something like; > > int

Re: [HACKERS] Performance degradation in commit ac1d794

Robert Haas writes: > Incidentally, if we're going to whack around the latch API, it would > be nice to pick a design which wouldn't be too hard to extend to > waiting on multiple sockets. The application I have in mind is to > send of queries to several foreign servers at

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > > Do we want to provide a backward compatible API for all this? I'm fine > > either way. > > How would that work? I'm thinking of something like; int WaitOnLatchSet(LatchEventSet *set, int wakeEvents, long timeout); int

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Jan 14, 2016 at 10:46 AM, Tom Lane wrote: > Robert Haas writes: >> Incidentally, if we're going to whack around the latch API, it would >> be nice to pick a design which wouldn't be too hard to extend to >> waiting on multiple sockets. The

Re: [HACKERS] Performance degradation in commit ac1d794

On 2016-01-14 11:31:03 -0500, Robert Haas wrote: > On Thu, Jan 14, 2016 at 10:56 AM, Andres Freund wrote: > I think your idea of a data structure the encapsulates a set of events > for which to wait is probably a good one. WaitLatch doesn't seem like > a great name. Maybe

Re: [HACKERS] Performance degradation in commit ac1d794

Robert Haas writes: > On Thu, Jan 14, 2016 at 12:06 PM, Andres Freund wrote: >> Do we want to provide a backward compatible API for all this? I'm fine >> either way. > How would that work? I see no great need to be backwards-compatible on this,

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Jan 14, 2016 at 12:14 PM, Andres Freund wrote: > On 2016-01-14 12:07:23 -0500, Robert Haas wrote: >> > Do we want to provide a backward compatible API for all this? I'm fine >> > either way. >> >> How would that work? > > I'm thinking of something like; > > int

Re: [HACKERS] Performance degradation in commit ac1d794

On 2015-12-26 12:22:48 +0100, Andres Freund wrote: > On 2015-12-25 16:29:53 -0500, Tom Lane wrote: > > Andres Freund writes: > > > There's a couple solutions I can think of to that problem: > > > 1) Use epoll()/kqueue, or other similar interfaces that don't require > > >

Re: [HACKERS] Performance degradation in commit ac1d794

Andres Freund writes: > 0004 currently contains one debatable optimization, which I'd like to > discuss: Currently the 'sock' passed to WaitLatchOrSocket is not > removed/added to the epoll fd, if it's the numerically same as in the > last call. That's good for performance,

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Jan 14, 2016 at 9:39 AM, Andres Freund wrote: > I finally got back to working on this. Attached is a WIP patch series > implementing: > 0001: Allow to easily choose between the readiness primitives in unix_latch.c > Pretty helpful for testing, not useful for

Re: [HACKERS] Performance degradation in commit ac1d794

On Thu, Jan 14, 2016 at 10:56 AM, Andres Freund wrote: > So, I'm wondering how we'd exactly use a hyptothetical > SetSocketToWaitOn, or SetSocketsToWaitOn (or whatever). I mean it can > make a fair bit of sense to sometimes wait on MyLatch/port->sock and > sometimes on

Re: [HACKERS] Performance degradation in commit ac1d794

On Sat, Dec 26, 2015 at 5:41 PM, Andres Freund wrote: > On 2015-12-26 12:22:48 +0100, Andres Freund wrote: > > > > 3) Replace the postmaster_alive_fds socketpair by some other > signalling > > > >mechanism. E.g. sending a procsignal to each backend, which sets > the > > >

Re: [HACKERS] Performance degradation in commit ac1d794

Andres Freund writes: > A unportable and easy version of this, actually making sense this time, > would be to use prctl(PR_SET_PDEATHSIG, SIGQUIT). That'd send SIGQUIT to > backends whenever postmaster dies. Obviously that's not portable > either - doing this for linux only

Re: [HACKERS] Performance degradation in commit ac1d794

On 2015-12-25 16:29:53 -0500, Tom Lane wrote: > Andres Freund writes: > > There's a couple solutions I can think of to that problem: > > 1) Use epoll()/kqueue, or other similar interfaces that don't require > >re-registering fds at every invocation. My guess is that that'd

Re: [HACKERS] Performance degradation in commit ac1d794

On 2015-12-26 12:22:48 +0100, Andres Freund wrote: > > > 3) Replace the postmaster_alive_fds socketpair by some other signalling > > >mechanism. E.g. sending a procsignal to each backend, which sets the > > >latch and a special flag in the latch structure. > > > > And what would send the

Re: [HACKERS] Performance degradation in commit ac1d794

Andres Freund writes: > On December 25, 2015 7:10:23 PM GMT+01:00, Tom Lane > wrote: >> Seems like what you've got here is a kernel bug. > I wouldn't go as far as calling it a kernel bug. Were still doing 300k tps. > And were triggering the performance

Re: [HACKERS] Performance degradation in commit ac1d794

2015-12-25 20:18 GMT+03:00 Andres Freund : > On December 25, 2015 6:08:15 PM GMT+01:00, "Васильев Дмитрий" < > d.vasil...@postgrespro.ru> wrote: > >Hello hackers! > > > >I suddenly found commit ac1d794 gives up to 3 times performance > >degradation. > > > >I tried to run

Re: [HACKERS] Performance degradation in commit ac1d794

On December 25, 2015 6:27:06 PM GMT+01:00, "Васильев Дмитрий" >>​If so, could you provide a hierarchical before/after profile? > >​Performance | Git hash commit| Date >~ 360k tps | c3e7c24a1d60dc6ad56e2a0723399f1570c54224 | Thu Nov 12 >09:12:18 >2015 -0500 >~ 360k tps |

Re: [HACKERS] Performance degradation in commit ac1d794

=?UTF-8?B?0JLQsNGB0LjQu9GM0LXQsiDQlNC80LjRgtGA0LjQuQ==?= writes: > ​ Samples: 1M of event 'cycles', Event count (approx.): 816922259995, UID: > pgpro > Overhead Shared Object Symbol > 69,72% [kernel][k] _raw_spin_lock_irqsave > 1,43%

Re: [HACKERS] Performance degradation in commit ac1d794

​ 2015-12-25 20:44 GMT+03:00 Andres Freund : > On December 25, 2015 6:27:06 PM GMT+01:00, "Васильев Дмитрий" > >>​If so, could you provide a hierarchical before/after profile? > > > >​Performance | Git hash commit| Date > >~ 360k tps |

Re: [HACKERS] Performance degradation in commit ac1d794

2015-12-25 20:27 GMT+03:00 Васильев Дмитрий : > > 2015-12-25 20:18 GMT+03:00 Andres Freund : > >> On December 25, 2015 6:08:15 PM GMT+01:00, "Васильев Дмитрий" < >> d.vasil...@postgrespro.ru> wrote: >> >Hello hackers! >> > >> >I suddenly found commit

Re: [HACKERS] Performance degradation in commit ac1d794

On December 25, 2015 7:10:23 PM GMT+01:00, Tom Lane wrote: >=?UTF-8?B?0JLQsNGB0LjQu9GM0LXQsiDQlNC80LjRgtGA0LjQuQ==?= > writes: >> ��� Samples: 1M of event 'cycles', Event count (approx.): >816922259995, UID: >> pgpro >> Overhead Shared Object

Re: [HACKERS] Performance degradation in commit ac1d794

On December 25, 2015 6:08:15 PM GMT+01:00, "Васильев Дмитрий" wrote: >Hello hackers! > >I suddenly found commit ac1d794 gives up to 3 times performance >degradation. > >I tried to run pgbench -s 1000 -j 48 -c 48 -S -M prepared on 70 >CPU-core >machine: >commit ac1d794

Re: [HACKERS] Performance degradation in commit ac1d794

​ 2015-12-25 21:28 GMT+03:00 Tom Lane : > Andres Freund writes: > > On December 25, 2015 7:10:23 PM GMT+01:00, Tom Lane > wrote: > >> Seems like what you've got here is a kernel bug. > > > I wouldn't go as far as calling it a kernel

Re: [HACKERS] Performance degradation in commit ac1d794

Andres Freund writes: > There's a couple solutions I can think of to that problem: > 1) Use epoll()/kqueue, or other similar interfaces that don't require >re-registering fds at every invocation. My guess is that that'd be >desirable for performance anyway.

Re: [HACKERS] Performance degradation in commit ac1d794

​ 2015-12-25 22:42 GMT+03:00 Васильев Дмитрий : > > > ​ > 2015-12-25 21:28 GMT+03:00 Tom Lane : > >> Andres Freund writes: >> > On December 25, 2015 7:10:23 PM GMT+01:00, Tom Lane >> wrote: >> >> Seems like

Re: [HACKERS] Performance degradation in commit ac1d794

On 2015-12-25 13:28:55 -0500, Tom Lane wrote: > Hmm. And all those FDs point to the same pipe. I wonder if we're looking > at contention for some pipe-related data structure inside the kernel. Sounds fairly likely - and not too surprising. In this scenario we've a couple 100k