Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-11-08 Thread Tom Lane
Peter Eisentraut writes: > On 10/7/17 16:46, Tom Lane wrote: >> Accordingly I propose the attached patch. If anyone's interested in >> experimenting on other platforms, we might be able to refine/complicate >> the FLUSH_DISTANCE selection further, but I think this is probably good >> enough for a

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-11-08 Thread Andres Freund
On November 8, 2017 7:31:17 AM PST, Peter Eisentraut wrote: >On 10/7/17 16:46, Tom Lane wrote: >> I wrote: >>> Current status is that I've filed a bug report with Apple and am >waiting >>> to see their response before deciding what to do next. If they fix >the >>> issue promptly then there's l

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-11-08 Thread Peter Eisentraut
On 10/7/17 16:46, Tom Lane wrote: > I wrote: >> Current status is that I've filed a bug report with Apple and am waiting >> to see their response before deciding what to do next. If they fix the >> issue promptly then there's little need for us to do anything. > Accordingly I propose the attached

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-07 Thread Tom Lane
I wrote: > Current status is that I've filed a bug report with Apple and am waiting > to see their response before deciding what to do next. If they fix the > issue promptly then there's little need for us to do anything. Not having heard a peep from Apple yet, I decided to do a bit more experime

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-04 Thread Tom Lane
Brent Dearth writes: > Is there an issue tracker I could be looking at to follow along on the > progress on this issue? This email thread is pretty much it ... Current status is that I've filed a bug report with Apple and am waiting to see their response before deciding what to do next. If they

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-04 Thread Brent Dearth
Tom, Andres - Is there an issue tracker I could be looking at to follow along on the progress on this issue? Thanks so much! On Mon, Oct 2, 2017 at 9:06 PM, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 19:50:51 -0400, Tom Lane wrote: > >> What I saw was that the backend process w

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 19:50:51 -0400, Tom Lane wrote: >> What I saw was that the backend process was consuming 100% of (one) CPU, >> while the I/O transaction rate viewed by "iostat 1" started pretty low >> --- under 10% of what the machine is capable of --- and dropped from >> the

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 19:50:51 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 18:33:17 -0400, Tom Lane wrote: > >> I'm kind of surprised that machine B doesn't show obvious tanking in this > >> test given that msync() makes it suck so badly at copying a database. > >> I wonder what is di

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 18:33:17 -0400, Tom Lane wrote: >> I'm kind of surprised that machine B doesn't show obvious tanking in this >> test given that msync() makes it suck so badly at copying a database. >> I wonder what is different from the kernel's standpoint ... maybe the >> sh

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
Hi, On 2017-10-02 18:33:17 -0400, Tom Lane wrote: > Andres Freund writes: > > To demonstrate what I'm observing here, on linux with a fairly fast ssd: > > ... > > I tried to replicate this test as closely as I could on the Mac hardware > I have laying about. Thanks! > I only bothered with the

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > To demonstrate what I'm observing here, on linux with a fairly fast ssd: > ... I tried to replicate this test as closely as I could on the Mac hardware I have laying about. I only bothered with the synchronous_commit=off case, though, since you say that shows the worst ef

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:59:05 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 15:54:43 -0400, Tom Lane wrote: > >> Should I expect there to be any difference at all? We don't enable > >> *_flush_after by default on non-Linux platforms. > > > Right, you'd have to enable that. But your p

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 15:54:43 -0400, Tom Lane wrote: >> Should I expect there to be any difference at all? We don't enable >> *_flush_after by default on non-Linux platforms. > Right, you'd have to enable that. But your patch would neuter an > intentionally enabled config too, n

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:54:43 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 15:42:25 -0400, Tom Lane wrote: > >> I experimented with this further by seeing whether the msync() code path > >> is of any value on Sierra either. The answer seems to be "no": cloning > >> a scale-1000 pgben

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 15:42:25 -0400, Tom Lane wrote: >> I experimented with this further by seeing whether the msync() code path >> is of any value on Sierra either. The answer seems to be "no": cloning >> a scale-1000 pgbench database takes about 17-18 seconds on my Sierra >> la

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:42:25 -0400, Tom Lane wrote: > I wrote: > > In short, therefore, APFS cannot cope with the way we're using msync(). > > I experimented with this further by seeing whether the msync() code path > is of any value on Sierra either. The answer seems to be "no": cloning > a scale-1000

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Brent Dearth
Thanks for this breakdown Tom! FWIW - I'm on Postgres 9.6.5 as bundled with Postgres.app (2.0.5) running on 2013 MBP (2.7GHz i7 / 16GB / SSD) setup. It looks like this might be a priority for an upcoming release, so I might try to hold out for downstream, but thanks for the patch. It will help if

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
I wrote: > In short, therefore, APFS cannot cope with the way we're using msync(). I experimented with this further by seeing whether the msync() code path is of any value on Sierra either. The answer seems to be "no": cloning a scale-1000 pgbench database takes about 17-18 seconds on my Sierra l

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Brent Dearth writes: > I just recently "upgraded" to High Sierra and experiencing horrendous CREATE > DATABASE performance. Creating a database from a 3G template DB used to > take ~1m but post-upgrade is taking ~22m at a sustained write of around > 4MB/s. Occasionally, attempting to create an emp