Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-01-19 Thread Andres Freund
Hi Greg, On Tuesday 19 January 2010 15:52:25 Greg Stark wrote: On Mon, Jan 18, 2010 at 4:35 PM, Greg Stark gsst...@mit.edu wrote: Looking at this patch for the commitfest I have a few questions. So I've touched this patch up a bit: 1) moved the posix_fadvise call to a new fd.c function

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-29 Thread Greg Stark
On Tue, Dec 29, 2009 at 2:05 AM, Andres Freund and...@anarazel.de wrote:  Reads Completed:        2,        8KiB  Writes Completed:     2362,     29672KiB New:  Reads Completed:        0,        0KiB  Writes Completed:      550,     5960KiB It looks like the new method is only doing 1/6th

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-29 Thread Andres Freund
On Tuesday 29 December 2009 11:48:10 Greg Stark wrote: On Tue, Dec 29, 2009 at 2:05 AM, Andres Freund and...@anarazel.de wrote: Reads Completed:2,8KiB Writes Completed: 2362, 29672KiB New: Reads Completed:0,0KiB Writes Completed: 550,

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-29 Thread Andres Freund
On Monday 28 December 2009 23:59:43 Andres Freund wrote: On Monday 28 December 2009 23:54:51 Andres Freund wrote: On Saturday 12 December 2009 21:38:41 Andres Freund wrote: On Saturday 12 December 2009 21:36:27 Michael Clemmons wrote: If ppl think its worth it I'll create a ticket

[HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Saturday 12 December 2009 21:38:41 Andres Freund wrote: On Saturday 12 December 2009 21:36:27 Michael Clemmons wrote: If ppl think its worth it I'll create a ticket Thanks, no need. I will post a patch tomorrow or so. Well. It was a long day... Anyway. In this patch I delay the fsync done

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Monday 28 December 2009 23:54:51 Andres Freund wrote: On Saturday 12 December 2009 21:38:41 Andres Freund wrote: On Saturday 12 December 2009 21:36:27 Michael Clemmons wrote: If ppl think its worth it I'll create a ticket Thanks, no need. I will post a patch tomorrow or so. Well.

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Tom Lane
Andres Freund and...@anarazel.de writes: This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s on my laptop. Still slower than with fsync off (~0.25) but quite a worthy improvement. I can't help wondering whether that's real or some kind of platform-specific artifact. I

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Tuesday 29 December 2009 00:06:28 Tom Lane wrote: Andres Freund and...@anarazel.de writes: This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s on my laptop. Still slower than with fsync off (~0.25) but quite a worthy improvement. I can't help wondering whether

Re: [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Tuesday 29 December 2009 00:06:28 Tom Lane wrote: Andres Freund and...@anarazel.de writes: This speeds up CREATE DATABASE from ~9 seconds to something around 0.8s on my laptop. Still slower than with fsync off (~0.25) but quite a worthy improvement. I can't help wondering whether

Re: [PERFORM] [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Greg Smith
Andres Freund wrote: As I said the real benefit only occurred after adding posix_fadvise(.., FADV_DONTNEED) which is somewhat plausible, because i.e. the directory entries don't need to get scheduled for every file and because the kernel can reorder a whole directory nearly sequentially.

Re: [PERFORM] [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Tuesday 29 December 2009 01:46:21 Greg Smith wrote: Andres Freund wrote: As I said the real benefit only occurred after adding posix_fadvise(.., FADV_DONTNEED) which is somewhat plausible, because i.e. the directory entries don't need to get scheduled for every file and because the

Re: [PERFORM] [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Michael Clemmons
Andres, Great job. Looking through the emails and thinking about why this works I think this patch should significantly speedup 8.4 on most any file system(obviously some more than others) unless the system has significantly reduced memory or a slow single core. On a Celeron with 256 memory I

Re: [PERFORM] [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Tuesday 29 December 2009 03:53:12 Michael Clemmons wrote: Andres, Great job. Looking through the emails and thinking about why this works I think this patch should significantly speedup 8.4 on most any file system(obviously some more than others) unless the system has significantly

Re: [PERFORM] [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Michael Clemmons
Maybe not crash out but in this situation. N=0 while(N=0): CREATE DATABASE new_db_N; Since the fsync is the part which takes the memory and time but is happening in the background want the fsyncs pile up in the background faster than can be run filling up the memory and stack. This is very

Re: [PERFORM] [HACKERS] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2009-12-28 Thread Andres Freund
On Tuesday 29 December 2009 04:04:06 Michael Clemmons wrote: Maybe not crash out but in this situation. N=0 while(N=0): CREATE DATABASE new_db_N; Since the fsync is the part which takes the memory and time but is happening in the background want the fsyncs pile up in the background