Re: [PATCHES] O_DIRECT for WAL writes

2005-06-04 Thread Bruce Momjian
I think the conclusion from the discussion is that O_DIRECT is in addition to the sync method, rather than in place of it, because O_DIRECT doesn't have the same media write guarantees as fsync(). Would you update the patch to do O_DIRECT in addition to O_SYNC or fsync() and see if there is a per

Re: [PATCHES] O_DIRECT for WAL writes

2005-06-03 Thread Bruno Wolff III
On Fri, Jun 03, 2005 at 09:43:13 -0700, Mary Edie Meredith <[EMAIL PROTECTED]> wrote: > > Looking at this from another angle, is there really any way that you can > say a write is truly guaranteed in the event of a failure? I think in > the end to be safe, you cannot. That's why (and I'm not t

Re: [PATCHES] O_DIRECT for WAL writes

2005-06-03 Thread Mary Edie Meredith
On Fri, 2005-06-03 at 10:37 +1000, Neil Conway wrote: > On Thu, 2005-06-02 at 11:49 -0700, Mary Edie Meredith wrote: > > My understanding is that O_DIRECT means "direct" as in "no buffering by > > the OS" which implies that if you write from your buffer, the write is > > not going to return unless

Re: [PATCHES] O_DIRECT for WAL writes

2005-06-02 Thread Neil Conway
On Thu, 2005-06-02 at 11:49 -0700, Mary Edie Meredith wrote: > My understanding is that O_DIRECT means "direct" as in "no buffering by > the OS" which implies that if you write from your buffer, the write is > not going to return unless the OS thinks the write is completed Right, I think that's d

Re: [PATCHES] O_DIRECT for WAL writes

2005-06-02 Thread Mary Edie Meredith
On Thu, 2005-06-02 at 11:39 +1000, Neil Conway wrote: > On Wed, 2005-06-01 at 17:08 -0700, Mary Edie Meredith wrote: > > I know I'm late to this discussion, and I haven't made it all the way > > through this thread to see if your questions on Linux writes were > > resolved. If you are still inter

Re: [PATCHES] O_DIRECT for WAL writes

2005-06-01 Thread Neil Conway
On Wed, 2005-06-01 at 17:08 -0700, Mary Edie Meredith wrote: > I know I'm late to this discussion, and I haven't made it all the way > through this thread to see if your questions on Linux writes were > resolved. If you are still interested, I recommend read a very good > one page description of

Re: [PATCHES] O_DIRECT for WAL writes

2005-06-01 Thread Mary Edie Meredith
On Mon, 2005-05-30 at 16:29 +1000, Neil Conway wrote: > On Mon, 2005-05-30 at 10:59 +0900, ITAGAKI Takahiro wrote: > > Yes, I've tested pgbench and dbt2 and their performances have improved. > > The two results are as follows: > > > > 1. pgbench -s 100 on one Pentium4, 1GB mem, 2 ATA disks, Linux

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-30 Thread Neil Conway
On Mon, 2005-05-30 at 11:24 -0400, Tom Lane wrote: > Wouldn't count on it :-(. One thing I'm particularly worried about is > buffer cache consistency: does the kernel guarantee to flush any buffers > it has that overlap the O_DIRECT write operation? At least on Linux I believe the kernel guarante

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-30 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > On Mon, 2005-05-30 at 02:52 -0400, Tom Lane wrote: > Well, that claims that "data is guaranteed to have been transferred", > but transferred to *where* is the question :) Oh, I see what you are worried about. I think you are right: what the doc promises i

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-30 Thread Neil Conway
On Mon, 2005-05-30 at 10:59 +0900, ITAGAKI Takahiro wrote: > Yes, I've tested pgbench and dbt2 and their performances have improved. > The two results are as follows: > > 1. pgbench -s 100 on one Pentium4, 1GB mem, 2 ATA disks, Linux 2.6.8 >(attached image) > tps | wal_sync_method > ---

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-30 Thread Ron Mayer
Tom Lane wrote: Neil Conway <[EMAIL PROTECTED]> writes: is opening a file with O_DIRECT sufficient to ensure that a write(2) does not return until the data has hit disk? Some googling suggests so, eg http://www.die.net/doc/linux/man/man2/open.2.html Really? On that page I read: "O_DIRECT..

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-30 Thread Neil Conway
On Mon, 2005-05-30 at 02:52 -0400, Tom Lane wrote: > Some googling suggests so, eg > http://www.die.net/doc/linux/man/man2/open.2.html Well, that claims that "data is guaranteed to have been transferred", but transferred to *where* is the question :) Transferring data to the disk's buffers and the

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-29 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > I wonder if we're benchmarking the right thing, though: is > opening a file with O_DIRECT sufficient to ensure that a write(2) does > not return until the data has hit disk? Some googling suggests so, eg http://www.die.net/doc/linux/man/man2/open.2.html T

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-29 Thread ITAGAKI Takahiro
Neil Conway <[EMAIL PROTECTED]> wrote: > > The patch adds a new choice "open_direct" to wal_sync_method. > Have you looked at what the performance difference of this option is? Yes, I've tested pgbench and dbt2 and their performances have improved. The two results are as follows: 1. pgbench -s

Re: [PATCHES] O_DIRECT for WAL writes

2005-05-26 Thread Neil Conway
ITAGAKI Takahiro wrote: The patch adds a new choice "open_direct" to wal_sync_method. It uses O_DIRECT flags for WAL writes, like O_SYNC. Have you looked at what the performance difference of this option is? For example, these benchmark results seem to indicate that an older version of the pa