Re: [HACKERS] WALWriteLock contention

2015-05-18 Thread Amit Kapila
On Mon, May 18, 2015 at 1:53 AM, Robert Haas wrote: > > On May 17, 2015, at 11:04 AM, Amit Kapila wrote: > > On Sun, May 17, 2015 at 7:45 AM, Robert Haas wrote: > > > > I wonder if we could write WAL to two different files in > > alternation, so that we could be writing to one file which fsync-i

Re: [HACKERS] WALWriteLock contention

2015-05-18 Thread Jeff Janes
> > > > > My goal there was to further improve group commit. When running pgbench > > -j10 -c10, it was common to see fsyncs that alternated between flushing 1 > > transaction, and 9 transactions. Because the first one to the gate would > go > > through it and slam it on all the others, and it wou

Re: [HACKERS] WALWriteLock contention

2015-05-17 Thread Robert Haas
On May 17, 2015, at 5:57 PM, Thomas Munro wrote: >> On Sun, May 17, 2015 at 2:15 PM, Robert Haas wrote: >> http://oldblog.antirez.com/post/fsync-different-thread-useless.html >> >> It suggests that an fsync in progress blocks out not only other >> fsyncs, but other writes to the same file, which

Re: [HACKERS] WALWriteLock contention

2015-05-17 Thread Thomas Munro
On Sun, May 17, 2015 at 2:15 PM, Robert Haas wrote: > http://oldblog.antirez.com/post/fsync-different-thread-useless.html > > It suggests that an fsync in progress blocks out not only other > fsyncs, but other writes to the same file, which for our purposes is > just awful. More Googling around r

Re: [HACKERS] WALWriteLock contention

2015-05-17 Thread Robert Haas
On May 17, 2015, at 11:04 AM, Amit Kapila wrote: > On Sun, May 17, 2015 at 7:45 AM, Robert Haas wrote: > > > > I wonder if we could write WAL to two different files in > > alternation, so that we could be writing to one file which fsync-ing > > the other. > > Won't the order of transactions repl

Re: [HACKERS] WALWriteLock contention

2015-05-17 Thread Amit Kapila
On Sun, May 17, 2015 at 7:45 AM, Robert Haas wrote: > > I wonder if we could write WAL to two different files in > alternation, so that we could be writing to one file which fsync-ing > the other. > Won't the order of transactions replay during recovery can cause problems if we do alternation whi

Re: [HACKERS] WALWriteLock contention

2015-05-16 Thread Robert Haas
On Fri, May 15, 2015 at 9:15 PM, Jeff Janes wrote: > I implemented this 2-3 years ago, just dropping the WALWriteLock immediately > before the fsync and then picking it up again immediately after, and was > surprised that I saw absolutely no improvement. Of course it surely depends > on the IO st

Re: [HACKERS] WALWriteLock contention

2015-05-15 Thread Jeff Janes
On Fri, May 15, 2015 at 9:06 AM, Robert Haas wrote: > WALWriteLock contention is measurable on some workloads. In studying > the problem briefly, a couple of questions emerged: > > ... > > 2. I don't really understand why WALWriteLock is set up to prohibit > two backends from flushing WAL at t

Re: [HACKERS] WALWriteLock contention

2015-05-15 Thread Robert Haas
On Fri, May 15, 2015 at 1:09 PM, Tom Lane wrote: > Robert Haas writes: >> WALWriteLock contention is measurable on some workloads. In studying >> the problem briefly, a couple of questions emerged: > >> 1. Doesn't it suck to rewrite an entire 8kB block every time, instead >> of only the new byte

Re: [HACKERS] WALWriteLock contention

2015-05-15 Thread Tom Lane
Robert Haas writes: > WALWriteLock contention is measurable on some workloads. In studying > the problem briefly, a couple of questions emerged: > 1. Doesn't it suck to rewrite an entire 8kB block every time, instead > of only the new bytes (and maybe a few bytes following that to spoil > any ol

Re: [HACKERS] WALWriteLock contention

2015-05-15 Thread Joshua D. Drake
On 05/15/2015 09:06 AM, Robert Haas wrote: 2. I don't really understand why WALWriteLock is set up to prohibit two backends from flushing WAL at the same time. That seems unnecessary. Suppose we've got two backends that flush WAL one after the other. Assume (as is not unlikely) that the seco

[HACKERS] WALWriteLock contention

2015-05-15 Thread Robert Haas
WALWriteLock contention is measurable on some workloads. In studying the problem briefly, a couple of questions emerged: 1. Doesn't it suck to rewrite an entire 8kB block every time, instead of only the new bytes (and maybe a few bytes following that to spoil any old data that might be there)? I