On Thu, Mar 15, 2012 at 5:52 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
When all those changes are put together, the patched version now beats or
matches the current code in the RAM drive tests, except that the
single-client case is still about 10% slower. I added the new
On 21.03.2012 13:14, Fujii Masao wrote:
PANIC: space reserved for WAL record does not match what was
written, CurrPos: C/0, EndPos: B/FF00
So I think that the patch would have a bug which handles WAL boundary wrongly.
Thanks for the testing! These WAL boundary issues are really
On Wed, Mar 21, 2012 at 7:52 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
So, although none of the issues alone is a show-stopper, but considering all
these things together, I'm starting to feel that this needs to be pushed to
9.3. Thoughts?
I think I agree. I like the
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
... although none of the issues alone is a show-stopper, but considering
all these things together, I'm starting to feel that this needs to be
pushed to 9.3. Thoughts?
Agreed. In particular, I think you are right that it'd be
On Wed, Mar 21, 2012 at 10:59 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
... although none of the issues alone is a show-stopper, but considering
all these things together, I'm starting to feel that this needs to be
pushed to 9.3.
On Fri, Mar 9, 2012 at 2:45 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Thanks!
BTW, I haven't forgotten about the recovery bugs Jeff found earlier. I'm
planning to do a longer run with his test script - I only run it for about
1000 iterations - to see if I can
On 09.03.2012 12:04, Heikki Linnakangas wrote:
I've been doing some performance testing with this, using a simple C
function that just inserts a dummy WAL record of given size. I'm not
totally satisfied. Although the patch helps with scalability at 3-4
concurrent backends doing WAL insertions,
On Tue, Mar 13, 2012 at 1:59 AM, Jeff Janes jeff.ja...@gmail.com wrote:
On Fri, Mar 9, 2012 at 2:45 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Thanks!
BTW, I haven't forgotten about the recovery bugs Jeff found earlier. I'm
planning to do a longer run with his test
On 07.03.2012 17:28, Tom Lane wrote:
Simon Riggssi...@2ndquadrant.com writes:
On Wed, Mar 7, 2012 at 3:04 PM, Tom Lanet...@sss.pgh.pa.us wrote:
Alvaro Herreraalvhe...@commandprompt.com writes:
So they are undoubtely rare. Not sure if as rare as Higgs bosons.
Even if they're rare,
On Fri, Mar 9, 2012 at 7:04 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Here's an updated patch. It now only loops once per segment that a record
crosses. Plus a lot of other small cleanup.
Thanks! But you forgot to attach the patch.
I've been doing some performance
On Tue, Mar 6, 2012 at 8:32 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 06.03.2012 17:12, Tom Lane wrote:
How long is the current locked code exactly --- does it contain a loop?
Perhaps best if you take a look for yourself, the
Excerpts from Simon Riggs's message of mié mar 07 05:35:44 -0300 2012:
On Tue, Mar 6, 2012 at 8:32 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 06.03.2012 17:12, Tom Lane wrote:
How long is the current locked code exactly --- does
Alvaro Herrera alvhe...@commandprompt.com writes:
Just to keep things in perspective -- For a commit record to reach one
megabyte, it would have to be a transaction that drops over 43k tables.
Or have 64k smgr inval messages (for example, a TRUNCATE might send half
a dozen of these messages).
On Wed, Mar 7, 2012 at 3:04 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Alvaro Herrera alvhe...@commandprompt.com writes:
Just to keep things in perspective -- For a commit record to reach one
megabyte, it would have to be a transaction that drops over 43k tables.
Or have 64k smgr inval messages
Simon Riggs si...@2ndquadrant.com writes:
On Wed, Mar 7, 2012 at 3:04 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Alvaro Herrera alvhe...@commandprompt.com writes:
So they are undoubtely rare. Not sure if as rare as Higgs bosons.
Even if they're rare, having a major performance hiccup when one
On Mon, Mar 5, 2012 at 8:50 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
That particular issue would be very hard to hit in practice, so I don't know
if this could explain the recovery failures that Jeff saw. I got the test
script running (thanks for that Jeff!), but
On Tue, Mar 6, 2012 at 2:17 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 21.02.2012 13:19, Fujii Masao wrote:
In some places, the spinlock insertpos_lck is taken while another
spinlock info_lck is being held. Is this OK? What if
On Tue, Mar 6, 2012 at 1:50 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
+ * An xlog-switch record consumes all the remaining space on the
+ * WAL segment. We have already reserved it for us, but we still
need
+ * to make sure it's been allocated and
On 06.03.2012 14:52, Fujii Masao wrote:
On Tue, Mar 6, 2012 at 2:17 AM, Tom Lanet...@sss.pgh.pa.us wrote:
Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes:
On 21.02.2012 13:19, Fujii Masao wrote:
In some places, the spinlock insertpos_lck is taken while another
spinlock info_lck
On Tue, Mar 6, 2012 at 10:07 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I admit that block is longer than any of our existing spinlock blocks.
However, it's important for performance. I tried using a lwlock earlier, and
that negated the gains. So if that's a serious
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 06.03.2012 14:52, Fujii Masao wrote:
This also strikes me that the usage of the spinlock insertpos_lck might
not be OK in ReserveXLogInsertLocation() because a few dozen instructions
can be performed while holding the
On 06.03.2012 17:12, Tom Lane wrote:
Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes:
On 06.03.2012 14:52, Fujii Masao wrote:
This also strikes me that the usage of the spinlock insertpos_lck might
not be OK in ReserveXLogInsertLocation() because a few dozen instructions
can be
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 06.03.2012 17:12, Tom Lane wrote:
How long is the current locked code exactly --- does it contain a loop?
Perhaps best if you take a look for yourself, the function is called
ReserveXLogInsertLocation() in patch. It calls a
On Wed, Mar 7, 2012 at 5:32 AM, Tom Lane t...@sss.pgh.pa.us wrote:
What I suggest is that it should not be necessary to crawl forward one
page at a time to figure out how many pages will be needed to store N
bytes worth of WAL data. You're basically implementing a division
problem as repeated
On 20.02.2012 08:00, Amit Kapila wrote:
I was trying to understand this patch and had few doubts:
1. In PerformXLogInsert(), why there is need to check freespace when
already during ReserveXLogInsertLocation(), the space is reserved. Is
it possible that the record size is more than actually
On 21.02.2012 13:19, Fujii Masao wrote:
On Sat, Feb 18, 2012 at 12:36 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Attached is a new version, fixing that, and off-by-one bug you pointed out
in the slot wraparound handling. I also moved code around a bit, I think
this new
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 21.02.2012 13:19, Fujii Masao wrote:
In some places, the spinlock insertpos_lck is taken while another
spinlock info_lck is being held. Is this OK? What if unfortunately
inner spinlock takes long to be taken?
Hmm, that's only
On Tue, Feb 21, 2012 at 5:34 PM, Fujii Masao masao.fu...@gmail.com wrote:
On Tue, Feb 21, 2012 at 8:19 PM, Fujii Masao masao.fu...@gmail.com wrote:
On Sat, Feb 18, 2012 at 12:36 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Attached is a new version, fixing that, and
On Sat, Feb 18, 2012 at 12:36 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Attached is a new version, fixing that, and off-by-one bug you pointed out
in the slot wraparound handling. I also moved code around a bit, I think
this new division of labor between the XLogInsert
On Tue, Feb 21, 2012 at 8:19 PM, Fujii Masao masao.fu...@gmail.com wrote:
On Sat, Feb 18, 2012 at 12:36 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Attached is a new version, fixing that, and off-by-one bug you pointed out
in the slot wraparound handling. I also moved
On Sun, Feb 19, 2012 at 3:01 AM, Jeff Janes jeff.ja...@gmail.com wrote:
I've tested your v9 patch. I no longer see any inconsistencies or
lost transactions in the recovered database. But occasionally I get
databases that fail to recover at all.
It has always been with the exact same failed
Janes; Robert Haas; PostgreSQL-development
Subject: Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work
outside WALInsertLock)
On 17.02.2012 07:27, Fujii Masao wrote:
Got another problem: when I ran pg_stop_backup to take an online
backup, it got stuck until I had generated new WAL
On Fri, Feb 17, 2012 at 7:36 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 17.02.2012 07:27, Fujii Masao wrote:
Got another problem: when I ran pg_stop_backup to take an online backup,
it got stuck until I had generated new WAL record. This happens because,
in the
On 16.02.2012 13:31, Fujii Masao wrote:
On Thu, Feb 16, 2012 at 6:15 PM, Fujii Masaomasao.fu...@gmail.com wrote:
BTW, when I ran the test on my Ubuntu, I could not reproduce the problem.
I could reproduce the problem only in MacOS.
+ nextslot = Insert-nextslot;
+ if
On Thu, Feb 16, 2012 at 5:02 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 15.02.2012 18:52, Fujii Masao wrote:
On Thu, Feb 16, 2012 at 1:01 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Are you still seeing this failure with the latest patch I
On Thu, Feb 16, 2012 at 6:15 PM, Fujii Masao masao.fu...@gmail.com wrote:
On Thu, Feb 16, 2012 at 5:02 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 15.02.2012 18:52, Fujii Masao wrote:
On Thu, Feb 16, 2012 at 1:01 AM, Heikki Linnakangas
On Mon, Feb 13, 2012 at 8:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 13.02.2012 01:04, Jeff Janes wrote:
Attached is my quick and dirty attempt to set XLP_FIRST_IS_CONTRECORD.
I have no idea if I did it correctly, in particular if calling
GetXLogBuffer(CurrPos)
On 13.02.2012 19:13, Fujii Masao wrote:
On Mon, Feb 13, 2012 at 8:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 13.02.2012 01:04, Jeff Janes wrote:
Attached is my quick and dirty attempt to set XLP_FIRST_IS_CONTRECORD.
I have no idea if I did it correctly, in
On Thu, Feb 16, 2012 at 1:01 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 13.02.2012 19:13, Fujii Masao wrote:
On Mon, Feb 13, 2012 at 8:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 13.02.2012 01:04, Jeff Janes wrote:
Attached is my
On 15.02.2012 18:52, Fujii Masao wrote:
On Thu, Feb 16, 2012 at 1:01 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Are you still seeing this failure with the latest patch I posted
(http://archives.postgresql.org/message-id/4f38f5e5.8050...@enterprisedb.com)?
Yes. Just to
On Mon, Feb 13, 2012 at 8:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 13.02.2012 01:04, Jeff Janes wrote:
Attached is my quick and dirty attempt to set XLP_FIRST_IS_CONTRECORD.
I have no idea if I did it correctly, in particular if calling
GetXLogBuffer(CurrPos)
On Thu, Feb 9, 2012 at 3:02 AM, Fujii Masao masao.fu...@gmail.com wrote:
On Thu, Feb 9, 2012 at 7:25 PM, Fujii Masao masao.fu...@gmail.com wrote:
On Thu, Feb 9, 2012 at 3:32 AM, Jeff Janes jeff.ja...@gmail.com wrote:
After applying this patch and then forcing crashes, upon recovery the
On Thu, Feb 9, 2012 at 3:32 AM, Jeff Janes jeff.ja...@gmail.com wrote:
On Wed, Feb 1, 2012 at 11:46 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 31.01.2012 17:35, Fujii Masao wrote:
On Fri, Jan 20, 2012 at 11:11 PM, Heikki Linnakangas
On Thu, Feb 9, 2012 at 7:25 PM, Fujii Masao masao.fu...@gmail.com wrote:
On Thu, Feb 9, 2012 at 3:32 AM, Jeff Janes jeff.ja...@gmail.com wrote:
On Wed, Feb 1, 2012 at 11:46 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 31.01.2012 17:35, Fujii Masao wrote:
On Fri, Jan
On Wed, Feb 1, 2012 at 11:46 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 31.01.2012 17:35, Fujii Masao wrote:
On Fri, Jan 20, 2012 at 11:11 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 20.01.2012 15:32, Robert Haas wrote:
On Sat, Jan 14,
On Fri, Jan 20, 2012 at 11:11 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 20.01.2012 15:32, Robert Haas wrote:
On Sat, Jan 14, 2012 at 9:32 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Here's another version of the patch to make XLogInsert less
On Sat, Jan 14, 2012 at 9:32 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Here's another version of the patch to make XLogInsert less of a bottleneck
on multi-CPU systems. The basic idea is the same as before, but several bugs
have been fixed, and lots of misc. clean up has
On Fri, Jan 20, 2012 at 2:11 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 20.01.2012 15:32, Robert Haas wrote:
On Sat, Jan 14, 2012 at 9:32 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Here's another version of the patch to make XLogInsert less
On Sat, Jan 7, 2012 at 9:31 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Anyway, here's a new version of the patch. It no longer busy-waits for
in-progress insertions to finish, and handles xlog-switches. This is now
feature-complete. It's a pretty complicated patch, so I
On 09.01.2012 15:44, Simon Riggs wrote:
On Sat, Jan 7, 2012 at 9:31 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Anyway, here's a new version of the patch. It no longer busy-waits for
in-progress insertions to finish, and handles xlog-switches. This is now
On Mon, Jan 9, 2012 at 2:29 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Can we also try aligning the actual insertions onto cache lines rather
than just MAXALIGNing them? The WAL header fills half a cache line as
it is, so many other records will fit nicely. I'd like to
On Sun, Dec 25, 2011 at 7:48 PM, Robert Haas robertmh...@gmail.com wrote:
m01 tps = 631.875547 (including connections establishing)
x01 tps = 611.443724 (including connections establishing)
m08 tps = 4573.701237 (including connections establishing)
x08 tps = 4576.242333 (including connections
On Fri, Dec 23, 2011 at 2:54 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Sorry. Last minute changes, didn't retest properly.. Here's another attempt.
I tried this one out on Nate Boley's system. Looks pretty good.
m = master, x = with xloginsert-scale-2 patch.
On Sat, Dec 24, 2011 at 4:54 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Sorry. Last minute changes, didn't retest properly.. Here's another attempt.
When I tested the patch, initdb failed:
$ initdb -D data
initializing dependencies ... PANIC: could not locate a
On Fri, Dec 16, 2011 at 3:27 AM, Tom Lane t...@sss.pgh.pa.us wrote:
On its own that sounds dangerous, but its not. When we need to confirm
the prev link we already know what we expect it to be, so CRC-ing it
is overkill. That isn't true of any other part of the WAL record, so
the prev link is
On 16.12.2011 15:42, Heikki Linnakangas wrote:
On 16.12.2011 15:03, Simon Riggs wrote:
On Fri, Dec 16, 2011 at 12:50 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 16.12.2011 14:37, Simon Riggs wrote:
I already proposed a design for that using page-level share locks any
On Fri, Dec 23, 2011 at 3:15 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 23.12.2011 10:13, Heikki Linnakangas wrote:
So, here's a WIP patch of what I've been working on.
And here's the patch I forgot to attach..
Fails regression tests for me. I found this in
On 16.12.2011 05:27, Tom Lane wrote:
* We write a WAL record that starts 8 bytes before a sector boundary,
so that the prev_link is in one sector and the rest of the record in
the next one(s).
prev-link is not the first field in the header. The CRC is.
* Time passes, and we recycle that WAL
On Fri, Dec 16, 2011 at 12:07 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Anyway, I'm looking at ways to make the memcpy() of the payload happen
without the lock, in parallel, and once you do that the record header CRC
calculation can be done in parallel, too. That makes
On 16.12.2011 14:37, Simon Riggs wrote:
On Fri, Dec 16, 2011 at 12:07 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Anyway, I'm looking at ways to make the memcpy() of the payload happen
without the lock, in parallel, and once you do that the record header CRC
calculation
On Fri, Dec 16, 2011 at 12:50 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 16.12.2011 14:37, Simon Riggs wrote:
On Fri, Dec 16, 2011 at 12:07 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Anyway, I'm looking at ways to make the memcpy() of the
On 16.12.2011 15:03, Simon Riggs wrote:
On Fri, Dec 16, 2011 at 12:50 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 16.12.2011 14:37, Simon Riggs wrote:
I already proposed a design for that using page-level share locks any
reason not to go with that?
Sorry, I must've
On Thursday, December 15, 2011 02:51:33 PM Heikki Linnakangas wrote:
I've been looking at various ways to make WALInsertLock less of a
bottleneck on multi-CPU servers. The key is going to be to separate the
two things that are done while holding the WALInsertLock: a) allocating
the required
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
I've been experimenting with different approaches to do that, but one
thing is common among all of them: you need to know the total amount of
WAL space needed for the record, including backup blocks, before you
take the lock.
On Thu, Dec 15, 2011 at 7:34 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
I've been experimenting with different approaches to do that, but one
thing is common among all of them: you need to know the total amount of
WAL space needed for
Jeff Janes jeff.ja...@gmail.com writes:
On Thu, Dec 15, 2011 at 7:34 AM, Tom Lane t...@sss.pgh.pa.us wrote:
This patch may or may not be useful, but this description of it is utter
nonsense, because we already do compute that before taking the lock.
Please try again to explain what you're
On 15.12.2011 18:48, Tom Lane wrote:
Jeff Janesjeff.ja...@gmail.com writes:
On Thu, Dec 15, 2011 at 7:34 AM, Tom Lanet...@sss.pgh.pa.us wrote:
This patch may or may not be useful, but this description of it is utter
nonsense, because we already do compute that before taking the lock.
Please
On 15.12.2011 17:34, Tom Lane wrote:
Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes:
I've been experimenting with different approaches to do that, but one
thing is common among all of them: you need to know the total amount of
WAL space needed for the record, including backup
On Thu, Dec 15, 2011 at 7:06 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Please try again to explain what you're doing?
Ok: I'm moving the creation of rdata entries for backup blocks outside the
critical section, so that it's done before grabbing the lock. I'm also
On Thu, Dec 15, 2011 at 6:50 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
unless you are proposing to remove
the prev_link from the scope of the CRC, which is not exactly a
penalty-free change.
We could CRC the rest of the record header before getting the lock, though,
Simon Riggs si...@2ndquadrant.com writes:
You missed your cue to discuss leaving the prev link out of the CRC
altogether.
On its own that sounds dangerous, but its not. When we need to confirm
the prev link we already know what we expect it to be, so CRC-ing it
is overkill. That isn't true
71 matches
Mail list logo
