Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

2015-01-26 Thread Andres Freund
On 2015-01-26 18:30:13 -0600, Jim Nasby wrote: On 12/23/14 11:41 AM, Andres Freund wrote: I think it'd generally be useful to have something like errhidecontext() akin to errhidestatement() to avoid things like the above. Under this proposal, do you want to suppress the

Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

2015-01-26 Thread Jim Nasby
On 12/23/14 11:41 AM, Andres Freund wrote: I think it'd generally be useful to have something like errhidecontext() akin to errhidestatement() to avoid things like the above. Under this proposal, do you want to suppress the context/statement unconditionally or via some hook/variable,

Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

2014-12-23 Thread Andres Freund
On 2014-12-22 10:35:35 +0530, Amit Kapila wrote: On Fri, Dec 19, 2014 at 9:36 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, When debugging lwlock issues I found PRINT_LWDEBUG/LOG_LWDEBUG rather painful to use because of the amount of elog contexts/statements emitted. Given the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.10

2014-12-23 Thread Andres Freund
Hi, Attached is a new version of the patchset which I intend to commit soon. Stuff changed since 0.9: * Greatly simplified locking logic - the whole concept that a lock could be spuriously acquired is gone. That cost a small bit of performance (0.5%, I thought it'd be much bigger) on x86,

Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

2014-12-21 Thread Amit Kapila
On Fri, Dec 19, 2014 at 9:36 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, When debugging lwlock issues I found PRINT_LWDEBUG/LOG_LWDEBUG rather painful to use because of the amount of elog contexts/statements emitted. Given the number of lwlock acquirations that's just not doable.

Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

2014-12-19 Thread Andres Freund
Hi, When debugging lwlock issues I found PRINT_LWDEBUG/LOG_LWDEBUG rather painful to use because of the amount of elog contexts/statements emitted. Given the number of lwlock acquirations that's just not doable. To solve that during development I've solved that by basically replacing: if

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-12-07 Thread Michael Paquier
On Wed, Dec 3, 2014 at 4:03 PM, Michael Paquier michael.paqu...@gmail.com wrote: Ping? This patch is in a stale state for a couple of weeks and still marked as waiting on author for this CF. Marked as returned with feedback. -- Michael -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-12-02 Thread Michael Paquier
On Tue, Nov 18, 2014 at 12:33 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Nov 17, 2014 at 10:31 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-17 10:21:04 -0500, Robert Haas wrote: Andres, where are we with this patch? 1. You're going to commit it, but haven't gotten

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-11-17 Thread Robert Haas
On Sat, Oct 25, 2014 at 1:50 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Fri, Oct 24, 2014 at 4:05 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-24 15:59:30 +0530, Amit Kapila wrote: and w.r.t performance it can lead extra function call, few checks and I think in

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-11-17 Thread Andres Freund
On 2014-11-17 10:21:04 -0500, Robert Haas wrote: Andres, where are we with this patch? 1. You're going to commit it, but haven't gotten around to it yet. 2. You're going to modify it some more and repost, but haven't gotten around to it yet. 3. You're willing to see it modified if

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-11-17 Thread Robert Haas
On Mon, Nov 17, 2014 at 10:31 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-17 10:21:04 -0500, Robert Haas wrote: Andres, where are we with this patch? 1. You're going to commit it, but haven't gotten around to it yet. 2. You're going to modify it some more and repost, but

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-30 Thread Andres Freund
On 2014-10-21 12:40:56 +0530, Amit Kapila wrote: While doing performance tests, I noticed a hang at higher client counts with patch. I have tried to check call stack for few of processes and it is as below: #0 0x008010933e54 in .semop () from /lib64/libc.so.6 #1 0x10286e48 in

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-30 Thread Amit Kapila
On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-21 12:40:56 +0530, Amit Kapila wrote: While doing performance tests, I noticed a hang at higher client counts with patch. I have tried to check call stack for few of processes and it is as below: #0

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-30 Thread Andres Freund
On 2014-10-30 18:54:57 +0530, Amit Kapila wrote: On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-21 12:40:56 +0530, Amit Kapila wrote: I have ran it for half an hour, but it doesn't came out even after ~2 hours. It doesn't get reproduced every

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-30 Thread Amit Kapila
On Thu, Oct 30, 2014 at 6:58 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-30 18:54:57 +0530, Amit Kapila wrote: On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. What commit did you apply the series ontop? I managed to reproduce a hang, but

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-30 Thread Andres Freund
On 2014-10-30 19:05:06 +0530, Amit Kapila wrote: On Thu, Oct 30, 2014 at 6:58 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-30 18:54:57 +0530, Amit Kapila wrote: On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. What commit did you apply

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-24 Thread Amit Kapila
On Wed, Oct 22, 2014 at 7:12 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-22 13:32:07 +0530, Amit Kapila wrote: On Tue, Oct 21, 2014 at 7:56 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund and...@2ndquadrant.com wrote:

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-24 Thread Amit Kapila
On Wed, Oct 22, 2014 at 8:04 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-21 19:56:05 +0530, Amit Kapila wrote: On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund and...@2ndquadrant.com wrote: spin_delay_count gives how much delay has happened to acquire spinlock which when

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-24 Thread Andres Freund
On 2014-10-24 15:59:30 +0530, Amit Kapila wrote: and w.r.t performance it can lead extra function call, few checks and I think in some cases even can acquire/release spinlock. I fail to see how that could be the case. Won't it happen incase first backend sets releaseOK to true and

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-22 Thread Amit Kapila
On Tue, Oct 21, 2014 at 7:56 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: Today, I have verified all previous comments raised by me and looked at new code and below

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-22 Thread Andres Freund
On 2014-10-22 13:32:07 +0530, Amit Kapila wrote: On Tue, Oct 21, 2014 at 7:56 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: Today, I have verified all

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-22 Thread Andres Freund
On 2014-10-21 19:56:05 +0530, Amit Kapila wrote: On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: 2. LWLockWakeup() { .. #ifdef LWLOCK_STATS lwstats-spin_delay_count += SpinLockAcquire(lock-mutex);

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-21 Thread Amit Kapila
On Fri, Oct 17, 2014 at 11:41 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-17 17:14:16 +0530, Amit Kapila wrote: On Tue, Oct 14, 2014 at 11:34 AM, Amit Kapila amit.kapil...@gmail.com wrote: HEAD – commit 494affb + wait free lw_shared_v2 Shared_buffers=8GB; Scale Factor =

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-21 Thread Amit Kapila
On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: 2. LWLockWakeup() { .. #ifdef LWLOCK_STATS lwstats-spin_delay_count += SpinLockAcquire(lock-mutex); #else SpinLockAcquire(lock-mutex); #endif .. }

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-17 Thread Amit Kapila
On Tue, Oct 14, 2014 at 11:34 AM, Amit Kapila amit.kapil...@gmail.com wrote: I am not sure why we are seeing difference even though running on same m/c with same configuration. I have tried many times, but I could not get the numbers you have posted above with HEAD, however now trying with

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-17 Thread Andres Freund
On 2014-10-17 17:14:16 +0530, Amit Kapila wrote: On Tue, Oct 14, 2014 at 11:34 AM, Amit Kapila amit.kapil...@gmail.com wrote: I am not sure why we are seeing difference even though running on same m/c with same configuration. I have tried many times, but I could not get the numbers

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-14 Thread Amit Kapila
On Sat, Oct 11, 2014 at 7:02 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Sat, Oct 11, 2014 at 6:40 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund and...@2ndquadrant.com And since

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-14 Thread Merlin Moncure
On Fri, Oct 10, 2014 at 11:00 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-10 16:41:39 +0200, Andres Freund wrote: FWIW, the profile always looks like - 48.61% postgres postgres [.] s_lock - s_lock + 96.67% StrategyGetBuffer + 1.19%

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-14 Thread Andres Freund
On 2014-10-14 08:40:49 -0500, Merlin Moncure wrote: On Fri, Oct 10, 2014 at 11:00 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-10 16:41:39 +0200, Andres Freund wrote: FWIW, the profile always looks like - 48.61% postgres postgres [.] s_lock - s_lock

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-14 Thread Merlin Moncure
On Tue, Oct 14, 2014 at 8:58 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-14 08:40:49 -0500, Merlin Moncure wrote: On Fri, Oct 10, 2014 at 11:00 AM, Andres Freund and...@2ndquadrant.com wrote: Which is nearly trivial now that atomics are in. Check out the attached WIP patch

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-14 Thread Amit Kapila
On Wed, Oct 15, 2014 at 12:06 AM, Merlin Moncure mmonc...@gmail.com wrote: A while back, I submitted a minor tweak to the clock sweep so that, instead of spinlocking every single buffer header as it swept it just did a single TAS as a kind of a trylock and punted to the next buffer if the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-11 Thread Andres Freund
Hi, On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund and...@2ndquadrant.com And since your general performance numbers are a fair bit lower than what I see with, hopefully, the same code on the same machine... You have reported numbers at

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-11 Thread Amit Kapila
On Sat, Oct 11, 2014 at 6:40 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund and...@2ndquadrant.com And since your general performance numbers are a fair bit lower than what I see with,

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-11 Thread Andres Freund
On 2014-10-11 15:10:45 +0200, Andres Freund wrote: Hi, On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund and...@2ndquadrant.com And since your general performance numbers are a fair bit lower than what I see with, hopefully, the same

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Andres Freund
Hi, On 2014-10-10 10:13:03 +0530, Amit Kapila wrote: I have done few performance tests for above patches and results of same is as below: Cool, thanks. Performance Data -- IBM POWER-7 16 cores, 64 hardware threads RAM = 64GB max_connections =210 Database

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-10 Thread Andres Freund
On 2014-10-08 20:07:35 -0400, Robert Haas wrote: On Wed, Oct 8, 2014 at 2:04 PM, Andres Freund and...@2ndquadrant.com wrote: So, what makes it work for me (among other unrelated stuff) seems to be the following in .gdbinit, defineing away some things that gdb doesn't handle: macro define

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Andres Freund
Hi Robert, On 2014-10-08 16:01:53 -0400, Robert Haas wrote: [ comment fixes ] Thanks, I've incorporated these + a bit more. Could you otherwise make sense of the explanation and the algorithm? +/* yipeyyahee */ Although this will be clear to individuals with a good command of

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Amit Kapila
On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-10 10:13:03 +0530, Amit Kapila wrote: I have done few performance tests for above patches and results of same is as below: Cool, thanks. Performance Data -- IBM POWER-7

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Andres Freund
On 2014-10-10 17:18:46 +0530, Amit Kapila wrote: On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund and...@2ndquadrant.com wrote: Observations -- a. The patch performs really well (increase upto ~40%) incase all the data fits in shared buffers (scale factor -100).

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Andres Freund
On 2014-10-10 16:41:39 +0200, Andres Freund wrote: FWIW, the profile always looks like - 48.61% postgres postgres [.] s_lock - s_lock + 96.67% StrategyGetBuffer + 1.19% UnpinBuffer + 0.90% PinBuffer + 0.70% hash_search_with_hash_value +

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Amit Kapila
On Fri, Oct 10, 2014 at 8:11 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-10 17:18:46 +0530, Amit Kapila wrote: On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund and...@2ndquadrant.com wrote: Observations -- a. The patch performs really well

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Andres Freund
On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: On Fri, Oct 10, 2014 at 8:11 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-10 17:18:46 +0530, Amit Kapila wrote: On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund and...@2ndquadrant.com wrote: Observations

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Amit Kapila
On Sat, Oct 11, 2014 at 6:29 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: I've run some short tests on hydra: scale 1000: base: 4GB: tps = 296273.004800 (including connections establishing) tps = 296373.978100 (excluding connections

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Andres Freund
On 2014-10-11 06:49:54 +0530, Amit Kapila wrote: On Sat, Oct 11, 2014 at 6:29 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: I've run some short tests on hydra: scale 1000: base: 4GB: tps = 296273.004800 (including connections

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-10 Thread Amit Kapila
On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-11 06:49:54 +0530, Amit Kapila wrote: On Sat, Oct 11, 2014 at 6:29 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: I've run some short tests on

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-09 Thread Jim Nasby
On 10/8/14, 8:35 AM, Andres Freund wrote: +#define EXCLUSIVE_LOCK (((uint32) 1) (31 - 1)) + +/* Must be greater than MAX_BACKENDS - which is 2^23-1, so we're fine. */ +#define SHARED_LOCK_MASK (~EXCLUSIVE_LOCK) There should at least be a comment where we define MAX_BACKENDS about the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-09 Thread Andres Freund
On 2014-10-09 16:52:46 -0500, Jim Nasby wrote: On 10/8/14, 8:35 AM, Andres Freund wrote: +#define EXCLUSIVE_LOCK (((uint32) 1) (31 - 1)) + +/* Must be greater than MAX_BACKENDS - which is 2^23-1, so we're fine. */ +#define SHARED_LOCK_MASK (~EXCLUSIVE_LOCK) There should at least be a

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-09 Thread Jim Nasby
On 10/9/14, 4:57 PM, Andres Freund wrote: If you modify either, you better grep for them... I don't think that's going to happen anyway. Requiring it during startup would mean exposing SHARED_LOCK_MASK outside of lwlock.c which'd be ugly. We could possibly stick a StaticAssert() someplace in

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-09 Thread Amit Kapila
On Wed, Oct 8, 2014 at 7:05 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, Attached you can find the next version of my LW_SHARED patchset. Now that atomics are committed, it seems like a good idea to also add their raison d'être. Since the last public version I have: * Addressed

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: 2. LWLockWakeup() { .. #ifdef LWLOCK_STATS lwstats-spin_delay_count += SpinLockAcquire(lock-mutex); #else SpinLockAcquire(lock-mutex); #endif .. } Earlier while releasing lock, we don't count it towards LWLock stats spin_delay_count.

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-10-08 14:47:44 +0200, Andres Freund wrote: On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: 5. LWLockWakeup() { .. dlist_foreach_modify(iter, (dlist_head *) wakeup) { PGPROC *waiter = dlist_container(PGPROC, lwWaitLink, iter.cur); LOG_LWDEBUG(LWLockRelease, l, mode,

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Andres Freund
Hi, Attached you can find the next version of my LW_SHARED patchset. Now that atomics are committed, it seems like a good idea to also add their raison d'être. Since the last public version I have: * Addressed lots of Amit's comments. Thanks! * Peformed a fair amount of testing. * Rebased the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund and...@2ndquadrant.com wrote: I don't see that as being relevant. The difference is an instruction or two - in the slow path we'll enter the kernel and sleep. This doesn't matter in comparison. And the code is *so* much more readable. I find the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-10-08 13:13:33 -0400, Robert Haas wrote: On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund and...@2ndquadrant.com wrote: I don't see that as being relevant. The difference is an instruction or two - in the slow path we'll enter the kernel and sleep. This doesn't matter in comparison.

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Alvaro Herrera
Robert Haas wrote: On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund and...@2ndquadrant.com wrote: I don't see that as being relevant. The difference is an instruction or two - in the slow path we'll enter the kernel and sleep. This doesn't matter in comparison. And the code is *so* much more

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Andres Freund
On 2014-10-08 14:23:44 -0300, Alvaro Herrera wrote: Robert Haas wrote: On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund and...@2ndquadrant.com wrote: I don't see that as being relevant. The difference is an instruction or two - in the slow path we'll enter the kernel and sleep. This

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Andres Freund
On 2014-10-08 15:23:22 -0400, Robert Haas wrote: On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund and...@2ndquadrant.com wrote: 1) Convert PGPROC-lwWaitLink into a dlist. The old code was frail and verbose. This also does: * changes the logic in LWLockRelease() to release all shared

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund and...@2ndquadrant.com wrote: 2) Implement the wait free LW_SHARED algorithm. + * too high for workloads/locks that were locked in shared mode very s/locked/taken/? + * frequently. Often we were spinning in the (obviously exlusive) spinlock,

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 2:04 PM, Andres Freund and...@2ndquadrant.com wrote: So, what makes it work for me (among other unrelated stuff) seems to be the following in .gdbinit, defineing away some things that gdb doesn't handle: macro define __builtin_offsetof(T, F) ((int) (((T *) 0)-F)) macro

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-07-01 Thread Andres Freund
Hi, Over at -performance Mark Kirkwood tested a recent version of this (http://archives.postgresql.org/message-id/53B283F3.7020005%40catalyst.net.nz) . I thought it's interesting to add the numbers to this thread: Test: pgbench Options: scale 500 read only Os: Ubuntu 14.04 Pg:

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-07-01 Thread Heikki Linnakangas
On 07/01/2014 01:08 PM, Andres Freund wrote: Hi, Over at -performance Mark Kirkwood tested a recent version of this (http://archives.postgresql.org/message-id/53B283F3.7020005%40catalyst.net.nz) . I thought it's interesting to add the numbers to this thread: Test: pgbench Options: scale 500

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-07-01 Thread Mark Kirkwood
On 01/07/14 23:25, Heikki Linnakangas wrote: On 07/01/2014 01:08 PM, Andres Freund wrote: Hi, Over at -performance Mark Kirkwood tested a recent version of this (http://archives.postgresql.org/message-id/53B283F3.7020005%40catalyst.net.nz) . I thought it's interesting to add the numbers to

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-25 Thread Amit Kapila
On Tue, Jun 24, 2014 at 9:33 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Jun 23, 2014 at 9:12 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: 7. LWLockWaitForVar() { .. /* * Add myself to wait queue. Note that this

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-23 Thread Amit Kapila
On Tue, Jun 17, 2014 at 8:56 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 20:47:51 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund and...@2ndquadrant.com wrote: You have followed it pretty well as far as I can understand from your replies, as

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-23 Thread Andres Freund
On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 8:56 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 20:47:51 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund and...@2ndquadrant.com wrote: You have followed it pretty

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-23 Thread Amit Kapila
On Mon, Jun 23, 2014 at 9:12 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 8:56 PM, Andres Freund and...@2ndquadrant.com wrote: 2. LWLockAcquireCommon() { .. if (!LWLockDequeueSelf(l)) { /* * Somebody

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-23 Thread Amit Kapila
On Mon, Jun 23, 2014 at 9:12 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: 12. #ifdef LWLOCK_DEBUG lock-owner = MyProc; #endif Shouldn't it be reset in LWLockRelease? That's actually intentional. It's quite useful to know the last

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-17 Thread Amit Kapila
On Fri, May 23, 2014 at 10:01 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund and...@2ndquadrant.com wrote: I've pushed a rebased version of the patchset to http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git branch rwlock

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-17 Thread Andres Freund
On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: On Fri, May 23, 2014 at 10:01 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund and...@2ndquadrant.com wrote: I've pushed a rebased version of the patchset to

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-17 Thread Amit Kapila
On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: On Fri, May 23, 2014 at 10:01 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund and...@2ndquadrant.com wrote:

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-17 Thread Andres Freund
On 2014-06-17 18:01:58 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund and...@2ndquadrant.com On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: 2. Handling of potentialy_spurious case seems to be pending in LWLock functions like LWLockAcquireCommon().

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-17 Thread Amit Kapila
On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 18:01:58 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund and...@2ndquadrant.com On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: 2. Handling of potentialy_spurious

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-06-17 Thread Andres Freund
On 2014-06-17 20:47:51 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-06-17 18:01:58 +0530, Amit Kapila wrote: On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund and...@2ndquadrant.com On 2014-06-17 12:41:26 +0530, Amit Kapila

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-05-23 Thread Amit Kapila
On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund and...@2ndquadrant.com wrote: I've pushed a rebased version of the patchset to http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git branch rwlock contention. 220b34331f77effdb46798ddd7cca0cffc1b2858 actually was the small problem,

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-10 Thread Heikki Linnakangas
On 01/31/2014 11:54 AM, Andres Freund wrote: Hi, On 2014-01-28 21:27:29 -0800, Peter Geoghegan wrote: On Fri, Nov 15, 2013 at 11:47 AM, Andres Freund and...@2ndquadrant.com wrote: 1) I've added an abstracted atomic ops implementation. Needs a fair amount of work, also submitted as a

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-04 Thread Andres Freund
On 2014-02-03 17:51:20 -0800, Peter Geoghegan wrote: On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-02-01 19:47:29 -0800, Peter Geoghegan wrote: Here are the results of a benchmark on Nathan Boley's 64-core, 4 socket server:

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Christian Kruse
Hi, I'm doing some benchmarks regarding this problem: one set with baseline and one set with your patch. Machine was a 32 core machine (4 CPUs with 8 cores), 252 gib RAM. Both versions have the type align patch applied. pgbench-tools config: SCALES=100 SETCLIENTS=1 4 8 16 32 48 64 96 128

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Peter Geoghegan
On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse christ...@2ndquadrant.com wrote: I'm doing some benchmarks regarding this problem: one set with baseline and one set with your patch. Machine was a 32 core machine (4 CPUs with 8 cores), 252 gib RAM. Both versions have the type align patch

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Peter Geoghegan
On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse christ...@2ndquadrant.com wrote: I added -M prepared to the pgbench call in the benchwarmer script. The read-only tests are finished, I come to similiar results as yours: http://wwwtech.de/pg/benchmarks-lwlock-read-only/ Note that Christian

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Andres Freund
On 2014-02-04 11:48:14 -0800, Peter Geoghegan wrote: On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse christ...@2ndquadrant.com wrote: I added -M prepared to the pgbench call in the benchwarmer script. The read-only tests are finished, I come to similiar results as yours:

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Peter Geoghegan
On Tue, Feb 4, 2014 at 11:50 AM, Andres Freund and...@2ndquadrant.com wrote: I think he has applied the patch to hack around the alignment issue I pushed to git for both branches. It's not nice enough to be applied yet, but it should fix the issue. I think the 201 is just a remembrance of

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Peter Geoghegan
On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse christ...@2ndquadrant.com wrote: I'm doing some benchmarks regarding this problem: one set with baseline and one set with your patch. Machine was a 32 core machine (4 CPUs with 8 cores), 252 gib RAM. What CPU model? Can you post /proc/cpuinfo?

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Andres Freund
On February 4, 2014 8:53:36 PM CET, Peter Geoghegan p...@heroku.com wrote: On Tue, Feb 4, 2014 at 11:50 AM, Andres Freund and...@2ndquadrant.com wrote: I think he has applied the patch to hack around the alignment issue I pushed to git for both branches. It's not nice enough to be applied yet,

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Christian Kruse
Hi, On 04/02/14 12:02, Peter Geoghegan wrote: On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse christ...@2ndquadrant.com wrote: I'm doing some benchmarks regarding this problem: one set with baseline and one set with your patch. Machine was a 32 core machine (4 CPUs with 8 cores), 252 gib

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Christian Kruse
Hi, On 04/02/14 21:03, Andres Freund wrote: Christian, could you rerun with master (the commit on which the branch is based on), the alignment patch, and then the lwlock patch? Best with max_connections 200. That's probably more important than the write tests as a first step.. Ok, benchmark

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Peter Geoghegan
On Tue, Feb 4, 2014 at 12:30 PM, Christian Kruse christ...@2ndquadrant.com wrote: Ok, benchmark for baseline+alignment patch is running. I see that you have enabled latency information. For this kind of thing I prefer to hack pgbench-tools to not collect this (i.e. to not pass the -l flag,

Re: [HACKERS] Wait free LW_SHARED acquisition

2014-02-04 Thread Andres Freund
On 2014-02-04 13:42:51 -0800, Peter Geoghegan wrote: On Tue, Feb 4, 2014 at 12:30 PM, Christian Kruse christ...@2ndquadrant.com wrote: Ok, benchmark for baseline+alignment patch is running. I see that you have enabled latency information. For this kind of thing I prefer to hack

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-03 Thread Jeff Janes
On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund and...@2ndquadrant.comwrote: Some background: The setups that triggered me into working on the patchset didn't really have a pgbench like workload, the individual queries were/are more complicated even though it's still an high throughput OLTP

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-03 Thread Peter Geoghegan
On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund and...@2ndquadrant.com wrote: The changed algorithm for lwlock imo is an *algorithmic* improvement, not one for a particular architecture. The advantage being that locking a lwlock which is primarily taken in shared mode will never need need to

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-03 Thread Peter Geoghegan
On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-02-01 19:47:29 -0800, Peter Geoghegan wrote: Here are the results of a benchmark on Nathan Boley's 64-core, 4 socket server: http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/amd-4-socket-rwlocks/

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-02 Thread Andres Freund
Hi, On 2014-02-01 19:47:29 -0800, Peter Geoghegan wrote: Here are the results of a benchmark on Nathan Boley's 64-core, 4 socket server: http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/amd-4-socket-rwlocks/ That's interesting. The maximum number of what you see here (~293125)

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Peter Geoghegan
I thought I'd try out what I was in an immediate position to do without having access to dedicated multi-socket hardware: A benchmark on AWS. This was a c3.8xlarge instance, which are reportedly backed by Intel Xeon E5-2680 processors. Since the Intel ARK website reports that these CPUs have 16

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Andres Freund
On 2014-01-31 17:52:58 -0800, Peter Geoghegan wrote: On Fri, Jan 31, 2014 at 1:54 AM, Andres Freund and...@2ndquadrant.com wrote: I plan to split the atomics patch into smaller chunks before reposting. Imo the Convert the PGPROC-lwWaitLink list into a dlist instead of open coding it. is

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Peter Geoghegan
On Sat, Feb 1, 2014 at 4:57 AM, Andres Freund and...@2ndquadrant.com wrote: I'm looking at alternative options, because this is not terribly helpful. With those big caveats in mind, consider the results of the benchmark, which show the patch performing somewhat worse than the master baseline

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Andres Freund
On 2014-02-01 13:40:20 -0800, Peter Geoghegan wrote: On Sat, Feb 1, 2014 at 4:57 AM, Andres Freund and...@2ndquadrant.com wrote: I'm looking at alternative options, because this is not terribly helpful. With those big caveats in mind, consider the results of the benchmark, which show the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Peter Geoghegan
On Sat, Feb 1, 2014 at 1:41 PM, Andres Freund and...@2ndquadrant.com wrote: However, I tested the most recent revision from your git remote on the AWS instance. But that was before my fix, right. Except you managed to timetravel :) Heh, okay. So Nathan Boley has generously made available a

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-01-31 Thread Andres Freund
Hi, On 2014-01-28 21:27:29 -0800, Peter Geoghegan wrote: On Fri, Nov 15, 2013 at 11:47 AM, Andres Freund and...@2ndquadrant.com wrote: 1) I've added an abstracted atomic ops implementation. Needs a fair amount of work, also submitted as a separate CF entry. (Patch 1 2) Commit

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-01-31 Thread Peter Geoghegan
On Fri, Jan 31, 2014 at 1:54 AM, Andres Freund and...@2ndquadrant.com wrote: I plan to split the atomics patch into smaller chunks before reposting. Imo the Convert the PGPROC-lwWaitLink list into a dlist instead of open coding it. is worth being applied independently from the rest of the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-01-28 Thread Peter Geoghegan
On Fri, Nov 15, 2013 at 11:47 AM, Andres Freund and...@2ndquadrant.com wrote: 1) I've added an abstracted atomic ops implementation. Needs a fair amount of work, also submitted as a separate CF entry. (Patch 1 2) Commit 220b34331f77effdb46798ddd7cca0cffc1b2858 caused bitrot when applying

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2013-12-21 Thread Peter Eisentraut
This patch didn't make it out of the 2013-11 commit fest. You should move it to the next commit fest (probably with an updated patch) before January 15th. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

  1   2   >