Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-17 Thread Amit Kapila
On Tue, Mar 17, 2020 at 6:32 PM Dilip Kumar wrote: > > Your changes look fine to me. I have also verified all the test and > everything works fine. > I have pushed the first patch. I will push the others in coming days. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-17 Thread Dilip Kumar
On Tue, Mar 17, 2020 at 5:14 PM Amit Kapila wrote: > > On Mon, Mar 16, 2020 at 3:24 PM Dilip Kumar wrote: > > > > + > + /* > + * Indicate that the lock is released for certain types of locks > + */ > +#ifdef USE_ASSERT_CHECKING > + CheckAndSetLockHeld(locallock, false); > +#endif > } > > /* >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-17 Thread Amit Kapila
On Mon, Mar 16, 2020 at 3:24 PM Dilip Kumar wrote: > + + /* + * Indicate that the lock is released for certain types of locks + */ +#ifdef USE_ASSERT_CHECKING + CheckAndSetLockHeld(locallock, false); +#endif } /* @@ -1618,6 +1666,11 @@ GrantLockLocal(LOCALLOCK *locallock, ResourceOwner owner)

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-16 Thread Dilip Kumar
On Mon, Mar 16, 2020 at 11:56 AM Kuntal Ghosh wrote: > > On Mon, Mar 16, 2020 at 9:43 AM Dilip Kumar wrote: > > On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada > > wrote: > > > IsRelationExtensionLockHeld and IsPageLockHeld are used only when > > > assertion is enabled. So how about making

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-16 Thread Kuntal Ghosh
On Mon, Mar 16, 2020 at 9:43 AM Dilip Kumar wrote: > On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada > wrote: > > IsRelationExtensionLockHeld and IsPageLockHeld are used only when > > assertion is enabled. So how about making CheckAndSetLockHeld work > > only if USE_ASSERT_CHECKING to avoid

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Mon, Mar 16, 2020 at 8:15 AM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 9:17 PM Dilip Kumar wrote: > > > > On Sun, Mar 15, 2020 at 5:58 PM Amit Kapila wrote: > > > > > > > > > 1. Group members wait for page locks. If you test that the leader > > > acquires the page lock and then member

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada wrote: > > On Mon, 16 Mar 2020 at 00:54, Dilip Kumar wrote: > > > > On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > > > > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > > > > > I have modified 0001 and 0002 slightly,

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Masahiko Sawada
On Mon, 16 Mar 2020 at 00:54, Dilip Kumar wrote: > > On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > > > I have modified 0001 and 0002 slightly, Basically, instead of two > > > function CheckAndSetLockHeld and

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Amit Kapila
On Sun, Mar 15, 2020 at 9:17 PM Dilip Kumar wrote: > > On Sun, Mar 15, 2020 at 5:58 PM Amit Kapila wrote: > > > > > > 1. Group members wait for page locks. If you test that the leader > > acquires the page lock and then member also tries to acquire the same > > lock on the same index, it

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila wrote: > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > > > I have modified 0001 and 0002 slightly, Basically, instead of two > > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created > > a one function. > > > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Amit Kapila
On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar wrote: > > I have modified 0001 and 0002 slightly, Basically, instead of two > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created > a one function. > +CheckAndSetLockHeld(LOCALLOCK *locallock, bool value) Can we rename the

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Amit Kapila
On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar wrote: > > On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > > > Apart from that, I have also extended the solution for the page lock. > > > And, I have also broken down the 3rd patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar wrote: > > On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > > > Apart from that, I have also extended the solution for the page lock. > > > And, I have also broken down the 3rd patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-15 Thread Dilip Kumar
On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila wrote: > > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > > > Apart from that, I have also extended the solution for the page lock. > > And, I have also broken down the 3rd patch in two parts for relation > > extension and for the page lock. >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-14 Thread Amit Kapila
On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar wrote: > > Apart from that, I have also extended the solution for the page lock. > And, I have also broken down the 3rd patch in two parts for relation > extension and for the page lock. > Thanks, I have made a number of cosmetic changes and written

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 3:39 PM Amit Kapila wrote: > > On Fri, Mar 13, 2020 at 8:37 AM Dilip Kumar wrote: > > > > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > > > > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar > > > wrote: > > > > > > > > I have fixed this in the attached patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 11:16 AM Dilip Kumar wrote: > > On Fri, Mar 13, 2020 at 11:08 AM Amit Kapila wrote: > > > > On Thu, Mar 12, 2020 at 3:04 PM Dilip Kumar wrote: > > > > > > On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila > > > wrote: > > > > > > > > > > > > If we have no other choice, then

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 2:32 PM Kuntal Ghosh wrote: > > On Fri, Mar 13, 2020 at 8:29 AM Amit Kapila wrote: > > > > On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh > > wrote: > > > I think moving them inside a macro is a good idea. Also, I think we > > > should move all the Assert related code

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Amit Kapila
On Fri, Mar 13, 2020 at 8:37 AM Dilip Kumar wrote: > > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar wrote: > > > > > > I have fixed this in the attached patch set. > > > > > > > I have modified your > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Kuntal Ghosh
On Fri, Mar 13, 2020 at 8:42 AM Dilip Kumar wrote: > > On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh > > wrote: > > > > > + /* > > > + * The relation extension or page lock can never participate in actual > > > + * deadlock cycle. See Asserts in LockAcquireExtended. So, there is > > > + * no

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-13 Thread Kuntal Ghosh
On Fri, Mar 13, 2020 at 8:29 AM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh > wrote: > > I think moving them inside a macro is a good idea. Also, I think we > > should move all the Assert related code inside some debugging macro > > similar to this: > > #ifdef LOCK_DEBUG

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 11:08 AM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 3:04 PM Dilip Kumar wrote: > > > > On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila wrote: > > > > > > > > > If we have no other choice, then I see a few downsides of adding a > > > special check in the LockRelease()

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Amit Kapila
On Thu, Mar 12, 2020 at 3:04 PM Dilip Kumar wrote: > > On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila wrote: > > > > > > If we have no other choice, then I see a few downsides of adding a > > special check in the LockRelease() call: > > > > 1. Instead of resetting/decrement the variable from

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Fri, Mar 13, 2020 at 8:29 AM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh > wrote: > > > > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > > > > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar > > > wrote: > > > > > > > > I have fixed this in the attached

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar wrote: > > > > I have fixed this in the attached patch set. > > > > I have modified your > v4-0003-Conflict-Extension-Page-lock-in-group-member patch. The > modifications are (a) Change

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Amit Kapila
On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh wrote: > > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar wrote: > > > > > > I have fixed this in the attached patch set. > > > > > > > I have modified your > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Kuntal Ghosh
On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila wrote: > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar wrote: > > > > I have fixed this in the attached patch set. > > > > I have modified your > v4-0003-Conflict-Extension-Page-lock-in-group-member patch. The > modifications are (a) Change

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Amit Kapila
On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar wrote: > > I have fixed this in the attached patch set. > I have modified your v4-0003-Conflict-Extension-Page-lock-in-group-member patch. The modifications are (a) Change src/backend/storage/lmgr/README to reflect new behaviour, (b) Introduce a new

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-12 Thread Dilip Kumar
On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila wrote: > > On Tue, Mar 10, 2020 at 6:39 PM Robert Haas wrote: > > > > On Fri, Mar 6, 2020 at 11:27 PM Dilip Kumar wrote: > > > I think instead of the flag we need to keep the counter because we can > > > acquire the same relation extension lock

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-11 Thread Dilip Kumar
On Wed, Mar 11, 2020 at 2:23 PM Amit Kapila wrote: > > On Tue, Mar 10, 2020 at 8:53 AM Dilip Kumar wrote: > > > > Please find the updated patch (summary of the changes) > > - Instead of searching the lock hash table for assert, it maintains a > > counter. > > - Also, handled the case where we

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-11 Thread Dilip Kumar
On Tue, Mar 10, 2020 at 4:11 PM Amit Kapila wrote: > > On Mon, Feb 24, 2020 at 3:38 PM Amit Kapila wrote: > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > What I'm advocating is that extension locks should continue to go > > > through lock.c. And yes, that requires some

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-11 Thread Dilip Kumar
On Wed, Mar 11, 2020 at 2:23 PM Amit Kapila wrote: > > On Tue, Mar 10, 2020 at 8:53 AM Dilip Kumar wrote: > > > > Please find the updated patch (summary of the changes) > > - Instead of searching the lock hash table for assert, it maintains a > > counter. > > - Also, handled the case where we

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-11 Thread Amit Kapila
On Tue, Mar 10, 2020 at 6:39 PM Robert Haas wrote: > > On Fri, Mar 6, 2020 at 11:27 PM Dilip Kumar wrote: > > I think instead of the flag we need to keep the counter because we can > > acquire the same relation extension lock multiple times. So > > basically, every time we acquire the lock we

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-11 Thread Amit Kapila
On Tue, Mar 10, 2020 at 8:53 AM Dilip Kumar wrote: > > Please find the updated patch (summary of the changes) > - Instead of searching the lock hash table for assert, it maintains a counter. > - Also, handled the case where we can acquire the relation extension > lock while holding the relation

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-10 Thread Amit Kapila
On Tue, Mar 10, 2020 at 6:48 PM Robert Haas wrote: > > On Sat, Mar 7, 2020 at 10:23 AM Tom Lane wrote: > > I continue to think that we'd be better off getting all of this > > out of the heavyweight lock manager. There is no reason why we > > should need deadlock detection, or multiple holds of

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-10 Thread Robert Haas
On Sat, Mar 7, 2020 at 10:23 AM Tom Lane wrote: > I continue to think that we'd be better off getting all of this > out of the heavyweight lock manager. There is no reason why we > should need deadlock detection, or multiple holds of the same > lock, or pretty much anything that LWLocks don't

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-10 Thread Robert Haas
On Fri, Mar 6, 2020 at 11:27 PM Dilip Kumar wrote: > I think instead of the flag we need to keep the counter because we can > acquire the same relation extension lock multiple times. So > basically, every time we acquire the lock we can increment the counter > and while releasing we can

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-10 Thread Amit Kapila
On Mon, Feb 24, 2020 at 3:38 PM Amit Kapila wrote: > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > What I'm advocating is that extension locks should continue to go > > through lock.c. And yes, that requires some changes to group locking, > > but I still don't see why they'd be

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-09 Thread Dilip Kumar
On Mon, Mar 9, 2020 at 2:36 PM Amit Kapila wrote: > > On Mon, Mar 9, 2020 at 2:09 PM Masahiko Sawada > wrote: >> >> On Mon, 9 Mar 2020 at 15:50, Amit Kapila wrote: >> > >> > On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada >> > wrote: >> >> >> >> On Mon, 9 Mar 2020 at 14:16, Amit Kapila

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-09 Thread Amit Kapila
On Mon, Mar 9, 2020 at 2:09 PM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > On Mon, 9 Mar 2020 at 15:50, Amit Kapila wrote: > > > > On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada < > masahiko.saw...@2ndquadrant.com> wrote: > >> > >> On Mon, 9 Mar 2020 at 14:16, Amit Kapila >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-09 Thread Masahiko Sawada
On Mon, 9 Mar 2020 at 15:50, Amit Kapila wrote: > > On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada > wrote: >> >> On Mon, 9 Mar 2020 at 14:16, Amit Kapila wrote: >> > >> > On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada >> > wrote: >> >> > >> >> > Fair position, as per initial analysis, I

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-09 Thread Amit Kapila
On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > On Mon, 9 Mar 2020 at 14:16, Amit Kapila wrote: > > > > On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada < > masahiko.saw...@2ndquadrant.com> wrote: > >> > > >> > Fair position, as per initial analysis, I

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-09 Thread Masahiko Sawada
On Mon, 9 Mar 2020 at 14:16, Amit Kapila wrote: > > On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada > wrote: >> >> On Mon, 24 Feb 2020 at 19:08, Amit Kapila wrote: >> > >> > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: >> > > >> > > Hi, >> > > >> > > On 2020-02-19 11:12:18 +0530, Amit

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-08 Thread Amit Kapila
On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > On Mon, 24 Feb 2020 at 19:08, Amit Kapila wrote: > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund > wrote: > > > > > > Hi, > > > > > > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > > > > I think

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Masahiko Sawada
On Mon, 24 Feb 2020 at 19:08, Amit Kapila wrote: > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > Hi, > > > > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > > > I think till we know the real need for changing group locking, going > > > in the direction of what Tom suggested

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Dilip Kumar
On Sat, Mar 7, 2020 at 8:53 PM Tom Lane wrote: > > Dilip Kumar writes: > > I think instead of the flag we need to keep the counter because we can > > acquire the same relation extension lock multiple times. > > Uh ... what? How would that not be broken usage on its face? Basically, if we can

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Tom Lane
Dilip Kumar writes: > I think instead of the flag we need to keep the counter because we can > acquire the same relation extension lock multiple times. Uh ... what? How would that not be broken usage on its face? I continue to think that we'd be better off getting all of this out of the

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Dilip Kumar
On Sat, Mar 7, 2020 at 3:26 PM Amit Kapila wrote: > > On Sat, Mar 7, 2020 at 11:14 AM Amit Kapila wrote: >> >> On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar wrote: >>> >>> On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila wrote: >>> > >>> > On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar wrote: >>> > > >>>

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-07 Thread Amit Kapila
On Sat, Mar 7, 2020 at 11:14 AM Amit Kapila wrote: > On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar wrote: > >> On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila >> wrote: >> > >> > On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar >> wrote: >> > > >> > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila >> wrote:

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-06 Thread Amit Kapila
On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar wrote: > On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila > wrote: > > > > On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar > wrote: > > > > > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila > wrote: > > > > > > > > > > > > 5. I have also tried to think of another

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-06 Thread Dilip Kumar
On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila wrote: > > On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar wrote: > > > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila wrote: > > > > > > > > > 5. I have also tried to think of another way to check if we already > > > hold lock type LOCKTAG_RELATION_EXTEND,

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-06 Thread Robert Haas
On Thu, Mar 5, 2020 at 11:45 PM Amit Kapila wrote: > I think we can keep such a flag in TopTransactionState. We free such > locks after the work is done (except during error where we free them > at transaction abort) rather than at transaction commit, so one might > say it is better not to

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-05 Thread Amit Kapila
On Fri, Mar 6, 2020 at 2:19 AM Robert Haas wrote: > > On Thu, Mar 5, 2020 at 2:18 PM Mahendra Singh Thalor > wrote: > > Here, attaching new patch set for review. > > I was kind of assuming that the way this would work is that it would > set a flag or increment a counter or something when we

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-05 Thread Amit Kapila
On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar wrote: > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila wrote: > > > > > > 5. I have also tried to think of another way to check if we already > > hold lock type LOCKTAG_RELATION_EXTEND, but couldn't come up with a > > cheaper way than this. Basically,

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-05 Thread Robert Haas
On Thu, Mar 5, 2020 at 2:18 PM Mahendra Singh Thalor wrote: > Here, attaching new patch set for review. I was kind of assuming that the way this would work is that it would set a flag or increment a counter or something when we acquire a relation extension lock, and then reverse the process when

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-05 Thread Mahendra Singh Thalor
On Wed, 4 Mar 2020 at 12:03, Dilip Kumar wrote: > > On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor > wrote: > > > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > > > > > Hi, > > > > > > > > On 2020-02-19

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-05 Thread Mahendra Singh Thalor
On Thu, 5 Mar 2020 at 13:54, Dilip Kumar wrote: > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila wrote: > > > > On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor > > wrote: > > > > > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > > > > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-05 Thread Dilip Kumar
On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila wrote: > > On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor > wrote: > > > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > > > > > What I'm advocating is that extension

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-04 Thread Amit Kapila
On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor wrote: > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > > > What I'm advocating is that extension locks should continue to go > > > through lock.c. And yes, that

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-04 Thread Mahendra Singh Thalor
On Wed, 4 Mar 2020 at 12:03, Dilip Kumar wrote: > > On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor > wrote: > > > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > > > > > Hi, > > > > > > > > On 2020-02-19

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-03 Thread Dilip Kumar
On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor wrote: > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > > > > I think till we know the real need

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-03-03 Thread Mahendra Singh Thalor
On Mon, 24 Feb 2020 at 15:39, Amit Kapila wrote: > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > > > Hi, > > > > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > > > I think till we know the real need for changing group locking, going > > > in the direction of what Tom suggested

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-24 Thread Amit Kapila
On Thu, Feb 20, 2020 at 8:06 AM Andres Freund wrote: > > Hi, > > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > > I think till we know the real need for changing group locking, going > > in the direction of what Tom suggested to use an array of LWLocks [1] > > to address the problems in hand

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-19 Thread Andres Freund
Hi, On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > I think till we know the real need for changing group locking, going > in the direction of what Tom suggested to use an array of LWLocks [1] > to address the problems in hand is a good idea. -many I think that building yet another locking

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-18 Thread Amit Kapila
On Mon, Feb 17, 2020 at 2:42 AM Tom Lane wrote: > > Andres Freund writes: > > On 2020-02-14 13:34:03 -0500, Robert Haas wrote: > >> I think the group locking + deadlock detection things are more > >> fundamental than you might be crediting, but I agree that having > >> parallel mechanisms has

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-16 Thread Tom Lane
Andres Freund writes: > On 2020-02-14 13:34:03 -0500, Robert Haas wrote: >> I think the group locking + deadlock detection things are more >> fundamental than you might be crediting, but I agree that having >> parallel mechanisms has its own set of pitfalls. > It's possible. But I'm also

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-16 Thread Andres Freund
Hi, On 2020-02-14 13:34:03 -0500, Robert Haas wrote: > On Fri, Feb 14, 2020 at 1:07 PM Andres Freund wrote: > > Yea, that seems possible. I'm not really sure it's needed however? As > > long as you're not teaching the locking mechanism new tricks that > > influence the wait graph, why would the

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Amit Kapila
On Fri, Feb 14, 2020 at 9:13 PM Tom Lane wrote: > > Robert Haas writes: > > On Wed, Feb 12, 2020 at 11:53 AM Tom Lane wrote: > > > That's an interesting idea, but it doesn't make the lock acquisition > > itself interruptible, which seems pretty important to me in this case. > > Good point: if

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Amit Kapila
On Fri, Feb 14, 2020 at 8:12 PM Tom Lane wrote: > > Amit Kapila writes: > > I think MaxBackends will generally limit the number of different > > relations that can simultaneously extend, but maybe tables with many > > partitions might change the situation. You are right that some tests > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Robert Haas
On Fri, Feb 14, 2020 at 1:07 PM Andres Freund wrote: > Yea, that seems possible. I'm not really sure it's needed however? As > long as you're not teaching the locking mechanism new tricks that > influence the wait graph, why would the deadlock detector care? That's > quite different from the

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Andres Freund
Hi, On 2020-02-14 12:08:45 -0500, Robert Haas wrote: > On Fri, Feb 14, 2020 at 11:40 AM Andres Freund wrote: > > IMO the right thing here is to extend lock.c so we can better represent > > whether certain types of lockmethods (& levels ?) are [not] to be > > shared. > > The part that I find

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Robert Haas
On Fri, Feb 14, 2020 at 11:40 AM Andres Freund wrote: > IMO the right thing here is to extend lock.c so we can better represent > whether certain types of lockmethods (& levels ?) are [not] to be > shared. The part that I find awkward about that is the whole thing with the deadlock detector. The

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Andres Freund
Hi, On 2020-02-14 09:42:40 -0500, Tom Lane wrote: > In the second place, it's ludicrous to expect that the underlying > platform/filesystem can support an infinite number of concurrent > file-extension operations. At some level (e.g. where disk blocks > are handed out, or where a record of the

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Andres Freund
Hi, On 2020-02-12 11:53:49 -0500, Tom Lane wrote: > In general, if we think there are issues with LWLock, it seems to me > we'd be better off to try to fix them, not to invent a whole new > single-purpose lock manager that we'll have to debug and maintain. My impression is that what's being

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Robert Haas
On Fri, Feb 14, 2020 at 10:43 AM Tom Lane wrote: > I remain unconvinced ... wouldn't both of those claims apply to any disk > I/O request? Are we going to try to ensure that no I/O ever happens > while holding an LWLock, and if so how? (Again, CheckpointLock is a > counterexample, which has

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Tom Lane
Robert Haas writes: > On Wed, Feb 12, 2020 at 11:53 AM Tom Lane wrote: >> * I see no reason to think that a relation extension lock would ever >> be held long enough for noninterruptibility to be a real issue. Our >> expectations for query cancel response time are in the tens to >> hundreds of

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Robert Haas
On Wed, Feb 12, 2020 at 11:53 AM Tom Lane wrote: > Yeah. I would say a couple more things: > > * I see no reason to think that a relation extension lock would ever > be held long enough for noninterruptibility to be a real issue. Our > expectations for query cancel response time are in the tens

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Tom Lane
Amit Kapila writes: > I think MaxBackends will generally limit the number of different > relations that can simultaneously extend, but maybe tables with many > partitions might change the situation. You are right that some tests > might suggest a good number, let Mahendra write a patch and then

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-14 Thread Amit Kapila
On Fri, Feb 14, 2020 at 11:42 AM Masahiko Sawada wrote: > > On Thu, 13 Feb 2020 at 13:16, Amit Kapila wrote: > > > > On Tue, Feb 11, 2020 at 9:13 PM Tom Lane wrote: > > > > > > I took a brief look through this patch. I agree with the fundamental > > > idea that we shouldn't need to use the

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-13 Thread Masahiko Sawada
On Thu, 13 Feb 2020 at 13:16, Amit Kapila wrote: > > On Tue, Feb 11, 2020 at 9:13 PM Tom Lane wrote: > > > > I took a brief look through this patch. I agree with the fundamental > > idea that we shouldn't need to use the heavyweight lock manager for > > relation extension, since deadlock is not

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Mahendra Singh Thalor
On Thu, 13 Feb 2020 at 09:46, Amit Kapila wrote: > > On Tue, Feb 11, 2020 at 9:13 PM Tom Lane wrote: > > > > I took a brief look through this patch. I agree with the fundamental > > idea that we shouldn't need to use the heavyweight lock manager for > > relation extension, since deadlock is not

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Dilip Kumar
On Thu, Feb 13, 2020 at 9:46 AM Amit Kapila wrote: > > On Tue, Feb 11, 2020 at 9:13 PM Tom Lane wrote: > > > > I took a brief look through this patch. I agree with the fundamental > > idea that we shouldn't need to use the heavyweight lock manager for > > relation extension, since deadlock is

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Amit Kapila
On Tue, Feb 11, 2020 at 9:13 PM Tom Lane wrote: > > I took a brief look through this patch. I agree with the fundamental > idea that we shouldn't need to use the heavyweight lock manager for > relation extension, since deadlock is not a concern and no backend > should ever need to hold more than

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Amit Kapila
On Wed, Feb 12, 2020 at 10:23 PM Tom Lane wrote: > > Amit Kapila writes: > > On Wed, Feb 12, 2020 at 7:36 AM Masahiko Sawada > > wrote: > >> On Wed, 12 Feb 2020 at 00:43, Tom Lane wrote: > >>> I would like to suggest that we do something similar to Robert Haas' > >>> excellent hack (daa7527af)

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Tom Lane
Amit Kapila writes: > On Wed, Feb 12, 2020 at 7:36 AM Masahiko Sawada > wrote: >> On Wed, 12 Feb 2020 at 00:43, Tom Lane wrote: >>> I would like to suggest that we do something similar to Robert Haas' >>> excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c, >> My original

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Amit Kapila
On Wed, Feb 12, 2020 at 10:24 AM Andres Freund wrote: > > Hi, > > On 2020-02-11 08:01:34 +0530, Amit Kapila wrote: > > I don't see much downside with the patch, rather there is a > > performance increase of 3-9% in various scenarios. > > As I wrote in [1] I started to look at this patch. My

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-12 Thread Amit Kapila
On Wed, Feb 12, 2020 at 7:36 AM Masahiko Sawada wrote: > > On Wed, 12 Feb 2020 at 00:43, Tom Lane wrote: > > > > I took a brief look through this patch. I agree with the fundamental > > idea that we shouldn't need to use the heavyweight lock manager for > > relation extension, since deadlock is

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-11 Thread Masahiko Sawada
On Tue, 11 Feb 2020 at 11:31, Amit Kapila wrote: > > On Wed, Feb 5, 2020 at 12:07 PM Masahiko Sawada wrote: > > > > > > Unfortunately the environment I used for performance verification is > > no longer available. > > > > I agree to run this test in a different environment. I've attached the > >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-11 Thread Andres Freund
Hi, On 2020-02-11 08:01:34 +0530, Amit Kapila wrote: > I don't see much downside with the patch, rather there is a > performance increase of 3-9% in various scenarios. As I wrote in [1] I started to look at this patch. My problem with itis that it just seems like the wrong direction

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-11 Thread Masahiko Sawada
On Wed, 12 Feb 2020 at 00:43, Tom Lane wrote: > > I took a brief look through this patch. I agree with the fundamental > idea that we shouldn't need to use the heavyweight lock manager for > relation extension, since deadlock is not a concern and no backend > should ever need to hold more than

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-11 Thread Tom Lane
I took a brief look through this patch. I agree with the fundamental idea that we shouldn't need to use the heavyweight lock manager for relation extension, since deadlock is not a concern and no backend should ever need to hold more than one such lock at once. But it feels to me like this

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-10 Thread Amit Kapila
On Wed, Feb 5, 2020 at 12:07 PM Masahiko Sawada wrote: > > > Unfortunately the environment I used for performance verification is > no longer available. > > I agree to run this test in a different environment. I've attached the > rebased version patch. I'm measuring the performance with/without >

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-10 Thread Amit Kapila
On Mon, Feb 10, 2020 at 10:28 PM Mahendra Singh Thalor wrote: > > On Sat, 8 Feb 2020 at 00:27, Mahendra Singh Thalor wrote: > > > > On Thu, 6 Feb 2020 at 09:44, Amit Kapila wrote: > > > > > > > > > The number at 56 and 74 client count seem slightly suspicious. Can > > > you please repeat

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-10 Thread Mahendra Singh Thalor
On Sat, 8 Feb 2020 at 00:27, Mahendra Singh Thalor wrote: > > On Thu, 6 Feb 2020 at 09:44, Amit Kapila wrote: > > > > On Thu, Feb 6, 2020 at 1:57 AM Mahendra Singh Thalor wrote: > > > > > > On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada wrote: > > > > > > > > On Mon, Feb 3, 2020 at 8:03 PM Amit

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-07 Thread Mahendra Singh Thalor
On Thu, 6 Feb 2020 at 09:44, Amit Kapila wrote: > > On Thu, Feb 6, 2020 at 1:57 AM Mahendra Singh Thalor wrote: > > > > On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada wrote: > > > > > > On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila wrote: > > > > > > > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-05 Thread Masahiko Sawada
On Thu, 6 Feb 2020 at 13:16, Amit Kapila wrote: > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada > wrote: > > > > Type of table: normal table, unlogged table > > Number of child tables : 16, 64 (all tables are located on the same > > tablespace) > > Number of clients : 32 > > Number of

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-05 Thread Amit Kapila
On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada wrote: > > Type of table: normal table, unlogged table > Number of child tables : 16, 64 (all tables are located on the same > tablespace) > Number of clients : 32 > Number of trials : 100 > Duration: 180 seconds for each trials > > The hardware

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-05 Thread Amit Kapila
On Thu, Feb 6, 2020 at 1:57 AM Mahendra Singh Thalor wrote: > > On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada wrote: > > > > On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila wrote: > > > > > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada > > > wrote: > > > > > > > > On Fri, Apr 27, 2018 at 4:25

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-05 Thread Mahendra Singh Thalor
On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada wrote: > > On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila wrote: > > > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada wrote: > > > > > > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas wrote: > > > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2020-02-04 Thread Masahiko Sawada
On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila wrote: > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada > wrote: > > > > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas wrote: > > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund wrote: > > >>> I think the real question is whether the scenario is

  1   2   >