Re: [rfc] migrate lagg to an rmlock
On Saturday, August 24, 2013 10:16:33 am Robert Watson wrote: There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. 3) should no longer be an issue. rmlocks now have full WITNESS and assertion support (including an rm_assert). However, one thing to consider is that rmlocks pin readers to CPUs while the read lock is held (which rwlocks do not do). -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On Thursday, August 29, 2013 11:37:08 am Scott Long wrote: On Aug 29, 2013, at 7:42 AM, John Baldwin j...@freebsd.org wrote: On Saturday, August 24, 2013 10:16:33 am Robert Watson wrote: There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. 3) should no longer be an issue. rmlocks now have full WITNESS and assertion support (including an rm_assert). However, one thing to consider is that rmlocks pin readers to CPUs while the read lock is held (which rwlocks do not do). And this is not a problem for the application that we're giving it in the lagg driver. That is likely true. I was merely tweaking Robert's general guidelines re: rmlock. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On Aug 29, 2013, at 7:42 AM, John Baldwin j...@freebsd.org wrote: On Saturday, August 24, 2013 10:16:33 am Robert Watson wrote: There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. 3) should no longer be an issue. rmlocks now have full WITNESS and assertion support (including an rm_assert). However, one thing to consider is that rmlocks pin readers to CPUs while the read lock is held (which rwlocks do not do). And this is not a problem for the application that we're giving it in the lagg driver. Scott ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On Sat, Aug 24, 2013 at 7:16 AM, Robert Watson rwat...@freebsd.org wrote: On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: On 24.08.2013 00:54, Adrian Chadd wrote: I'd like to commit this to -10. It migrates the if_lagg locking from a rw lock to a rm lock. We see a bit of contention between the transmit and We're running lagg with rmlock on several hundred heavily loaded machines, it really works better. However, there should not be any contention between receive and transmit side since there is actually no _real_ need to lock RX (and even use lagg receive code at all): http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place I'm not sure what you mean here with (3), because from my understanding of the code WITNESS is implemented both in the sleepable and non-sleepable case, but there could be something I'm missing. Something I think we lack in rmlock code is fully supported LOCK_PROFILING as we have in all the other primitives, but again, if I'm wrong feel free to correct me. to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Thanks, -- Davide There are no solved problems; there are only problems that are more or less solved -- Henri Poincare ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24.08.2013 00:54, Adrian Chadd wrote: Hi, I'd like to commit this to -10. It migrates the if_lagg locking from a rw lock to a rm lock. We see a bit of contention between the transmit and We're running lagg with rmlock on several hundred heavily loaded machines, it really works better. However, there should not be any contention between receive and transmit side since there is actually no _real_ need to lock RX (and even use lagg receive code at all): http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html receive sides of lagg during traffic loads (10+ gigabit per second.) Using rmlocks eliminate this. http://people.freebsd.org/~adrian/netflix/20130819-lagg-rmlock-1.diff Thanks, -adrian ___ freebsd-...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlIYqjYACgkQwcJ4iSZ1q2n9fgCePHOfC3tzBIG54ayNg7d8TKMC gIMAn2/paUBpDIRVd+3s7snNFCmZNWgd =i6Ye -END PGP SIGNATURE- ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: On 24.08.2013 00:54, Adrian Chadd wrote: I'd like to commit this to -10. It migrates the if_lagg locking from a rw lock to a rm lock. We see a bit of contention between the transmit and We're running lagg with rmlock on several hundred heavily loaded machines, it really works better. However, there should not be any contention between receive and transmit side since there is actually no _real_ need to lock RX (and even use lagg receive code at all): http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
Sorry, I meant line contention rather than lock contention. Yes, you're right. -adrian On 24 August 2013 07:16, Robert Watson rwat...@freebsd.org wrote: On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: On 24.08.2013 00:54, Adrian Chadd wrote: I'd like to commit this to -10. It migrates the if_lagg locking from a rw lock to a rm lock. We see a bit of contention between the transmit and We're running lagg with rmlock on several hundred heavily loaded machines, it really works better. However, there should not be any contention between receive and transmit side since there is actually no _real_ need to lock RX (and even use lagg receive code at all): http://lists.freebsd.org/**pipermail/svn-src-all/2013-**April/067570.htmlhttp://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On 8/24/13 7:16 AM, Robert Watson wrote: On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: On 24.08.2013 00:54, Adrian Chadd wrote: I'd like to commit this to -10. It migrates the if_lagg locking from a rw lock to a rm lock. We see a bit of contention between the transmit and We're running lagg with rmlock on several hundred heavily loaded machines, it really works better. However, there should not be any contention between receive and transmit side since there is actually no _real_ need to lock RX (and even use lagg receive code at all): http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert ___ freebsd-...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org Robert, what do you think about a quick swap of the ifnet structures to counter before 10.x? -Alfred -- Alfred Perlstein ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On 24 Aug 2013, at 17:36, Alfred Perlstein wrote: We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert, what do you think about a quick swap of the ifnet structures to counter before 10.x? Could you be more specific about the proposal you're making? Robert ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On 8/24/13 10:47 AM, Robert N. M. Watson wrote: On 24 Aug 2013, at 17:36, Alfred Perlstein wrote: We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert, what do you think about a quick swap of the ifnet structures to counter before 10.x? Could you be more specific about the proposal you're making? Robert The lagg patch referred to in the thread seems to indicate that zero locking is needed if we just switched to counter(9), that makes me wonder if we could do better with locking in other places if we switched to counter(9) while we have the chance. This is the thread: http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html / //Perfect solution would be to convert ifnet(9) to counters(9), but this //requires much more work, and unfortunately ABI change, so temporarily //patch lagg(4) manually. // //We store counters in the softc, and once per second push their values //to legacy ifnet counters./ -- Alfred Perlstein ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
Hihi, There's two parts to my patch: * one is migrating the rwlock to rmlock - not because of counters, but because the lock is protecting consistency when changing the lagg config * one is adding a new lock specifically to ensure that the callout is atomically created/called/destroyed The latter has nothing to do with the actual counters - they're already using the atomic counter API, so the lock doesn't need to be held just to read them for _counter_ consistency. It's just held to make sure the callout is never called parallel to the destruction of the lagg interface. -adrian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [rfc] migrate lagg to an rmlock
On Aug 24, 2013, at 12:14 PM, Alfred Perlstein bri...@mu.org wrote: On 8/24/13 10:47 AM, Robert N. M. Watson wrote: On 24 Aug 2013, at 17:36, Alfred Perlstein wrote: We should distinguish lock contention from line contention. When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as contention. In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert, what do you think about a quick swap of the ifnet structures to counter before 10.x? Could you be more specific about the proposal you're making? Robert The lagg patch referred to in the thread seems to indicate that zero locking is needed if we just switched to counter(9), that makes me wonder if we could do better with locking in other places if we switched to counter(9) while we have the chance. This is the thread: http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html / //Perfect solution would be to convert ifnet(9) to counters(9), but this //requires much more work, and unfortunately ABI change, so temporarily //patch lagg(4) manually. ////We store counters in the softc, and once per second push their values //to legacy ifnet counters./ Some sort of gatekeeper semantic is needed to ensure that configuration changes to the lagg state don't cause incorrect behavior to the data path. It's not about protecting the integrity of counters. This can be done several ways, but right now it's via a read/write semantic. Scott ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[rfc] migrate lagg to an rmlock
Hi, I'd like to commit this to -10. It migrates the if_lagg locking from a rw lock to a rm lock. We see a bit of contention between the transmit and receive sides of lagg during traffic loads (10+ gigabit per second.) Using rmlocks eliminate this. http://people.freebsd.org/~adrian/netflix/20130819-lagg-rmlock-1.diff Thanks, -adrian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
