Re: [RFC] serialising net80211 TX
On Tue, Feb 19, 2013 at 7:01 PM, Adrian Chadd wrote: > On 19 February 2013 09:44, Monthadar Al Jaberi wrote: > >> I have tested between two meshSTAand run iperf UDP until I get close >> to getting lost packets. >> >> Current FreeBSD (r246983): 17 Mbits/ >> Current FreeBSD (r246983) + with your patch: 15 Mbit/s >> >> Good news no panic! But looks like a bit slower. I should add I have >> noticed before that mesh is slower than other modes. > > This is with witness, right? No I forgot that part, will try again :) > > You can do some lock profiling to see whether there's any locking > issues creeping up. > > and yeah, mesh will be a bit slower, at least until we figure out how > to enable aggressive mode properly for it and IBSS modes. > > > > Adrian -- Monthadar Al Jaberi ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On 19 February 2013 09:44, Monthadar Al Jaberi wrote: > I have tested between two meshSTAand run iperf UDP until I get close > to getting lost packets. > > Current FreeBSD (r246983): 17 Mbits/ > Current FreeBSD (r246983) + with your patch: 15 Mbit/s > > Good news no panic! But looks like a bit slower. I should add I have > noticed before that mesh is slower than other modes. This is with witness, right? You can do some lock profiling to see whether there's any locking issues creeping up. and yeah, mesh will be a bit slower, at least until we figure out how to enable aggressive mode properly for it and IBSS modes. Adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On Tue, Feb 19, 2013 at 5:49 PM, Adrian Chadd wrote: > On 19 February 2013 07:50, Monthadar Al Jaberi wrote: >> On Tue, Feb 19, 2013 at 4:10 PM, Adrian Chadd wrote: >>> Here's my current patch. I've tested it lightly in AP mode. >> >> Its identical to 20130216-net80211-tx-lock-2.diff? >> > > Good. That means I haven't accidentally changed anything since the > last diff. :-) Good then I dont need to re-apply ;) I have tested between two meshSTAand run iperf UDP until I get close to getting lost packets. Current FreeBSD (r246983): 17 Mbits/ Current FreeBSD (r246983) + with your patch: 15 Mbit/s Good news no panic! But looks like a bit slower. I should add I have noticed before that mesh is slower than other modes. br, > > I'm using that on my test (open) AP at the moment. Nothing that uses > active powersave is actually using it yet though. > > Thanks, > > > > adrian -- Monthadar Al Jaberi ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On 19 February 2013 07:50, Monthadar Al Jaberi wrote: > On Tue, Feb 19, 2013 at 4:10 PM, Adrian Chadd wrote: >> Here's my current patch. I've tested it lightly in AP mode. > > Its identical to 20130216-net80211-tx-lock-2.diff? > Good. That means I haven't accidentally changed anything since the last diff. :-) I'm using that on my test (open) AP at the moment. Nothing that uses active powersave is actually using it yet though. Thanks, adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On Tue, Feb 19, 2013 at 4:10 PM, Adrian Chadd wrote: > Here's my current patch. I've tested it lightly in AP mode. Its identical to 20130216-net80211-tx-lock-2.diff? > > > > Adrian -- Monthadar Al Jaberi ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
Here's my current patch. I've tested it lightly in AP mode. Adrian 20130218-net80211-tx-lock-3.diff Description: Binary data ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
oh.. right. We talked about this on irc a bit. Yes, mesh_transmit_to_gate() looks like a copy of what I put in ieee80211_start_pkt(). We'll refactor this common code out later. :-) Same as what we should do with the rest of the duplicate code in the input path for various net80211 modules.. adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On 18 February 2013 02:05, Monthadar Al Jaberi wrote: > On Sun, Feb 17, 2013 at 1:46 AM, Adrian Chadd wrote: >> Here's version two of the patch: it includes the rest of the TX paths >> locked as well as some extra assertions about the state of the TX lock >> requirements. > > Why not make ieee80211_start_pkt public? So that we can call it from > mesh_transmit_to_gate in mesh.c? My aim is to have ieee80211_start_pkt() be what dispatches an actual packet, whilst going via if_transmit() will handle queuing, hecking for interface status/CAC, etc. adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On Sun, Feb 17, 2013 at 1:46 AM, Adrian Chadd wrote: > Here's version two of the patch: it includes the rest of the TX paths > locked as well as some extra assertions about the state of the TX lock > requirements. Why not make ieee80211_start_pkt public? So that we can call it from mesh_transmit_to_gate in mesh.c? > > I've only lightly tested this in STA mode. I haven't yet really tested > this in hostap mode. I gather it's going to panic all the things. > Thus, please only run this on a test device. :-) > > > > Adrian -- Monthadar Al Jaberi ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
Here's version two of the patch: it includes the rest of the TX paths locked as well as some extra assertions about the state of the TX lock requirements. I've only lightly tested this in STA mode. I haven't yet really tested this in hostap mode. I gather it's going to panic all the things. Thus, please only run this on a test device. :-) Adrian 20130216-net80211-tx-lock-2.diff Description: Binary data ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
Hi, Here's the patch: * break out the per-packet send code into ieee80211_start_pkt() * introduce a per-vap TX lock * wrap the normal TX path ieee80211_encap(), parent->if_transmit and ic->ic_raw_xmit with the TX lock TODO: * the rest of the encap calls need to be wrapped in the TX lock - wds, superg, mesh. * the TX lock is not re-entrant but the TX path itself is (eg start -> start aggregation -> addba send -> mgmt_send -> (re) grab lock) so a bunch of this stuff needs to be rethought to be "clean" locking wise. I bet wds, superg and mesh is being called from the normal TX path and thus could be called with the TX lock already held. I should investigate this! * the TX aggregation state code should be protected by a lock - do that later? * add some lock / unlock assertion checks (eg assert the TX lock is held in ieee80211_encap()) and debug what code paths are / aren't being engaged. I'm still testing this. I've only tested it in STA mode thus far (iwn and ath.) thanks, Adrian Adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
Yeah - gluing TX into a single send queue is going to work, but there's definite queuing / QoS issues. You really want the management frames to be able to go out even if there's a _large_ amount of data frames going out. That's a side-issue though. I have a test patch that just wraps the output, mgmt_output and encapsulation/send routines with a TX lock. I can't wrap the whole TX path in a lock because there's some re-entrant behaviour going on (specifically net80211_start -> ampdu check -> send addba -> mgmt_output -> (re) grab TX lock). Thanks, Adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On Fri, Feb 15, 2013 at 4:03 PM, Adrian Chadd wrote: > The (many) problems include the two TX paths - one is via _start / > _transmit, and one is the raw xmit path. > > So in the short term it'll be easier (!) to just wrap all TX entry > points with a VAP TX lock. > Once that's done and we've debugged it, we can look at breaking it > into a queue with a taskqueue thread for deferred transmit. > > How's that sound? I think that's good because if_start/transmit == data packet && if_raw_xmit == mgmt packet is no longer true. http://svnweb.freebsd.org/base?view=revision&revision=222682 And, all-in-one Tx path works with run(4) at least. AK ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
The (many) problems include the two TX paths - one is via _start / _transmit, and one is the raw xmit path. So in the short term it'll be easier (!) to just wrap all TX entry points with a VAP TX lock. Once that's done and we've debugged it, we can look at breaking it into a queue with a taskqueue thread for deferred transmit. How's that sound? Adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
> -- > > Message: 2 > Date: Wed, 13 Feb 2013 21:14:53 -0800 > From: Adrian Chadd > To: freebsd-wireless@freebsd.org > Subject: [RFC] serialising net80211 TX > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > I'd like to work on the net80211 TX serialisation now. I'll worry > about driver serialisation and if_transmit methods later. > > The 30 second version - it happens in parallel, which means preemption > and multi-core devices can and will hit lots of subtle and > hard-to-debug races in the TX path. > > We actually need an end-to-end serialisation method - not only for the > 802.11 state (sequence number, correct aggregation handling, etc) but > to line up 802.11 sequence number allocation with the encryption IV/PN > values. Otherwise you end up with lots of crazy subtle out of order > packets occuring. The other is the seqno/CCMP IV race between the raw > transmit path and the normal transmit path. There are other nagging > issues that I'm trying to resolve - but, one thing at a time. > > So there are three current contenders: > > * wrap everything in net80211 TX in a per-vap TX lock; grab it at the > beginning of ieee80211_output() and ieee80211_start(), and don't > release it until the frame is queued to something (a power save queue, > an age queue, the driver.) That guarantees that the driver is called > in lock-step with each frame being processed. Long held locks could be worse. While another thread holding a lock, when one thread try to grab a lock, it will spin a bit then be suspended. This could be more expensive than context switching by the scheduler. If a thread hold a lock longer, there will be more chance this to happen. > * do deferred transmit- ie, the net80211 entry points simply queue > mbufs to a queue, and a taskqueue runs over the ifnet queue and runs > those frames in-order. There's no need for a lock here as there's only > one sending context (either per-VAP or per-IC). I tried taskqueue on run(4), changed it from 1) to 2). Queue with shared thaskqueue, i.e taskqueue_thread, didn't work good, but with own taskqueue works good (60mbp->70+mbp). The catch is I only tested on core2 duo and dual core + ht atom. I saw you patched ath(4) to use taskqueue. How is it working? There are threads other than Tx thread are running on a system, so context switching will happen one way or other. We want to do it smart way, i.e. rather than changing thread on every tx, switch on multiple tx. Currently, tx threads are kept alive until packets are passed to h/w. Instead, if we kill them right after queuing a frame to if_snd and let one thread (would be new ieee80211_tx thread) handle all the packets, there will be fewer threads, so should be less context switching. Anyhow, I don't think we need to queue/dequeue twice, one on vap->iv_ifp->if_snd and one on ic->ic_ifp->if_snd. AK ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On Fri, Feb 15, 2013 at 2:49 AM, Adrian Chadd wrote: > On 14 February 2013 11:50, Monthadar Al Jaberi wrote: > >> Seems like the best architectural wise, first-in first out. I am just >> thinking of one can extend this too have like more than one queue, >> more like the QoS concept, and each packet have a time-stamp assigned >> to it.Would that help? > > Well, the queue thing is a bit orthogonal. Yes, we could run multiple > queues and multiple kernel threads. But what would multiple kernel > threads get us? I was thinking one kernel thread. This thread have some headroom to take from the higher priority queue if the frame on the other queue is not old "enough". > > > > Adrian -- Monthadar Al Jaberi ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On 14 February 2013 11:50, Monthadar Al Jaberi wrote: > Seems like the best architectural wise, first-in first out. I am just > thinking of one can extend this too have like more than one queue, > more like the QoS concept, and each packet have a time-stamp assigned > to it.Would that help? Well, the queue thing is a bit orthogonal. Yes, we could run multiple queues and multiple kernel threads. But what would multiple kernel threads get us? Adrian ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
Re: [RFC] serialising net80211 TX
On Thu, Feb 14, 2013 at 6:14 AM, Adrian Chadd wrote: > Hi, > > I'd like to work on the net80211 TX serialisation now. I'll worry > about driver serialisation and if_transmit methods later. > > The 30 second version - it happens in parallel, which means preemption > and multi-core devices can and will hit lots of subtle and > hard-to-debug races in the TX path. > > We actually need an end-to-end serialisation method - not only for the > 802.11 state (sequence number, correct aggregation handling, etc) but > to line up 802.11 sequence number allocation with the encryption IV/PN > values. Otherwise you end up with lots of crazy subtle out of order > packets occuring. The other is the seqno/CCMP IV race between the raw > transmit path and the normal transmit path. There are other nagging > issues that I'm trying to resolve - but, one thing at a time. > > So there are three current contenders: > > * wrap everything in net80211 TX in a per-vap TX lock; grab it at the > beginning of ieee80211_output() and ieee80211_start(), and don't > release it until the frame is queued to something (a power save queue, > an age queue, the driver.) That guarantees that the driver is called > in lock-step with each frame being processed. > * do deferred transmit- ie, the net80211 entry points simply queue > mbufs to a queue, and a taskqueue runs over the ifnet queue and runs > those frames in-order. There's no need for a lock here as there's only > one sending context (either per-VAP or per-IC). Seems like the best architectural wise, first-in first out. I am just thinking of one can extend this too have like more than one queue, more like the QoS concept, and each packet have a time-stamp assigned to it.Would that help? What I am thinking is more of an optimization maybe. the Wifi can run in different modes BSS (AP/STA), IBSS and MBSS. And for the MBSS mode there is not just data comming from up the stack, or sending beacons. The MBSS also needs to handle data forwarding and management frame propagation. Would it help to have different queues? A mesh path discovery would be delayed too much if the complete system is highly active? Should the path request frames have higher priority to lets say data frames? Would that help for when sending beacons from the stack too. Maybe I am way off :) > * A hybrid setup - use a per-vap TX lock; do a try-acquire on it and > direct dispatch from the queue head if we have it; otherwise defer > frames into a queue and have a taskqueue handle those. That maybe gives the most speed up? > > 1) is what drivers like iwn(4) do internally. > 2) is what I've tinkered with - but we become a slave to the > scheduler. Task switching is expensive and unpredictable; doubly so > for a non-preemption kernel. We'd have to run the TX taskqueue at some > very high priority to get something resembling direct-dispatch > behaviour. > 3) is what the gige/10ge drivers do. They hold a big TX lock for each > TX (from xxx_transmit() to hardware dispatch) and if they can't > acquire the TX lock, they defer it to a drbd lockless ring buffer and > service that via a taskqueue. > > I can implement any of the above. architecturally I'd prefer 2) - it > massively simplifies and streamlines things, but the scheduling > latency is just plain stab-worthy.I'm tempted to just do 1) for now > and turn it into 3) if we need to. > > The main reason against doing 1) (and why 2) is nicer) is recursion - > if the TX path wants to call the net80211 TX code for some odd reason, > we'll hit lock recursion. I'd rather have the system crash at this > point (and then fix the misbehaving driver) but that's just me. > > So - what do people think? > > Once this is done I'd like to make sure that the wifi chipset drivers > do the same - ie, ensure that the frame order is preserved both > between the normal and the raw xmit paths. > That should fix all of the odd CCMP out of order crap that I see under > heavy, heavy test conditions. > > Thanks, > > > Adrian > ___ > freebsd-wireless@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-wireless > To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org" -- Monthadar Al Jaberi ___ freebsd-wireless@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-wireless To unsubscribe, send any mail to "freebsd-wireless-unsubscr...@freebsd.org"
