Kais Belgaied wrote: > The following is mainly a capture of parts of multiple off-line > discussions within members of the Crossbow team > (Gopi, Thiru, Roamer, May-Lin, Thirumailai, Nitin, KB, ...), I thought > I'd open it up to other participants. > > Crossbow's core scheduling involves switching a NIC (or individual Rx > rings on the NIC) to polling mode. > The receive interrupt will become not only rarer, but more importantly > outside the control of the NIC drivers. > Some drivers, on the other hand, were designed and written before that > change. They used to > piggy-back the tx descriptor reclaiming at the end of the Rx interrupt > for example. At the same time, they > disable to transmit interrupt all together, in an effort to minimize the > number of interrupts to the host (and > the entailed context switches). As expected, that sort of approach will > (and it was actually observed) lead > to quick exhaustion of the Tx descriptors, with no one to reclaim them > soon enough under polling mode. > > Multiple choices have been suggested, and some actually implemented in > some drivers: > . A timeout thread that just reclaims periodically > wonder, what would be priority of such a garbage collection thread? if it never gets scheduled, or gets scheduled less periodically compared to other control/data plane threads, then we still have the issue of out-of-sync reclaims. > . Reclaim some descriptors at the beginning of or before returning from > the tx routine > would TX app be blocked until Tx is done and Tx desc. is recycled back to pool? unless it is called out for some later times to be finished. (tell app, Tx is done, while scheduling of actual descriptor recycling routine, scheduled to be executed at some later time)? > . Turn on tx interrrupts per transmit ring. > . Have a single NIC-wide interrupt shared for all tx ring's transmit > completion events, and turn that interrupt ON. > no comment, as do not understand above. > . Have a high water mark/low water mark for triggering the reclaiming > from the transmit routine. > this could be optimization when to trigger a reclaiming thread to collect the descriptor. > . Some internal tunables to choose one or a combination of the > approaches above (how to detect the right time > to change the tunables' setting?) > > I am wondering what has been the experience of folks I have no direct experience, but If performance is observed to be lost, because of unavailability of Tx descriptor (say we observe thruput loss, in data-path due to always wait/block on Tx descriptor)., then having a solution to find TX descriptor and keep for future need should buy perf. gain, I think.
so a combined solution of above ideas, i.e. a track of high watermark+low watermark of available descriptors, and kicking the garbage/Tx-desc collection thread when available descriptors is < =low watermark, but with higher or equal priority, should help. 2 cents, -Deepti > the performance gain or penalty > from the various choices above, or if they used different way for > efficiently and promptly reclaim. > > ideas? comments? > > Kais > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
