roughly same experience over couple commercial, heavy-lifting vendors on my side. modulo that implementation in user space internally sometimes manages queues per interface & different queues in/out per ISIS packet types + plays with size of socket buffers across interfaces (don't forget, that's not infinite resource either). As Les said, each stage can loose independently and there are no feedbacks in any implementaiton/platform I know of between the different buffering stages (albeit they could be built on some platforms but that would make the protocol code mostly nightmarishly platform dependent). Observe that each stage has different timing characterics in terms of processing & that e.g. line card load & CPU load can change the characterics very drastically (that has direct implications on what the max. capacity to "soak packets in" is and with that what the "acceptable flooding rate" is). It's not unheard of on low end, highly loaded platforms to introduce 10s of millisecs delay between packet hitting the link and being read out the socket. On high end or very simple boxes you may be in tens of microsecs, that's a cool 3-4 orders of magnitude. With the advent of VMs and containers things got _much_ worse, both in variation and range. I personally spent a day puzzling over an impossible failing test case in a virtual env until I measured the delay between one node sending a packet and reading it out on the other side in my protocol implementation in this setup. It was _5_ seconds ....
-- tony On Fri, Apr 17, 2020 at 1:42 PM Les Ginsberg (ginsberg) <ginsberg= 40cisco....@dmarc.ietf.org> wrote: > Bruno – > > > > Returning to this old thread… > > > > The following is a generic description of how receive path for IS-IS PDUs > functions today – based on examination of implementations on a variety of > platforms from my employer. It is, for obvious reasons, generic and > intentionally omits any discussion of implementation details. > > But it is hopefully detailed enough to illustrate some of the challenges > in regards to providing real-time accurate feedback to the control plane > that could be used by a receiver based flow control mechanism. > > > > Stage 1: Input Policing > > > > As a first step, a policer operates at ingress to rate limit the number of > packets which will be punted for local processing. This typically operates > in an interface independent manner - limiting the total number of packets > per second across all interfaces. > > The policer may rate limit different classes of packets or simply limit > all types of packets. A given platform may apply a limit specific to IS-IS > PDUs or apply a limit to a class of packets (e.g., OSPF+BGP+IS-IS combined). > > > > Stage 2: Punt Queue Shaping > > > > Received packets are then placed in a queue for eventual transfer to the > control plane. The number of queues varies. In some cases a single queue > for all packet types is used. In other cases packets are placed on > different queues based on packet classes. > > Each queue is typically bounded to a maximum number of packets. As the > queue usage approaches a limit, shaping policies are applied to prioritize > certain packet types over others. > > An upper limit specific to IS-IS packets may be employed - or a limit may > be applied to a larger class of packet types of which IS-IS is only one of > the packet types in the class. > > > > Stage 3: Transfer to control Plane > > > > Packets are then transferred to the control plane. Control plane input > queues may map 1-1 with the data plane queues or map many to 1. > > If the incoming packets are encapsulated (for example GRE) they may be > transferred to a media specific control plane queue to process the > encapsulation header. In some cases encapsulation may be processed in the > data plane prior to transfer to the control plane. > > > > Stage 4: Transfer to IS-IS > > > > Packets are then transferred to a queue which is read directly by IS-IS. > In the event there are multiple IS-IS instances, implementations may choose > to have a shared queue which drives the execution of all instances or have > instance specific queues filtered based on incoming interface. > > > > A single queue is typically used for all interfaces and all IS-IS packet > types. Subsequent processing may requeue packets based on packet type e.g.., > separating processing of hellos from processing of LSPs/SNPs. > > > > ********* > > > > A receiver based flow control mechanism which attempts to make dynamic > adjustments needs to obtain real-time feedback from one or more of the > above stages. Monitoring the state of the input queue to IS-IS is easy to > do, but would not account for drops at previous stages. > > Obtaining feedback from earlier stages requires real-time updates from > data plane to IS-IS in the control plane. This is much more challenging. > Idiosyncrasies of platforms will have a significant impact on how to > meaningfully interpret and use the data. How to integrate data from the > various stages – especially when the numbers are not specific to IS-IS > packets – is not intuitive. > > > > https://tools.ietf.org/html/draft-decraene-lsr-isis-flooding-speed-03 > provides no guidance on how information from the dataplane could be used. > > As an alternative, it suggests that static parameters derived from offline > tests could be advertised. But static parameters would necessarily have to > be conservative as they would need to account for the worst case scenarios. > > > > draft-ginsberg-lsr-isis-flooding-scale proposes dynamic flow control based > on the state of the transmitter. In this model, there is no dependency on > platform implementation. The number of unacknowledged LSPs sent on an > interface is used as input to the flow control algorithm. This accounts for > all reasons why a receiver may be slow to acknowledge without requiring > knowledge of which stage(s) described above are affecting the receiver’s > ability to provide timely acknowledgements. > > > > Les > > > > > > *From:* bruno.decra...@orange.com <bruno.decra...@orange.com> > *Sent:* Wednesday, February 26, 2020 11:03 AM > *To:* Les Ginsberg (ginsberg) <ginsb...@cisco.com> > *Cc:* lsr@ietf.org > *Subject:* RE: Flow Control Discussion for IS-IS Flooding Speed > > > > Les, > > > > Please see inline[Bruno] > > > > *From:* Lsr [mailto:lsr-boun...@ietf.org <lsr-boun...@ietf.org>] *On > Behalf Of *Les Ginsberg (ginsberg) > *Sent:* Wednesday, February 19, 2020 3:32 AM > *To:* lsr@ietf.org > *Subject:* Re: [Lsr] Flow Control Discussion for IS-IS Flooding Speed > > > > Base protocol operation of the Update process tracks the flooding of > > LSPs/interface and guarantees timer-based retransmission on P2P interfaces > > until an acknowledgment is received. > > > > Using this base protocol mechanism in combination with exponential backoff > of the > > retransmission timer provides flow control in the event of temporary > overload > > of the receiver. > > > > This mechanism works without protocol extensions, is dynamic, operates > > independent of the reason for delayed acknowledgment (dropped packets, CPU > > overload), and does not require additional signaling during the overloaded > > period. > > > > This is consistent with the recommendations in RFC 4222 (OSPF). > > > > Receiver-based flow control (as proposed in > https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ ) > > requires protocol extensions and introduces additional signaling during > > periods of high load. The asserted reason for this is to optimize > throughput - > > but there is no evidence that it will achieve this goal. > > > > Mention has been made to TCP-like flow control mechanisms as a model - > which > > are indeed receiver based. However, there are significant differences > between > > TCP sessions and IGP flooding. > > > > TCP consists of a single session between two endpoints. Resources > > (primarily buffer space) for this session are typically allocated in the > > control plane and current usage is easily measurable.. > > > > IGP flooding is point-to-multi-point, resources to support IGP flooding > > involve both control plane queues and dataplane queues, both of which are > > typically not per interface - nor even dedicated to a particular protocol > > instance. What input is required to optimize receiver-based flow control > is not fully specified. > > > > https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ > suggests (Section 5) that the values > > to be advertised: > > > > "use a formula based on an off line tests of > > the overall LSPDU processing speed for a particular set of hardware > > and the number of interfaces configured for IS-IS" > > > > implying that the advertised value is intentionally not dynamic. As such, > > it could just as easily be configured on the transmit side and not require > > additional signaling. As a static value, it would necessarily be somewhat > > conservative as it has to account for the worst case under the current > > configuration - which means it needs to consider concurrent use of the CPU > > and dataplane by all protocols/features which are enabled on a router - > not all of whose > > use is likely to be synchronized with peak IS-IS flooding load. > > [Bruno] _*Assuming*_ that the parameters are static, those parameters > > - are the same as the one implemented (configured) on multiple > implementations, including the one from your employer. Now do you > believe > that those parameters can be determined? > > > - If yes, how do you do _*today*_ on your implementation? (this seems > to contradict your statement that you have no way to figure out how > to find > the right value) > - If no, why did you implement those parameters, and ask network > operator to configure them? > - There is also the option to reply: I don’t know but don’t care > as I leave the issue to the network operator. > > > - can still provide some form of dynamicity, by using the PSNP as > dynamic acknowledgement. > - are really dependent on the receiver, not the sender. > > > - the sender will never overload itself. > - The receiver has more information, knowing its processing > power (low end, high end, 80s, 20s (currently we are stuck with 20 > years > old value assuming the worst possible receiver (and worst there were, > including with packet processing partly done in the control plane > processor)), its expected IS-IS load (#neighbors), its preference for > bursty LSP reception (high delay between IS-IS CPU allocation cycles, > memory not an issue up to x kilo-octet…), its expected control plane > load > if IS-IS traffic has not higher priority over other control plane > traffic…), it’s expected level of QoS prioritization [1] > > > - [1] looks for “Extended SPD Headroom”. E.g. “Since IGP and link > stability are more tenuous and more crucial than BGP stability, > such > packets are now given the highest priority and are given extended > SPD > headroom with a default of 10 packets. This means that these > packets are > not dropped if the size of the input hold queue is lower than 185 > (input > queue default size + spd headroom size + spd extended headroom).” > > > - And this is for distributed architecture, 15 years ago. So what > about using the above number (in the router configuration), > applies Tony’s > proposal (*oversubscription/number of IS neighbhors), and > advertise this > value to your LSP sender? > > > > [1] > https://www.cisco.com/c/en/us/support/docs/routers/12000-series-routers/29920-spd.html > > > > - > > --Bruno > > > > > > Unless a good case can be made as to why transmit-based flow control is > not a good > > fit and why receiver-based flow control is demonstrably better, it seems > > unnecessary to extend the protocol. > > > > Les > > > > > > *From:* Lsr <lsr-boun...@ietf.org> *On Behalf Of *Les Ginsberg (ginsberg) > *Sent:* Tuesday, February 18, 2020 6:25 PM > *To:* lsr@ietf.org > *Subject:* [Lsr] Flow Control Discussion for IS-IS Flooding Speed > > > > Two recent drafts advocate for the use of faster LSP flooding speeds in > IS-IS: > > > > https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ > > https://datatracker.ietf.org/doc/draft-ginsberg-lsr-isis-flooding-scale/ > > > > There is strong agreement on two key points: > > > > 1)Modern networks require much faster flooding speeds than are commonly in > use today > > > > 2)To deploy faster flooding speeds safely some form of flow control is > needed > > > > The key point of contention between the two drafts is how flow control > should be implemented. > > > > https://datatracker.ietf.org/doc/draft-decraene-lsr-isis-flooding-speed/ > advocates for a receiver based flow control where the receiver advertises > in hellos the parameters which indicate the rate/burst size which the > receiver is capable of supporting on the interface. Senders are required to > limit the rate of LSP transmission on that interface in accordance with the > values advertised by the receiver. > > > > https://datatracker.ietf.org/doc/draft-ginsberg-lsr-isis-flooding-scale/ > advocates for a transmit based flow control where the transmitter monitors > the number of unacknowledged LSPs sent on each interface and implements a > backoff algorithm to slow the rate of sending LSPs based on the length of > the per interface unacknowledged queue. > > > > While other differences between the two drafts exist, it is fair to say > that if agreement could be reached on the form of flow control then it is > likely other issues could be resolved easily. > > > > This email starts the discussion regarding the flow control issue. > > > > > > _________________________________________________________________________________________________________________________ > > > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > > they should not be distributed, used or copied without authorisation. > > If you have received this email in error, please notify the sender and delete > this message and its attachments. > > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > > Thank you. > > _______________________________________________ > Lsr mailing list > Lsr@ietf.org > https://www.ietf.org/mailman/listinfo/lsr >
_______________________________________________ Lsr mailing list Lsr@ietf.org https://www.ietf.org/mailman/listinfo/lsr
