Tiger (and other FARE authors), I've made some of these comments at the IETF mic before.
When I discuss the BGP link-bandwidth community with people learning BGP, I start by focusing on the fact that the WECMP values are derived from the received bandwidth values at a given BGP speaker. The implementations largely flatten the much larger floating point value into a small index range ECMP value in the line cards. The remaining part of my discussion is the same signaling mechanism, link-bandwidth, may run at many places with different agendas. Our most boring example, the original use case, is simply biasing the ECMP for multiple BGP routes, typically received from an external AS. A common second use case is that when link-bw is used by the routes used for resolving BGP next hops, we often are interested in the cumulative bandwidth at a given node since it's the available bandwidth to the destination next hop. In that case, we need to "do math" to handle the cumulative operation. Exactly what form that math takes will depend on the operator's desires. The result is simply used to bias the ECMP of service routes resolving over that ECMP. Another use case, primarily the one addressed in your draft, is the computation of the minimal bandwidth. In cases where the traffic to a destination may consist of heavy "elephant flows" that need to know the *least* bandwidth available on the path to the next hop, especially when it's more than one hop away. (A degenerate case are other bandwidth reservation schemes that may partition a link or path among multiple use cases.) In each of these cases, the link-bandwidth community in one or both of its transitivity behaviors (usually transitive, overlapping why Juniper's implementation was transitive prior to RFC 10005) is sufficient to signal the capacity and permit the receiving BGP speakers to correctly do the ECMP. However, what is required in such cases is careful operational management of each scenario at a given ECMP layer (service, underlay, etc.) to get the expected result. The scenarios tended to be limited based on either the automatic "math" infrastructure the implementations had, or available options in the policy language. To the above point, the existing link-bandwidth "just works". However, the semantic is overloaded. In some scenarios, particularly where the underlay routes may want cumulative bandwidth, and overlay or egress routes may want minimum bandwidth, operational care to not let the bandwidth use cases cross is needed. As I've said previously, an additional community that helps to segregate the use cases at the different scenario layers could be helpful. However, since we're at adoption time, it'd be helpful as part of your case to do the following: - Recognize "this can be done using link-bw" (the dispute about who created the use case first aside - see the prior discussion for where this matter current stands) - Discuss the scenarios where clarity of community can ease deployment considerations. (Removing some of the need for "operational care") - And, most importantly, discuss how the FARE path-bw community interacts with the existing link-bw community. This is needed for interop no matter what. This latter point certainly should be in your draft, and obviously is an excellent candidate for presenting the scenarios at the upcoming IETF meeting. -- Jeff > On Sep 7, 2026, at 09:31, Tiger Xu <[email protected]> wrote: > > Hi Jeff, > > Sorry for the late reply since I just found this email in my junk folder. Let > me try to clarify the logic of FARE as follows. > > 发件人: Jeff Tantsura <[email protected]> > 日期: 星期一, 2026年9月7日 11:09 > 收件人: Robert Raszuk <[email protected]> > 抄送: Stephane Litkowski (slitkows) <[email protected]>, idr <[email protected]> > 主题: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) > > Hi Robert, > > If you were to overwrite remotely received BW by locally configured one, you > would lose the “cumulative” nature of the proposal. It accumulates the value > rather than overwriting it, making it working in RFC7938-like fabrics. > If not all paths received for a prefix have BW, advertising “incomplete” set > would be ambiguous. Common policy would be: > If none present: > - don’t advertise ∑BW (safest, since lack of BW is most likely > misconfiguration/bug) > or (local policy) > - set default (locally configured) BW (defats the purpose, since it > overwrites the previously accumulated value) > > All good points. And worth discussing in the wg. > > in line with what Stephane is saying and how I see this evolving: > to separate what is encoded from how it is processed - > FARE's single-session three-stage MIN-then-SUM recurrence does not require a > new Extended Community. Given identical multipath membership and local > policy, it is a profile of RFC 10005 and eBGP-DMZ: Section 4.2 gives per-path > MIN(local-link-bandwidth, received-bandwidth), Section 4.3 by default sums it > across the selected multipath set - the arithmetic RFC 10005 Section 3.4 > delegates there. No new wire information. > > [Tiger] FARE does not require a single session per adjacent pair. Instead, it > is fine to run multiple sessions per pair (e.g., one per physical link). > Admittedly, the existing link-bandwidth extended community could be abused as > a container for path bandwidth to realize the path-bandwidth-aware WECMP > logic described in the FARE draft. However, see below... > > Only the five-stage case carries two differently scoped values; RFC 10005 > Sections 3.1 and 4 already provides two communities on one prefix. > > [Tiger] Details matter here. Please review Section 7.1 (Inconsistent > Deployment) alongside Sections 3.1 and 3.2 of RFC10005 to see how these two > EC types should be used according to the specification. > Publish eBGP-DMZ, it meets all IDR/BESS criteria - widely deployed, large > number of interoperating implementations. > Publish additional use cases as separate drafts, building on top of existing > encodings, that complement it rather than creating ambiguity. > > [Tiger] The FARE protocol has more than 8 implementations and has been widely > deployed (see > https://datatracker.ietf.org/meeting/126/materials/slides-126-idr-sessa-1-5-1-fully-adaptive-routing-ethernet-fare-00). > > The FARE draft already thoroughly describes the path-bandwidth-aware WECMP > logic for both three-stage and five-stage CLOS networks. This was introduced > more than a year before the eBGP-DMZ draft duplicated the same logic, which > only addresses the three-stage case with minimal elaboration (see > https://mailarchive.ietf.org/arch/msg/idr/nzjkS2675LChF5WqrBmfUfpukMM/). To > avoid ambiguity between the semantics and intended usages of path bandwidth > versus link bandwidth, and to deliver comprehensive, suitable solutions for > both 3-stage and 5-stage CLOS networks, I suggest that the eBGP-DMZ draft > strictly limit itself to the existing use cases described in Section 3—i.e., > the scenarios for which the link-bandwidth attribute was originally > designed—rather than duplicating functionality already covered elsewhere. > This approach would both enhance the quality of the draft and uphold > professional ethics. > BTW, based on previous discussions on the same draft (see > https://mailarchive.ietf.org/arch/msg/idr/I3KPpGUuyouHp7Nca4qnEYwI_H4/), it > would be inappropriate to advance eBGP-DMZ as a Standards Track > RFC—particularly within the BESS WG. > > Best regards, > Tiger > > >> On Sep 6, 2026, at 16:46, Robert Raszuk <[email protected]> wrote: >> >> Hi Jeff, >> >> > (as a safety measure - most implementation would not advertise BW if not >> > all contributing routes have it attached) >> >> I am not following this "safety measure". What reception of a path with link >> bw extended community would have to do with advertising any bw - which >> clearly is taken from local interface(s) configuration. >> >> > nothing prevent an implementation to choose the lowest value rather than a >> > sum of all values. >> >> Well the receiver will see some value in the extended community. >> >> Such value can be either a: >> >> - bw of a link best path was received from >> - as you say cumulative bw across all paths eligible and installed in RIB >> for multipath >> - as you say min bw across all paths eligible and installed in RIB for >> multipath >> >> Today there is no way to include in the advertisement any indicator which >> one of those three is being advertised so that would need to be configured >> by hand on a per peer basis. >> >> IMO if we have a real use case for all three they should be assigned >> different extended communities to avoid configuration mismatch and mistakes. >> >> And what happens when some peers advertise different meanings of the bw ? >> >> Also the behaviour when building hash on the receiving router would be quite >> different - especially when we are considering the reception of the min bw. >> The problem is that if we just limit the advertisement to min bw (and that >> is also actually a question to fare authors) we can not construct the has to >> accurately create the weighted load balancing. >> >> Best, >> R. >> >> >> On Sun, Sep 6, 2026 at 12:31 AM Jeff Tantsura <[email protected] >> <mailto:[email protected]>> wrote: >>> Hi Robert, >>> >>> Your question was specific enough, my response is specific to cumulative >>> bw. This is exactly what is deployed in one of the hyperscalers and most of >>> AIDCs. >>> Common use is to advertise the cumulative weight of all next-hops the route >>> has been learnt from (as a safety measure - most implementation would not >>> advertise BW if not all contributing routes have it attached), nothing >>> prevent an implementation to chose the lowest value rather than a sum of >>> all values. >>> >>> Thanks, >>> Jeff >>> >>> >>>> On Sep 4, 2026, at 03:57, Robert Raszuk <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi Jeff, >>>> >>>> I know there are many implementations of link bandwidth extended >>>> community. >>>> >>>> Maybe my question was not articulated very clearly. >>>> >>>> I was specifically asking which shipping implementations support >>>> cumulative bw and minimum bw advertisements among ECMP links. >>>> >>>> But even if there is not many of them now it can be added so not trying to >>>> justify -fare progress based on those missing. >>>> >>>> The only justification to -fare progressing (with hopefully all AI stuff >>>> removed from the draft) would be finding a case where it is beneficial to >>>> advertise say more then one bw parameter for a group of ECMP ebgp links. >>>> For example - I want to know the min bw of all links + cumulative bw of >>>> all links. That today I am afraid is not possible with link bandwidth >>>> extended community. >>>> >>>> Cheers, >>>> Robert >>>> >>>> >>>> On Fri, Sep 4, 2026 at 3:05 AM Jeff Tantsura <[email protected] >>>> <mailto:[email protected]>> wrote: >>>>> Robert, >>>>> >>>>> There’s at least 4 interworking implementations of >>>>> draft-ietf-bess-ebgp-dmz. >>>>> Cumulus, FRR, EOS, IOS-* , possibly more. >>>>> It is deployed in hyperscale networks as well as in AI clusters with >>>>> 10.000+ BGP speakers. >>>>> >>>>> Stephane - let’s take care of wiki. >>>>> >>>>> Thanks, >>>>> Jeff >>>>> >>>>>> On Aug 31, 2026, at 02:54, Robert Raszuk <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> HI, >>>>>> >>>>>> I indeed see cumulative bw, but do not see minimum bw. >>>>>> >>>>>> Btw I was commenting on RFC10005 not a BESS WG draft ... >>>>>> >>>>>> Honestly such cumulative or minimum should be described IMO in IDR >>>>>> RFC10005 as it is about what to encode. >>>>>> >>>>>> > (sigma bw of all paths and minimum) as implementations support it >>>>>> > already. >>>>>> Which implementations support it ? Is there draft-ietf-bess-ebgp-dmz >>>>>> implementation report wiki ? >>>>>> >>>>>> And for RFC10005 ( draft-ietf-idr-link-bandwidth) I do not see any >>>>>> implementation report on https://wiki.ietf.org/group/idr/implementations >>>>>> >>>>>> Thx, >>>>>> R. >>>>>> >>>>>> >>>>>> On Mon, Aug 31, 2026 at 11:41 AM Stephane Litkowski (slitkows) >>>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>>>> Hi Robert, >>>>>>> >>>>>>> >>>>>>> >>>>>>> If you look at draft-ietf-bess-ebgp-dmz which details how LBW extcom is >>>>>>> used today, it already defines both cases (sigma bw of all paths and >>>>>>> minimum) as implementations support it already. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: Robert Raszuk <[email protected] <mailto:[email protected]>> >>>>>>> Sent: Monday, August 31, 2026 11:28 AM >>>>>>> To: Stephane Litkowski (slitkows) <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> Cc: idr <[email protected] <mailto:[email protected]>> >>>>>>> Subject: Re: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to >>>>>>> 9/1/2026) >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Stephane, >>>>>>> >>>>>>> >>>>>>> >>>>>>> In respect to RFC10005 I do see a difference. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Today per RFC10005 we propagate bw of the interface over which best >>>>>>> path was received. However when you have say 32 or 64 paths it very >>>>>>> well may get down to rtr_id best path selection step and advertised bw >>>>>>> may be of that interface. >>>>>>> >>>>>>> >>>>>>> >>>>>>> This draft is asking to advertise the minimum bw of all paths. Is it >>>>>>> useful or not - I am not sure. I would actually prefer to see it >>>>>>> advertising sigma bw of all paths. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Now could we advertise it using the existing Link Bandwidth Extended >>>>>>> Community - of course. One page informational draft could propose that. >>>>>>> >>>>>>> >>>>>>> >>>>>>> - - - >>>>>>> >>>>>>> >>>>>>> >>>>>>> Now as I previously pointed out, fare draft still is trying to sell >>>>>>> itself as a way to use vanilla IP networks to construct real AI >>>>>>> training clusters. That's wrong .. very wrong .. Yes - it is very >>>>>>> attractive to this community, but does not make it technically sound. >>>>>>> It does not even fall under the "poor man's AI fabric" in my books. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Robert >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 31, 2026 at 10:09 AM Stephane Litkowski (slitkows) >>>>>>> <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hi WG, >>>>>>> >>>>>>> >>>>>>> I oppose to the adoption of this document as WG item. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Assumptions about RFC10005 are wrong (likely outdated): >>>>>>> RFC10005 supports transitive community so the restriction mentioned >>>>>>> does not apply >>>>>>> The draft mentions that link BW is to carry external link BW but the >>>>>>> actual definition from RFC has a wider scope: “The Link Bandwidth >>>>>>> Extended Community is defined as a BGP extended community that carries >>>>>>> the bandwidth information of a router (represented by BGP next hop) >>>>>>> that is connecting to a remote network. “. >>>>>>> It says that we can’t attach more that one LBW CT to a route which is >>>>>>> true, but doesn’t talk about the use case which may require more than >>>>>>> one. >>>>>>> >>>>>>> >>>>>>> I don’t like the EC to be called “Path Bandwidth”, by definition, the >>>>>>> existing RFC10005 community is already a path bandwidth as it is >>>>>>> attached to a BGP path. A BGP speaker advertising a bandwidth community >>>>>>> to its neighbor, tell to its neighbor about the BW available if this >>>>>>> neighbor uses the path advertised from this BGP speaker. As >>>>>>> demonstrated in some other email thread, the use case described is >>>>>>> already achieved for a long time by some implementation using the >>>>>>> existing community (Shraddha is mentioning it as well) . BTW the new >>>>>>> community proposed as the same weaknesses as the existing one so does >>>>>>> not bring any value. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Coming to Shraddha’s point about multiple level of loadbalancing, I >>>>>>> don’t think this is what the draft is talking about. The use case is >>>>>>> still the plain hop-by-hop BGP fabric. >>>>>>> >>>>>>> Considering a case where we have for instance a BGP service route which >>>>>>> NH is resolved over a BGP underlay route, there is no restriction today >>>>>>> that would prevent attaching a LBW com to the service route and another >>>>>>> one to the BGP underlay route so each level can have some different >>>>>>> UCMP behavior. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I’m still wondering reading the draft what is the problem to be solved >>>>>>> and why we can’t solve it using current mechanisms. Until this is fully >>>>>>> clear, I don’t see how we can adopt the draft. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Brgds, >>>>>>> >>>>>>> >>>>>>> >>>>>>> Stephane >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: Hegde, Shraddha <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> Sent: Thursday, August 27, 2026 11:01 AM >>>>>>> To: Zhangqingliang <[email protected] >>>>>>> <mailto:[email protected]>>; Susan Hares <[email protected] >>>>>>> <mailto:[email protected]>>; idr@ietf. org <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> Subject: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to >>>>>>> 9/1/2026) >>>>>>> >>>>>>> >>>>>>> >>>>>>> I support the WG adoption of the draft draft-xu-idr-fare >>>>>>> >>>>>>> I am not aware of any IPR on this. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I believe that ability to send multiple bandwidth communities is useful. >>>>>>> >>>>>>> In a traditional MPLS VPN network, there is a BGP overlay and IGP >>>>>>> underlay. The BGP carried link bandwidth is used to load-balance >>>>>>> traffic based on external bandwidth availability and the IGP bandwidth >>>>>>> based on internal network bandwidth availability so its possible to >>>>>>> apply >>>>>>> >>>>>>> two levels of bandwidth based load-balancing. >>>>>>> >>>>>>> >>>>>>> >>>>>>> In a DC network BGP plays the role of underlay transport. Many a times >>>>>>> the service prefixes >>>>>>> >>>>>>> are injected into the fabric in DC. Carrying two different bandwidth >>>>>>> values is useful to appropriately balance the traffic based on external >>>>>>> and internal bandwidth availability. >>>>>>> >>>>>>> >>>>>>> >>>>>>> draft-xu-idr-fare proposes path bandwidth community which is a good >>>>>>> starting point for the WG to work on. >>>>>>> >>>>>>> >>>>>>> >>>>>>> However, we should deliberate on >>>>>>> >>>>>>> how the RFC 10005 link bandwidth community and the path bandwidth >>>>>>> community interact. RFC 10005 is very generic, and it is possible to >>>>>>> advertise, minimum bandwidth or additive bandwidth or any other form of >>>>>>> bandwidth calculation when a nexthop change is done. Which means most >>>>>>> of the capabilities path bandwidth proposes to do is already available >>>>>>> with >>>>>>> >>>>>>> link bandwidth community and might have been deployed already by >>>>>>> operators. >>>>>>> >>>>>>> >>>>>>> >>>>>>> When both link bandwidth and path bandwidth communities are carried in >>>>>>> a prefix, which one carries what information is something that needs >>>>>>> to be specified in a backward compatible way given that RFC 10005 has >>>>>>> been implemented by many vendors and widely deployed. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Rgds >>>>>>> >>>>>>> Shraddha >>>>>>> >>>>>>> From: Zhangqingliang <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> Sent: 27 August 2026 12:15 >>>>>>> To: Susan Hares <[email protected] <mailto:[email protected]>>; idr@ietf. >>>>>>> org <[email protected] <mailto:[email protected]>> >>>>>>> Subject: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to >>>>>>> 9/1/2026) >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Sue, >>>>>>> >>>>>>> >>>>>>> >>>>>>> As a contributor, I support the WG adoption of this draft and am not >>>>>>> aware of any relevant IPR. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Best Regards, >>>>>>> >>>>>>> Qingliang Zhang >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: Susan Hares <[email protected] <mailto:[email protected]>> >>>>>>> Date: Tuesday, August 18, 2026 at 3:47 AM >>>>>>> To: idr@ietf. org <[email protected] <mailto:[email protected]>> >>>>>>> Subject: [Idr] WG adoption of draft-xu-idr-fare-08 (8/18/206 to >>>>>>> 9/1/2026) >>>>>>> >>>>>>> This begins a 2-week WG Adoption call for draft-xu-fare-08.txt from >>>>>>> 8/18 /2026 to 9/1/2026. >>>>>>> >>>>>>> >>>>>>> >>>>>>> The authors and contributors should reply to this email with an IPR >>>>>>> statement. >>>>>>> >>>>>>> >>>>>>> >>>>>>> The WG may wish to consider the following comments: >>>>>>> >>>>>>> >>>>>>> >>>>>>> 1) Is the use case described in draft-xu-fare-08 found >>>>>>> >>>>>>> AI training clusters in data centers? >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2) Does the Path Bandwidth Extended Community provide >>>>>>> >>>>>>> Reliable bandwidth allocation for the 3-stage and 5-stage >>>>>>> >>>>>>> CLOS environments? >>>>>>> >>>>>>> >>>>>>> >>>>>>> 3) Should this community be both transitive and non-transitive. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Cheerily, Sue Hares >>>>>>> >>>>>>> [EXTERNAL] >>>>>>> >>>>>>> ------------------------------------------------------------------------------------------------------------------------------------- >>>>>>> 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出的个人或群组。 >>>>>>> 禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。 >>>>>>> 如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ >>>>>>> This e-mail and its attachments contain confidential information from >>>>>>> New H3C, which is intended only for the person or entity whose address >>>>>>> is listed above. >>>>>>> Any use of the information contained herein in any way (including, but >>>>>>> not limited to, total or partial disclosure, reproduction, or >>>>>>> dissemination) by persons other than the intended recipient(s) is >>>>>>> prohibited. >>>>>>> If you receive this e-mail in error, please notify the sender by phone >>>>>>> or email immediately and delete it! >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Idr mailing list -- [email protected] <mailto:[email protected]> >>>>>>> To unsubscribe send an email to [email protected] >>>>>>> <mailto:[email protected]>_______________________________________________ >>>>>> Idr mailing list -- [email protected] <mailto:[email protected]> >>>>>> To unsubscribe send an email to [email protected] >>>>>> <mailto:[email protected]> >>>>> >>> > > _______________________________________________ > Idr mailing list -- [email protected] > To unsubscribe send an email to [email protected]
_______________________________________________ BESS mailing list -- [email protected] To unsubscribe send an email to [email protected]
