Hi Jeff, 5 topology hops is 5/3->66% worse than 3 hops for latency, reliability, and cost. Why do you assume 5 hops if the needed scale is possible to achieve by 3 hops? In fact, modern 1-ASIC switches (4.8Tbps, 51.2Tbps) permit 390k 100GE end-points for Megafly 3-hops topology (with 50% oversubscription) or 195k*100GE wire-speed. Warning: Megafly in general demands equal load distribution at the application level – this restriction always exists for full mesh instead of centralized boxes. PS: You said “stage” which probably means that the Server/Processor port is not included in the hops calculation. 3 topology hops with end-points are 5 hops overall. Eduard From: Lsr [mailto:[email protected]] On Behalf Of Jeff Tantsura Sent: Tuesday, November 28, 2023 2:32 AM To: Robert Raszuk <[email protected]> Cc: Acee Lindem <[email protected]>; Les Ginsberg (ginsberg) <[email protected]>; Tony Li <[email protected]>; [email protected]; [email protected] Subject: Re: [Lsr] New Version Notification for draft-xu-lsr-flooding-reduction-in-clos-01.txt
Robert, In context of LLM (10% of that for DLRM) training clusters, towards 2024/25 we would be looking to up to 8K end-points in a 3 stage leaf-spine fabric and up to 64-256K in 5 stages. Virtualization and how it is instantiated might significantly change amount/distribution of state in underlay/overlay. Obviously, these are hyperscale size deployments, many will be running 10-30 switches fabrics, where anything could work. BGP seems to work just fine, some data plane signaling could be used as a near real time augmentation to “slow but stable” control plane. Cheers, Jeff On Nov 26, 2023, at 14:30, Robert Raszuk <[email protected]<mailto:[email protected]>> wrote: Hey Jeff, Could you be so kind and defined term: "scaled-out leaf-spine fabrics" ? Specifically folks watching us here would highly appreciate if we state max target nodes with vanilla ISIS and max target nodes when your ISIS implementation supports draft-ietf-lsr-dynamic-flooding<https://datatracker.ietf.org/doc/html/draft-ietf-lsr-dynamic-flooding> While I am a BGP person I feel pretty strongly that BGP is not a best fit for the vast majority of DC fabrics in use today. Cheers, Robert On Sun, Nov 26, 2023 at 10:49 PM Jeff Tantsura <[email protected]<mailto:[email protected]>> wrote: I agree with all aforementioned comments. Wrt AI/ML networking - if a controller is used, what is required is link state exposure northbound and not link state protocol in the fabric. (I could argue for RIFT though ;-)) I’d urge you to take a look at Meta’s deployment in their ML clusters (publicly available) - they use BGP as the routing protocol to exchange reachability (and build ECMP sets) and provide a backup if controller computed next hop goes away/before new one has been computed. Open R is used northbound to expose the topology (in exactly same way - BGP-LS could be used). To summarize: an LS protocol brings no additional value in scaled-out leaf-spine fabrics, without significant modifications - it doesn’t work in irregular topologies such as DF, etc. Existing proposals - there are shipping implementations and experience in operating it, have proven their relative value in suitable deployments. Cheers, Jeff > On Nov 26, 2023, at 12:20, Acee Lindem > <[email protected]<mailto:[email protected]>> wrote: > > Speaking as WG member: > > I agree. The whole Data Center IGP flooding discussion went on years ago and > the simplistic enhancement proposed in the subject draft is neither relevant > or useful now. > > Thanks, > Acee > >> On Nov 24, 2023, at 11:55 PM, Les Ginsberg (ginsberg) >> <[email protected]<mailto:[email protected]>> >> wrote: >> >> Xiaohu – >> I also point out that there are at least two existing drafts which >> specifically address IS-IS flooding reduction in CLOS networks and do so in >> greater detail and with more robustness than what is in your draft: >> https://datatracker.ietf.org/doc/draft-ietf-lsr-distoptflood/ >> https://datatracker.ietf.org/doc/draft-ietf-lsr-isis-spine-leaf-ext/ >> I do not see a need for yet another draft specifically aimed at CLOS >> networks. >> Note that work on draft-ietf-lsr-isis-spine-leaf-ext was suspended due to >> lack of interest in deploying an IGP solution in CLOS networks. >> You are suggesting in draft-xu-lsr-fare that AI is going to change this. >> Well, maybe, but if so I think we should return to the solutions already >> available and prioritize work on them. >> Les >> From: Lsr <[email protected]<mailto:[email protected]>> On Behalf Of >> Tony Li >> Sent: Thursday, November 23, 2023 8:39 AM >> To: [email protected]<mailto:[email protected]> >> Cc: [email protected]<mailto:[email protected]> >> Subject: Re: [Lsr] New Version Notification for >> draft-xu-lsr-flooding-reduction-in-clos-01.txt >> Hi, >> What you’re proposing is already described in IS-IS Mesh Groups >> (https://www.rfc-editor.org/rfc/rfc2973.html) and improved upon in Dynamic >> Flooding >> (https://datatracker.ietf.org/doc/html/draft-ietf-lsr-dynamic-flooding). >> Regards, >> Tony >> >> >> On Nov 23, 2023, at 8:29 AM, >> [email protected]<mailto:[email protected]> wrote: >> Hi all, >> Any comments or suggestions are welcome. >> Best regards, >> Xiaohu >> 发件人: [email protected]<mailto:[email protected]> >> <[email protected]<mailto:[email protected]>> >> 日期: 星期三, 2023年11月22日 11:37 >> 收件人: Xiaohu Xu <[email protected]<mailto:[email protected]>> >> 主题: New Version Notification for >> draft-xu-lsr-flooding-reduction-in-clos-01.txt >> A new version of Internet-Draft >> draft-xu-lsr-flooding-reduction-in-clos-01.txt >> has been successfully submitted by Xiaohu Xu and posted to the >> IETF repository. >> >> Name: draft-xu-lsr-flooding-reduction-in-clos >> Revision: 01 >> Title: Flooding Reduction in CLOS Networks >> Date: 2023-11-22 >> Group: Individual Submission >> Pages: 6 >> URL: >> https://www.ietf.org/archive/id/draft-xu-lsr-flooding-reduction-in-clos-01.txt >> Status: >> https://datatracker.ietf.org/doc/draft-xu-lsr-flooding-reduction-in-clos/ >> HTMLized: >> https://datatracker.ietf.org/doc/html/draft-xu-lsr-flooding-reduction-in-clos >> Diff: >> https://author-tools.ietf.org/iddiff?url2=draft-xu-lsr-flooding-reduction-in-clos-01 >> >> Abstract: >> >> In a CLOS topology, an OSPF (or ISIS) router may receive identical >> copies of an LSA (or LSP) from multiple OSPF (or ISIS) neighbors. >> Moreover, two OSPF (or ISIS) neighbors may exchange the same LSA (or >> LSP) simultaneously. This results in unnecessary flooding of link- >> state information, which wastes the precious resources of OSPF (or >> ISIS) routers. Therefore, this document proposes extensions to OSPF >> (or ISIS) to reduce this flooding within CLOS networks. The >> reduction of OSPF (or ISIS) flooding is highly beneficial for >> improving the scalability of CLOS networks. >> >> >> >> The IETF Secretariat >> >> _______________________________________________ >> Lsr mailing list >> [email protected]<mailto:[email protected]> >> https://www.ietf.org/mailman/listinfo/lsr >> _______________________________________________ >> Lsr mailing list >> [email protected]<mailto:[email protected]> >> https://www.ietf.org/mailman/listinfo/lsr > > > _______________________________________________ > Lsr mailing list > [email protected]<mailto:[email protected]> > https://www.ietf.org/mailman/listinfo/lsr _______________________________________________ Lsr mailing list [email protected]<mailto:[email protected]> https://www.ietf.org/mailman/listinfo/lsr
_______________________________________________ Lsr mailing list [email protected] https://www.ietf.org/mailman/listinfo/lsr
