RE: [WISPA] Optimally taking advantage of GB Ethernet
Hi Tom, Well that's where I disagree. And where I am looking for clarification. We were NOT using systems with Buses limited to 32bit/33Mhz or Basic PCI. The routers that we used for testing were using the Intel 7501 chipset, spec'd at allowing 3GB of throughput, and PCI-X Bus. There are actually three potential limits... 1) Hardware Depending on the Ethernet card, this is still a real possibility. 2) Software This is definitely a limitation. 3) Ethernet Protocol theoretical limit. My understanding, although not confirmed, was that there was an Ethernet (cdma) timing Limitation, that prevented the Full 1GB from being reached using a 1500MTU. There is no such limitation on Ethernet. I'm assuming that when you say 1500 MTU you mean 1500 byte packets and not 1500 MTU. In any case, a quick Google search can confirm that equipment does exist that passes GigE at wirespeed. I remember Network World magazine doing tests back in 1999 or 2000 and passing wirespeed GigE over a variety of equipment with an array of packet sizes. Myhardware can handle it. I'm investigating whether the limit is related to my Linux Software config versus Ethernet theoretical limits. Given our experiences, I know that the former is affecting you. The latter limits you mention do not exist. If you want wirespeed GigE performance with multiple packet streams at a more reasonable packet size (remember that 1500 byte packets aren't realistic), you'll need to make some adjustments to the kernel. Well, thats my point... 1500MTU is a requirement that is usually beyond the control of the ISP. The ISP may may control the GB connection for Higher MTU, but NOT necessarily backend Transit or Front End last mile connections. The Internet is full of less than 384K average size packets, and not much an ISP can do about that. Part of the question becomes, can near 1GB be acheived at 1500MTU? Yes, without question. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. Actually, it is. Utilities top and vmstat don't necessarily reflect all of the CPU utilization, and can't account for PCI bus contention/overhead. Excellent Point. Any way to tell that (FULL CPU utilization)? You would need a debugger to see the amount of time that the CPU was waiting for various operations. That would be the only way that I know to ferret out everything. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. Well, sure...you've just taken your packet rate down significantly. The reason we thought this was an Ethernet limitation and not a CPU/hardware limitation is that, we were able to pass a larger speed by combining multiple 100 mbps connections. For example, we were able to get 400 mbps with 4- 100mbps connections simultaneously.(didn't have more machines to test.) You've changed too many variables at once (drivers, hardware) to make any conclusions. Does ImageStream have any data, from live tests, proving the speed they can get across their GB routers at 1500 MTU? Again, I'm assuming that you mean packet size when you're saying MTU and not actually setting the MTU to less than 1500 bytes. We have tested our GigE performance on routers with GigE wirespeed specs to ensure that we can pass 1 Gbps at various packet sizes. Regards, Jeff Tom DeReggi Regards, Jeff Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Optimally taking advantage of GB Ethernet
Jeff, Thanks for the feedback, to steer me in the right direction. PS. The good thing about software and configuration problems is that... they can be fixed :-) Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Jeff Broadwick [EMAIL PROTECTED] To: 'WISPA General List' wireless@wispa.org Sent: Tuesday, January 09, 2007 1:46 PM Subject: RE: [WISPA] Optimally taking advantage of GB Ethernet Hi Tom, Well that's where I disagree. And where I am looking for clarification. We were NOT using systems with Buses limited to 32bit/33Mhz or Basic PCI. The routers that we used for testing were using the Intel 7501 chipset, spec'd at allowing 3GB of throughput, and PCI-X Bus. There are actually three potential limits... 1) Hardware Depending on the Ethernet card, this is still a real possibility. 2) Software This is definitely a limitation. 3) Ethernet Protocol theoretical limit. My understanding, although not confirmed, was that there was an Ethernet (cdma) timing Limitation, that prevented the Full 1GB from being reached using a 1500MTU. There is no such limitation on Ethernet. I'm assuming that when you say 1500 MTU you mean 1500 byte packets and not 1500 MTU. In any case, a quick Google search can confirm that equipment does exist that passes GigE at wirespeed. I remember Network World magazine doing tests back in 1999 or 2000 and passing wirespeed GigE over a variety of equipment with an array of packet sizes. Myhardware can handle it. I'm investigating whether the limit is related to my Linux Software config versus Ethernet theoretical limits. Given our experiences, I know that the former is affecting you. The latter limits you mention do not exist. If you want wirespeed GigE performance with multiple packet streams at a more reasonable packet size (remember that 1500 byte packets aren't realistic), you'll need to make some adjustments to the kernel. Well, thats my point... 1500MTU is a requirement that is usually beyond the control of the ISP. The ISP may may control the GB connection for Higher MTU, but NOT necessarily backend Transit or Front End last mile connections. The Internet is full of less than 384K average size packets, and not much an ISP can do about that. Part of the question becomes, can near 1GB be acheived at 1500MTU? Yes, without question. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. Actually, it is. Utilities top and vmstat don't necessarily reflect all of the CPU utilization, and can't account for PCI bus contention/overhead. Excellent Point. Any way to tell that (FULL CPU utilization)? You would need a debugger to see the amount of time that the CPU was waiting for various operations. That would be the only way that I know to ferret out everything. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. Well, sure...you've just taken your packet rate down significantly. The reason we thought this was an Ethernet limitation and not a CPU/hardware limitation is that, we were able to pass a larger speed by combining multiple 100 mbps connections. For example, we were able to get 400 mbps with 4- 100mbps connections simultaneously.(didn't have more machines to test.) You've changed too many variables at once (drivers, hardware) to make any conclusions. Does ImageStream have any data, from live tests, proving the speed they can get across their GB routers at 1500 MTU? Again, I'm assuming that you mean packet size when you're saying MTU and not actually setting the MTU to less than 1500 bytes. We have tested our GigE performance on routers with GigE wirespeed specs to ensure that we can pass 1 Gbps at various packet sizes. Regards, Jeff Tom DeReggi Regards, Jeff Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Optimally taking advantage of GB Ethernet
1GB Fiber is typically using Packet over SONET. I have typically seen over 500 mbps on these links, even from Telcove/L3. But the distance is usually a factor. Running from NoLa to ATL on a long-haul, single hop 1GB fiber run introduces lots of regen issues, so 500 mbps would be a blessing. Most people don't need as much pipe as they buy either. - Peter Tom DeReggi wrote: I do not believe that people like ATTT are passing over 200mbps on their GB Ethernet fiber links, when they are using them as backbones or extensions to existing customer's connection, for the reason I brought up. I just don't think that the end user custoemr base is smart enough to know the difference. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Optimally taking advantage of GB Ethernet
Jeff, The solution is simply a hardware upgrade for starters. A 32-bit/33 MHz bus will top out around 200 Mbps. If you look for a bus with higher speed slots, you can triple your throughput without adjusting ANYTHING in your Linux kernel. Well thats where I disagree. And where I am looking for clarification. We were NOT using systems with Buses limited to 32bit/33Mhz or Basic PCI. The routers that we used for testing were using the Intel 7501 chipset, spec'd at allowing 3GB of throughput, and PCI-X Bus. There are actually three potential limits... 1) Hardware 2) Software 3) Ethernet Protocol theoretical limit. My understanding, although not confirmed, was that there was an Ethernet (cdma) timing Limitation, that prevented the Full 1GB from being reached using a 1500MTU. Myhardware can handle it. I'm investigating whether the limit is related to my Linux Software config versus Ethernet theoretical limits. If you want wirespeed GigE performance with multiple packet streams at a more reasonable packet size (remember that 1500 byte packets aren't realistic), you'll need to make some adjustments to the kernel. Well, thats my point... 1500MTU is a requirement that is usually beyond the control of the ISP. The ISP may may control the GB connection for Higher MTU, but NOT necessarily backend Transit or Front End last mile connections. The Internet is full of less than 384K average size packets, and not much an ISP can do about that. Part of the question becomes, can near 1GB be acheived at 1500MTU? With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. Actually, it is. Utilities top and vmstat don't necessarily reflect all of the CPU utilization, and can't account for PCI bus contention/overhead. Excellent Point. Any way to tell that (FULL CPU utilization)? The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. Well, sure...you've just taken your packet rate down significantly. The reason we thought this was an Ethernet limitation and not a CPU/hardware limitation is that, we were able to pass a larger speed by combining multiple 100 mbps connections. For example, we were able to get 400 mbps with 4- 100mbps connections simultaneously.(didn't have more machines to test.) You still have bus overhead and the limitations of a 32/33 bus (1 Gbps burst capacity). Again, we were using PCI-X not limited in that way, to my understanding. Does Image Stream have any data, from live tests, proving the speed they can get across their GB routers at 1500 MTU? Maybe thats a solution for us? Tom DeReggi Regards, Jeff Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Optimally taking advantage of GB Ethernet
Understand a major difference ATT for example sells GB fiber for $8000 per month. But they are selling layers 2 end to end to the subsciber. So because GB fiber is the customers first HOP, the customer's MTU can adjust to 9600MTU jumbo frames. For example if the customer has a GB switch on premise, they are already using Jumbo frames, and easy to interface to GB Ethernet WAN connection. The other LECs doing GB fiber are often using something other than Ethernet, such as Sonet, ATM, or whatever. There may be something there taht deals with it. The problem I brought up is that ISPs hookkup the customer's initial connection with less than 100mbps which is NOT Jumbo frames. The reason is that most Ethernet Fiber/CAT5 (100mbps) equipment does not allow over 1500MTU, and only a few equipment manufacturers even support allowing allow around 1540 MTU to supprot things like MPLS and VLANs. I do not believe that people like ATTT are passing over 200mbps on their GB Ethernet fiber links, when they are using them as backbones or extensions to existing customer's connection, for the reason I brought up. I just don't think that the end user custoemr base is smart enough to know the difference. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Travis Johnson [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Saturday, January 06, 2007 11:42 PM Subject: Re: [WISPA] Optimally taking advantage of GB Ethernet Tom, How are the big boys doing it? Surely ATT and others are transporting more than 200Mbps across their 1GB fiber links. Travis Microserv Tom DeReggi wrote: Gigabit Ethernet, can pass 1 gbps when it uses greater than a 9600 MTU frame. But with a 1500MTU frame, it can barely pass 200 mbps. The problem is that most Internet and subscriber traffic is using a 1500MTU or smaller frame. So in theory, its would be just as efficient and fast to bond two 100 mbps fiber connections than it would to buy 1- 1GB fiber connection. So the question is How do we most efficiently use 1GB fiber to get the advantage of the full 1GB of capacity? Do we need to use some sort of packet agreegation/stuffing technology? Is GB etherner pointless for Internet transit backbones? Is GB just good for high capacity Transports, recognizing that routers will likely split traffic to different smaller bandwidth peers? Is there a special router or router feature used to solve this problem? Is that method available to Linux? The reason I ask is several fold. In a network design where all traffic flows to a single source (for example many 100mbps baclhauls to remote areas to 1 central data center), it would be beneficial because the cost of 1 big 1GB pipe could be shared to deliver capacity to everything, better apt to handle peak traffic and get higher oversubscription rates. However, if teh GB INternet pipe can not be efficiently used, this method would be severally flawed. It might be better to have multipel 100mvps transit connections spread out across one's network, so there was a shorter path to transit, and the network's bandwdith spread out amungst multiple 100mbps transit connection, for better over all throughput. In other words, in a 10 city network, 1- 100mbps pipe in each of teh 10 cities would allow a full combined 1 gbps of Internet transit, where as agregating 100mbps from each city to one central source where their was a single 1GB transit, would result in only a 200mbps throughput, assuming traffic was delivered to it as a 1500 MTU. Any feedback? Take note that my comment that a 1500MTU frame 1 Gbps Ethernet card could only pass 200kbps was based on some lab tests. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. These tests were replicated going PC to PC (no switch) and with a high end SMC GB switch in-line. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Optimally taking advantage of GB Ethernet
Yes we also can psuh 800 mbps on a GB link Miktoik router to Mikrotik Router. But the test initiates on a Jumbo frame device and end on a jumbo frame device. Now try this test Connect 4 computers each to its own 100mbps switch (support only 1500mtu), Then take the 4 switches and plug into 100m/1000gb switch, then plug that switch (9600MTU) to the end router on a GB ethernet port. Do a simultaneous test from all 4 pcs to the end GB router, and see what you get. I bet you'll find that the agreegate throughput is around 200mbps FDX. Linux on most Ethernet ports will auto adjust its MTU, so testing in a lab router to router may not show desired results as the testing PC will start with Jumbo frames from the beginning. .Checking the router Interfaces show a 1500MTU setting Because it is set to 1500MTU, does not necessarilly mean that it is pushing only 1500 MTU. Many ethernet drivers are configured to allow larger size packets to pass. I won;t try and try to explain that situation because I will get it wrong. The customer;s traffic is almost always using a 1500 MTU. so 1500 byte packets or smaller is what will pass across any backbone transport links. The gear must be capable of pushing the 1500 MTU packets at full speed. For what ever reason it usually is NOT possible. Most GB gear will only push full capacity when pushing Jumbo 9600 or greater packets. Unless there is some sort of trunking mode that agreegates the packets. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Brad Belton [EMAIL PROTECTED] To: 'WISPA General List' wireless@wispa.org Sent: Saturday, January 06, 2007 11:50 PM Subject: RE: [WISPA] Optimally taking advantage of GB Ethernet Hello Tom, First let me saydamn Cowboys...sigh I'm not sure I follow exactly what you are saying, but we have pushed better than 800Mbps HDX and more than 700Mbps FDX aggregate between GigE MT routers. Checking the router Interfaces show a 1500MTU setting. Is that what you are talking about? Brad -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Travis Johnson Sent: Saturday, January 06, 2007 10:43 PM To: WISPA General List Subject: Re: [WISPA] Optimally taking advantage of GB Ethernet Tom, How are the big boys doing it? Surely ATT and others are transporting more than 200Mbps across their 1GB fiber links. Travis Microserv Tom DeReggi wrote: Gigabit Ethernet, can pass 1 gbps when it uses greater than a 9600 MTU frame. But with a 1500MTU frame, it can barely pass 200 mbps. The problem is that most Internet and subscriber traffic is using a 1500MTU or smaller frame. So in theory, its would be just as efficient and fast to bond two 100 mbps fiber connections than it would to buy 1- 1GB fiber connection. So the question is How do we most efficiently use 1GB fiber to get the advantage of the full 1GB of capacity? Do we need to use some sort of packet agreegation/stuffing technology? Is GB etherner pointless for Internet transit backbones? Is GB just good for high capacity Transports, recognizing that routers will likely split traffic to different smaller bandwidth peers? Is there a special router or router feature used to solve this problem? Is that method available to Linux? The reason I ask is several fold. In a network design where all traffic flows to a single source (for example many 100mbps baclhauls to remote areas to 1 central data center), it would be beneficial because the cost of 1 big 1GB pipe could be shared to deliver capacity to everything, better apt to handle peak traffic and get higher oversubscription rates. However, if teh GB INternet pipe can not be efficiently used, this method would be severally flawed. It might be better to have multipel 100mvps transit connections spread out across one's network, so there was a shorter path to transit, and the network's bandwdith spread out amungst multiple 100mbps transit connection, for better over all throughput. In other words, in a 10 city network, 1- 100mbps pipe in each of teh 10 cities would allow a full combined 1 gbps of Internet transit, where as agregating 100mbps from each city to one central source where their was a single 1GB transit, would result in only a 200mbps throughput, assuming traffic was delivered to it as a 1500 MTU. Any feedback? Take note that my comment that a 1500MTU frame 1 Gbps Ethernet card could only pass 200kbps was based on some lab tests. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. These tests were replicated going PC to PC (no switch) and with a high end SMC GB switch in-line. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org
Re: [WISPA] Optimally taking advantage of GB Ethernet
Take note that this problem that I am discussing with GB ethernet also may apply to GB wireless. I'm trying to determine if GB wireless needs to be the full path to be useful, or if its an adequate backhaul in specific areas. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Tom DeReggi [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Sunday, January 07, 2007 8:16 PM Subject: Re: [WISPA] Optimally taking advantage of GB Ethernet Understand a major difference ATT for example sells GB fiber for $8000 per month. But they are selling layers 2 end to end to the subsciber. So because GB fiber is the customers first HOP, the customer's MTU can adjust to 9600MTU jumbo frames. For example if the customer has a GB switch on premise, they are already using Jumbo frames, and easy to interface to GB Ethernet WAN connection. The other LECs doing GB fiber are often using something other than Ethernet, such as Sonet, ATM, or whatever. There may be something there taht deals with it. The problem I brought up is that ISPs hookkup the customer's initial connection with less than 100mbps which is NOT Jumbo frames. The reason is that most Ethernet Fiber/CAT5 (100mbps) equipment does not allow over 1500MTU, and only a few equipment manufacturers even support allowing allow around 1540 MTU to supprot things like MPLS and VLANs. I do not believe that people like ATTT are passing over 200mbps on their GB Ethernet fiber links, when they are using them as backbones or extensions to existing customer's connection, for the reason I brought up. I just don't think that the end user custoemr base is smart enough to know the difference. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband - Original Message - From: Travis Johnson [EMAIL PROTECTED] To: WISPA General List wireless@wispa.org Sent: Saturday, January 06, 2007 11:42 PM Subject: Re: [WISPA] Optimally taking advantage of GB Ethernet Tom, How are the big boys doing it? Surely ATT and others are transporting more than 200Mbps across their 1GB fiber links. Travis Microserv Tom DeReggi wrote: Gigabit Ethernet, can pass 1 gbps when it uses greater than a 9600 MTU frame. But with a 1500MTU frame, it can barely pass 200 mbps. The problem is that most Internet and subscriber traffic is using a 1500MTU or smaller frame. So in theory, its would be just as efficient and fast to bond two 100 mbps fiber connections than it would to buy 1- 1GB fiber connection. So the question is How do we most efficiently use 1GB fiber to get the advantage of the full 1GB of capacity? Do we need to use some sort of packet agreegation/stuffing technology? Is GB etherner pointless for Internet transit backbones? Is GB just good for high capacity Transports, recognizing that routers will likely split traffic to different smaller bandwidth peers? Is there a special router or router feature used to solve this problem? Is that method available to Linux? The reason I ask is several fold. In a network design where all traffic flows to a single source (for example many 100mbps baclhauls to remote areas to 1 central data center), it would be beneficial because the cost of 1 big 1GB pipe could be shared to deliver capacity to everything, better apt to handle peak traffic and get higher oversubscription rates. However, if teh GB INternet pipe can not be efficiently used, this method would be severally flawed. It might be better to have multipel 100mvps transit connections spread out across one's network, so there was a shorter path to transit, and the network's bandwdith spread out amungst multiple 100mbps transit connection, for better over all throughput. In other words, in a 10 city network, 1- 100mbps pipe in each of teh 10 cities would allow a full combined 1 gbps of Internet transit, where as agregating 100mbps from each city to one central source where their was a single 1GB transit, would result in only a 200mbps throughput, assuming traffic was delivered to it as a 1500 MTU. Any feedback? Take note that my comment that a 1500MTU frame 1 Gbps Ethernet card could only pass 200kbps was based on some lab tests. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. These tests were replicated going PC to PC (no switch) and with a high end SMC GB switch in-line. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives
[WISPA] Optimally taking advantage of GB Ethernet
Gigabit Ethernet, can pass 1 gbps when it uses greater than a 9600 MTU frame. But with a 1500MTU frame, it can barely pass 200 mbps. The problem is that most Internet and subscriber traffic is using a 1500MTU or smaller frame. So in theory, its would be just as efficient and fast to bond two 100 mbps fiber connections than it would to buy 1- 1GB fiber connection. So the question is How do we most efficiently use 1GB fiber to get the advantage of the full 1GB of capacity? Do we need to use some sort of packet agreegation/stuffing technology? Is GB etherner pointless for Internet transit backbones? Is GB just good for high capacity Transports, recognizing that routers will likely split traffic to different smaller bandwidth peers? Is there a special router or router feature used to solve this problem? Is that method available to Linux? The reason I ask is several fold. In a network design where all traffic flows to a single source (for example many 100mbps baclhauls to remote areas to 1 central data center), it would be beneficial because the cost of 1 big 1GB pipe could be shared to deliver capacity to everything, better apt to handle peak traffic and get higher oversubscription rates. However, if teh GB INternet pipe can not be efficiently used, this method would be severally flawed. It might be better to have multipel 100mvps transit connections spread out across one's network, so there was a shorter path to transit, and the network's bandwdith spread out amungst multiple 100mbps transit connection, for better over all throughput. In other words, in a 10 city network, 1- 100mbps pipe in each of teh 10 cities would allow a full combined 1 gbps of Internet transit, where as agregating 100mbps from each city to one central source where their was a single 1GB transit, would result in only a 200mbps throughput, assuming traffic was delivered to it as a 1500 MTU. Any feedback? Take note that my comment that a 1500MTU frame 1 Gbps Ethernet card could only pass 200kbps was based on some lab tests. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. These tests were replicated going PC to PC (no switch) and with a high end SMC GB switch in-line. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
Re: [WISPA] Optimally taking advantage of GB Ethernet
Tom, How are the big boys doing it? Surely ATT and others are transporting more than 200Mbps across their 1GB fiber links. Travis Microserv Tom DeReggi wrote: Gigabit Ethernet, can pass 1 gbps when it uses greater than a 9600 MTU frame. But with a 1500MTU frame, it can barely pass 200 mbps. The problem is that most Internet and subscriber traffic is using a 1500MTU or smaller frame. So in theory, its would be just as efficient and fast to bond two 100 mbps fiber connections than it would to buy 1- 1GB fiber connection. So the question is How do we most efficiently use 1GB fiber to get the advantage of the full 1GB of capacity? Do we need to use some sort of packet agreegation/stuffing technology? Is GB etherner pointless for Internet transit backbones? Is GB just good for high capacity Transports, recognizing that routers will likely split traffic to different smaller bandwidth peers? Is there a special router or router feature used to solve this problem? Is that method available to Linux? The reason I ask is several fold. In a network design where all traffic flows to a single source (for example many 100mbps baclhauls to remote areas to 1 central data center), it would be beneficial because the cost of 1 big 1GB pipe could be shared to deliver capacity to everything, better apt to handle peak traffic and get higher oversubscription rates. However, if teh GB INternet pipe can not be efficiently used, this method would be severally flawed. It might be better to have multipel 100mvps transit connections spread out across one's network, so there was a shorter path to transit, and the network's bandwdith spread out amungst multiple 100mbps transit connection, for better over all throughput. In other words, in a 10 city network, 1- 100mbps pipe in each of teh 10 cities would allow a full combined 1 gbps of Internet transit, where as agregating 100mbps from each city to one central source where their was a single 1GB transit, would result in only a 200mbps throughput, assuming traffic was delivered to it as a 1500 MTU. Any feedback? Take note that my comment that a 1500MTU frame 1 Gbps Ethernet card could only pass 200kbps was based on some lab tests. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. These tests were replicated going PC to PC (no switch) and with a high end SMC GB switch in-line. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
RE: [WISPA] Optimally taking advantage of GB Ethernet
Hello Tom, First let me saydamn Cowboys...sigh I'm not sure I follow exactly what you are saying, but we have pushed better than 800Mbps HDX and more than 700Mbps FDX aggregate between GigE MT routers. Checking the router Interfaces show a 1500MTU setting. Is that what you are talking about? Brad -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Travis Johnson Sent: Saturday, January 06, 2007 10:43 PM To: WISPA General List Subject: Re: [WISPA] Optimally taking advantage of GB Ethernet Tom, How are the big boys doing it? Surely ATT and others are transporting more than 200Mbps across their 1GB fiber links. Travis Microserv Tom DeReggi wrote: Gigabit Ethernet, can pass 1 gbps when it uses greater than a 9600 MTU frame. But with a 1500MTU frame, it can barely pass 200 mbps. The problem is that most Internet and subscriber traffic is using a 1500MTU or smaller frame. So in theory, its would be just as efficient and fast to bond two 100 mbps fiber connections than it would to buy 1- 1GB fiber connection. So the question is How do we most efficiently use 1GB fiber to get the advantage of the full 1GB of capacity? Do we need to use some sort of packet agreegation/stuffing technology? Is GB etherner pointless for Internet transit backbones? Is GB just good for high capacity Transports, recognizing that routers will likely split traffic to different smaller bandwidth peers? Is there a special router or router feature used to solve this problem? Is that method available to Linux? The reason I ask is several fold. In a network design where all traffic flows to a single source (for example many 100mbps baclhauls to remote areas to 1 central data center), it would be beneficial because the cost of 1 big 1GB pipe could be shared to deliver capacity to everything, better apt to handle peak traffic and get higher oversubscription rates. However, if teh GB INternet pipe can not be efficiently used, this method would be severally flawed. It might be better to have multipel 100mvps transit connections spread out across one's network, so there was a shorter path to transit, and the network's bandwdith spread out amungst multiple 100mbps transit connection, for better over all throughput. In other words, in a 10 city network, 1- 100mbps pipe in each of teh 10 cities would allow a full combined 1 gbps of Internet transit, where as agregating 100mbps from each city to one central source where their was a single 1GB transit, would result in only a 200mbps throughput, assuming traffic was delivered to it as a 1500 MTU. Any feedback? Take note that my comment that a 1500MTU frame 1 Gbps Ethernet card could only pass 200kbps was based on some lab tests. With the 1500MTU frame acheiving only 200kbps, our routers CPU utilization was less than 20%, so it was not a saturated router. The second we changed MTU to 9600, we got over 800 mbps, and CPU utilization was still very low, forget exact number but under 40%. These tests were replicated going PC to PC (no switch) and with a high end SMC GB switch in-line. Tom DeReggi RapidDSL Wireless, Inc IntAirNet- Fixed Wireless Broadband -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/ -- WISPA Wireless List: wireless@wispa.org Subscribe/Unsubscribe: http://lists.wispa.org/mailman/listinfo/wireless Archives: http://lists.wispa.org/pipermail/wireless/
