Re: [CentOS] Linux Kernel Physical Interface Limit
On 7/14/10, William Warren hescomins...@emmanuelcomputerconsulting.com wrote: On 7/14/2010 1:16 AM, Emmanuel Noobadmin wrote: Googling up a really old 2005 newsgroup thread says some people had 24 physical NIC (6x 4quad) in a system before and one person vaguely remembers a hard 256 limit which would make sense if physical interface count is a byte value. can you give me a link to that thread? My googling skills are apparently not up to snuff to find that..:) 2001 poster says kernel 2.4 = 16 at least http://groups.google.com/group/comp.os.linux.hardware/browse_thread/thread/fbb020bfcc0abd/f25e760b552c9246?pli=1 2005 poster says he's seen 6x quad in a Linux server http://linux.derkeiler.com/Newsgroups/comp.os.linux.hardware/2005-01/0431.html ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
-Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Eero Volotinen Sent: Tuesday, July 13, 2010 8:58 PM To: CentOS mailing list Subject: Re: [CentOS] Linux Kernel Physical Interface Limit 2010/7/14 William Warren hescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? 13x4 = 52 ether chips is possible with COTS hardware. *** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept for the presence of computer viruses. www.Hubbell.com - Hubbell Incorporated** ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
- Brian T. Brunner bbrun...@gai-tronics.com wrote: -Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Eero Volotinen Sent: Tuesday, July 13, 2010 8:58 PM To: CentOS mailing list Subject: Re: [CentOS] Linux Kernel Physical Interface Limit 2010/7/14 William Warren hescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? 13x4 = 52 ether chips is possible with COTS hardware. Is there some sort of reference for this information? Where does '13x4' come from? --Tim ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On 7/14/2010 1:16 AM, Emmanuel Noobadmin wrote: On 7/14/10, William Warrenhescomins...@emmanuelcomputerconsulting.com wrote: ok let me specify. Is there any real limit? I've seen some folks(and been told by a few) that you can't more than 10 physical interfaces in a linux system. Googling up a really old 2005 newsgroup thread says some people had 24 physical NIC (6x 4quad) in a system before and one person vaguely remembers a hard 256 limit which would make sense if physical interface count is a byte value. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos can you give me a link to that thread? My googling skills are apparently not up to snuff to find that..:) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
- Brian T. Brunner bbrun...@gai-tronics.com wrote: -Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Eero Volotinen Sent: Tuesday, July 13, 2010 8:58 PM To: CentOS mailing list Subject: Re: [CentOS] Linux Kernel Physical Interface Limit 2010/7/14 William Warren hescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? 13x4 = 52 ether chips is possible with COTS hardware. Even if the limit were lower, such as 10 physical interfaces as mentioned before, I have to imagine that the host system would have issues dealing with the number of interrupts needed to *PROPERLY* service all of those interfaces in addition to the other system hardware. They may all work, but your performance may be horrible. In that case, I would think one or two 10Gbe interfaces to a nice switch and the necessary VLANs would perform nicely. Just my $0.02 USD. --Tim ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
Start with a 4 port PCI ethercard http://www.cyberresearch.com/store/industrial-computers-rugged-pcs/industrial-computer-accessories-pc-peripherals/bus-adapters-expansion-chassis/bus-adapters-expansion-chassis-pci/PRO_3212_8873.2.htm Expand a PCIex1 slot into several PCI slots, claiming 13 PCI slots possible. I haven't tried to assure this is the max possible. http://www.cyberresearch.com/store/industrial-computers-rugged-pcs/industrial-computer-accessories-pc-peripherals/bus-adapters-expansion-chassis/bus-adapters-expansion-chassis-pci/PRO_3212_8873.2.htm 10 PCI slots and a PCIEx16 makes(?) 14 slots http://www.cyberresearch.com/store/industrial-computers-rugged-pcs/industrial-computer-accessories-pc-peripherals/bus-adapters-expansion-chassis/bus-adapters-expansion-chassis-pci/PRO_3212_8873.2.htm so 13 or 14 X 4 makes 52 or 56 etherchips on one system with COTS. 7 PCIeX16 slots (http://www.evga.com/articles/00501/) @PCIE-4PCI = 28 PCI slots; @x4 chips/slot is 112 (+2 on the mobo) is 1114 interfaces. Useful? Not my call. Silly enough that somebody will try? Probably. -Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Tim Nelson Sent: Wednesday, July 14, 2010 10:35 AM To: CentOS mailing list Subject: Re: [CentOS] Linux Kernel Physical Interface Limit - Brian T. Brunner bbrun...@gai-tronics.com wrote: -Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Eero Volotinen Sent: Tuesday, July 13, 2010 8:58 PM To: CentOS mailing list Subject: Re: [CentOS] Linux Kernel Physical Interface Limit 2010/7/14 William Warren hescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? 13x4 = 52 ether chips is possible with COTS hardware. Is there some sort of reference for this information? Where does '13x4' come from? --Tim ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos *** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept for the presence of computer viruses. www.Hubbell.com - Hubbell Incorporated** ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On Wed, Jul 14, 2010 at 09:51:51AM -0500, Tim Nelson wrote: Even if the limit were lower, such as 10 physical interfaces as mentioned before, I have to imagine that the host system would have issues dealing with the number of interrupts needed to *PROPERLY* service all of those interfaces in addition to the other system hardware. There may (or may not) be another problem. As of a couple of years ago, on some Linux variants (didn't try RHEL/CentOS), I was having trouble even getting 6 NICs (on 3 cards) to work if I had IPv6 turned on. 4 NICs worked fine. Filed some bug reports, and it was evident from the response that very, very few Linux users ever go 4 eth's on a system. Thus the lack of properly debugged IPv6 support for that then. Fortunately I don't (yet) need IPv6. When I do, it'll be curious to see if the bug is still there. Whit ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On 07/14/2010 08:17 AM, Whit Blauvelt wrote: On Wed, Jul 14, 2010 at 09:51:51AM -0500, Tim Nelson wrote: Even if the limit were lower, such as 10 physical interfaces as mentioned before, I have to imagine that the host system would have issues dealing with the number of interrupts needed to *PROPERLY* service all of those interfaces in addition to the other system hardware. There may (or may not) be another problem. As of a couple of years ago, on some Linux variants (didn't try RHEL/CentOS), I was having trouble even getting 6 NICs (on 3 cards) to work if I had IPv6 turned on. 4 NICs worked fine. Filed some bug reports, and it was evident from the response that very, very few Linux users ever go 4 eth's on a system. Thus the lack of properly debugged IPv6 support for that then. Fortunately I don't (yet) need IPv6. When I do, it'll be curious to see if the bug is still there. I've got six machines with 6 Gb interfaces (two on motherboard, 4 on a card) right now (the design called for 3 bonded pairs on separate nets for redundancy). I haven't tried IPV6 on them. I had 'issues' with bonding and VMs though. -- Benjamin Franz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On Wed, Jul 14, 2010 at 08:52:02AM -0700, Benjamin Franz wrote: Filed some bug reports, and it was evident from the response that very, very few Linux users ever go 4 eth's on a system. Thus the lack of properly debugged IPv6 support for that then. Fortunately I don't (yet) need IPv6. When I do, it'll be curious to see if the bug is still there. I've got six machines with 6 Gb interfaces (two on motherboard, 4 on a card) right now (the design called for 3 bonded pairs on separate nets for redundancy). I haven't tried IPV6 on them. I had 'issues' with bonding and VMs though. My trouble didn't involve bonding or VMs, just using 5 interfaces at once (dual-homed WAN, DMZ, LAN, direct crossover failover backup). Having IPv6 turned on was screwing up using more than 4 for IPv4. Suspect it was a kernel limitation that in likelihood is since fixed. Whit ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On Wed, 2010-07-14 at 08:52 -0700, Benjamin Franz wrote: I've got six machines with 6 Gb interfaces (two on motherboard, 4 on a card) right now (the design called for 3 bonded pairs on separate nets for redundancy). I haven't tried IPV6 on them. I had 'issues' with bonding and VMs though. --- Can you give me sar -I SUM the last timed entry intr/s? How many CPUs? I'm not questioning you but on the curious side. John ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On 07/14/2010 10:30 AM, JohnS wrote: On Wed, 2010-07-14 at 08:52 -0700, Benjamin Franz wrote: I've got six machines with 6 Gb interfaces (two on motherboard, 4 on a card) right now (the design called for 3 bonded pairs on separate nets for redundancy). I haven't tried IPV6 on them. I had 'issues' with bonding and VMs though. --- Can you give me sar -I SUM the last timed entry intr/s? How many CPUs? I'm not questioning you but on the curious side. On the heaviest loaded machine: 10:10:01 AM sum 1637.48 10:20:01 AM sum 1640.73 10:30:01 AM sum 1653.58 10:40:01 AM sum 1617.78 10:50:01 AM sum 1727.97 11:00:01 AM sum 1767.88 11:10:01 AM sum 1798.93 11:20:01 AM sum 1782.14 Average: INTRintr/s Average: sum 1365.55 This is on a dual processor machine with a total of 8 cores. The highest I see on any of the machines for the last 24 hours is a brief (one ten minute interval) peak of 5300 intr/second during system backups and nothing over 3000 otherwise. -- Benjamin Franz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On 07/13/2010 05:51 PM, William Warren wrote: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? Most of the limits on the number of any device come from the major/minor numbers on the device nodes. Since ethernet adapters don't have device nodes, I'd be surprised if there were any practical limit. I'm able to create at least 1025 VLANs on my ethernet adapter, and VLANs are real interfaces in the kernel. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
2010/7/14 William Warren hescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? -- Eero ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On Jul 13, 2010, at 8:58 PM, Eero Volotinen eero.voloti...@iki.fi wrote: 2010/7/14 William Warren hescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? With tagged VLANs, bridges and virtual switches, is there a need for that many physical interfaces? -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On 7/13/2010 9:11 PM, Ross Walker wrote: On Jul 13, 2010, at 8:58 PM, Eero Volotineneero.voloti...@iki.fi wrote: 2010/7/14 William Warrenhescomins...@emmanuelcomputerconsulting.com: I think it's baloney mainly because i can't find a mention of it anywhere. Is there REALLY a limit on the number of physical network interfaces in the Kernel? can you really create hardware with huge number or real ethernet controllers? With tagged VLANs, bridges and virtual switches, is there a need for that many physical interfaces? -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ok let me specify. Is there any real limit? I've seen some folks(and been told by a few) that you can't more than 10 physical interfaces in a linux system. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Linux Kernel Physical Interface Limit
On 7/14/10, William Warren hescomins...@emmanuelcomputerconsulting.com wrote: ok let me specify. Is there any real limit? I've seen some folks(and been told by a few) that you can't more than 10 physical interfaces in a linux system. Googling up a really old 2005 newsgroup thread says some people had 24 physical NIC (6x 4quad) in a system before and one person vaguely remembers a hard 256 limit which would make sense if physical interface count is a byte value. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
