Re: Suggestions for Gigabit cards for -CURRENT
Of all the gin joints in all the towns in all the world, Kenneth D. Merry had to walk into mine and say: Talking of the XMAC II, there's one other thing I forgot to mention earlier. The FreeBSD sk driver does jumbo frames, but the SysKonnect drivers don't. At least, not yet. The XMAC II's receive FIFO is 8K. By default, the chip operates in 'store and forward' mode in order to perform error checking on received frames (it has to get the entire frame in the FIFO in order to do a CRC on it, I think). This is fine for normal frames, but if you want to handle jumbograms larger than 8192 bytes, you have to put the chip into 'streaming' mode, otherwise any frame larger than 8192 bytes will be truncated. To get 'streaming' mode to work, you have to disable all of the RX error checking. That is unfortunate, since it means you can't do checksum offloading with jumbo frames. Uhm. I'm not sure about that. The 8K FIFO limitation is in the XMAC II, not in the GEnesis controller. And I believe it's the GEnesis that actually does the hardware checksumming stuff. Oh, and the XMAC appears to have a 4K TX FIFO, not 2K. My mistake. FWIW, of the three gigabit ethernet implementations I've seen anything of (Alteon, Intel, SysKonnect), none have implemented all of the hooks necessary for a seamless zero copy receive implementation. Alteon comes the closest, but they don't support splitting out the headers (yet), which is a requirement for us. The only way to do zero copy receive with our VM architecture (that I know of) is page flipping, i.e. receive the page in the kernel, and then trade it for the user's page. You can't do it on anything less than page-sized granularity, and things have to be page aligned. (The IO-Lite stuff from Rice is an exception to all this.) The nice thing about the Alteon boards, though, is that you can modify the firmware, and so header splitting is an option there. It would even be possible to split the headers off of IPv6 packets, or any other protocol that you have knowlege of. If you can actually modify the firmware to do this then you have a lot more guru points than I do. :) I've looked at the Alteon firmware code but it's all quite opaque to me. -Bill -- = -Bill Paul(212) 854-6020 | System Manager, Master of Unix-Fu Work: [EMAIL PROTECTED] | Center for Telecommunications Research Home: [EMAIL PROTECTED] | Columbia University, New York City = "It is not I who am crazy; it is I who am mad!" - Ren Hoek, "Space Madness" = To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Suggestions for Gigabit cards for -CURRENT
Of all the gin joints in all the towns in all the world, Kenneth D. Merry had to walk into mine and say: On Wed, Feb 02, 2000 at 13:03:09 -0500, Thomas Stromberg wrote: We're currently looking at upgrading several of our FreeBSD servers (dual PIII-600's, 66MHz PCI) and some Sun Ultra's to Gigabit Ethernet. We plan to hook these machines into our Cisco Catalyst 5000 server. They will most likely move to be running FreeBSD 4.x by the time that we actually get our budget approved. What experiences do you guys have with the cards? Currently we're looking at the ~$1000 range, specifically at Alteon 512k's ($1000) for the FreeBSD servers and Sun Gigabit 2.0's ($2000) for the Sun servers. I was interested in the Myrinet cards (for obvious reasons), but they appear to require a Myrinet switch (though I found myself slightly confused so I may be wrong) rather then being able to hook into our Catalyst 5000. The Intel PRO/1000 Gigabit cards look rather nice too, but I haven't seen drivers yet for FreeBSD (Linux yes). I'm pretty much purchasing on marketing and reputation rather then any experience here, so any help would be much appreciated. I would recommend getting Alteon boards. It is likely that the Sun boards are Alteon OEM, although I'm not positive. I think the first gigabit cards Sun had on the market were OEMed from Alteon, but I've been told that their newer cards are something else entirely. I don't know exactly what, but they're not Tigon-based. One thing to keep in mind is that both Netgear and 3Com are OEMing Alteon boards, and you'll get them much cheaper that way. The boards are pretty much identical to the Alteon branded boards (which have no identifying marks on them). The performance is the same, at least for the Netgear boards. (I don't have any 3Com boards.) There are a number of companies selling OEM'ed alteon boards for various prices. IBM sells two cards, one for PC-based hardware and one for RS/6000s which I think are basically the same hardware with different driver kits. Of course, the RS/6000 card is $2100 while the PC-based one is probably around $600 or so. My guess is they're Alteon cards with different PCI device IDs, but I can't confirm this as I don't have one. The SGI gigabit adapter, NEC gigabit adapter, DEC EtherWORKS/1000, 3Com 3c985 and 3c985B, and the Netgear GA620 are all Tigon boards (not to mention the Alteon ACEnic) and should all work fine with the ti driver. Oh, I found another one recently: Farallon also sells a gigabit PCI NIC for the Mac which is Tigon-based. The Netgear GA620 is a 512K Tigon 2 board, and generally goes for around $300 or so. The 3Com boards have 1MB of SRAM, but I'm not sure whether they're Tigon 1 or Tigon 2. You really want a Tigon 2 board. Maybe someone who has one can comment. The original 3Com 3c985 was a Tigon 1 board (I have one) and the 3c985B is a Tigon 2. The Tigon 1 is no longer in production, though of course I try to maintain support for it for those people who still have them. The Tigon 1 had only a single R4000 CPU in it while the Tigon 2 has two. The Netgear GA620 is by far the cheapest at about $320. The various OEM cards sold for the PC are usually around $600, give or take $100. The GA620 only has 512K of SRAM compared to 1MB on most of the others, however you're not likely to notice a problem with that unless you try to push the card really hard with a really big TCP window size and jumbo frames. The Intel cards may look nice, and there is a FreeBSD driver for them, but I wouldn't get one. The first problem with the Intel boards is that there are no docs for them. Supposedly they're using a Cisco chip, and the specs for the chip are top secret. This is why I don't buy or recommend Intel NICs. But that's just my personal bias. The FreeBSD driver (written by Matt Jacob) is based on the Linux driver, which Intel wrote, and he hasn't yet managed to get decent throughput through the cards. (Maybe Matt will comment.) They also only have 64K of memory on board, which is insufficient for a heavily loaded server, IMO. Even with the 512K Alteon boards, you have a minimum of about 200K, and probably more like 300K of cache for transmit and receive. The Alteon cards also need a certain amount of SRAM to run the firmware. The Intel boards also don't have the features necessary to really support zero copy TCP receive. The Alteon boards, on the other hand, have most of the features necessary, and if I get some time, I may add the last feature (header splitting) to the firmware. The other alternative is SysKonnect, and that might actually be a good alternative. I haven't seen the boards, don't know how much they cost, etc. etc. You might want to ask Bill Paul about them, he wrote the driver. The SysKonnect cards aren't bad. A single port multimode fiber card is around $700, I think. The single mode cards are more expensive. However
Re: Suggestions for Gigabit cards for -CURRENT
[ Thanks for the info Bill! ] On Thu, Feb 03, 2000 at 21:29:27 -0500, Bill Paul wrote: Of all the gin joints in all the towns in all the world, Kenneth D. Merry had to walk into mine and say: The Netgear GA620 is a 512K Tigon 2 board, and generally goes for around $300 or so. The 3Com boards have 1MB of SRAM, but I'm not sure whether they're Tigon 1 or Tigon 2. You really want a Tigon 2 board. Maybe someone who has one can comment. The original 3Com 3c985 was a Tigon 1 board (I have one) and the 3c985B is a Tigon 2. The Tigon 1 is no longer in production, though of course I try to maintain support for it for those people who still have them. The Tigon 1 had only a single R4000 CPU in it while the Tigon 2 has two. Ahh, that's good to know, I was wondering whether they had a Tigon 2 board out, since it would make a cheaper alternative to the 1MB ACEnic. The Netgear GA620 is by far the cheapest at about $320. The various OEM cards sold for the PC are usually around $600, give or take $100. The GA620 only has 512K of SRAM compared to 1MB on most of the others, however you're not likely to notice a problem with that unless you try to push the card really hard with a really big TCP window size and jumbo frames. That has been my experience as well. The FreeBSD driver (written by Matt Jacob) is based on the Linux driver, which Intel wrote, and he hasn't yet managed to get decent throughput through the cards. (Maybe Matt will comment.) They also only have 64K of memory on board, which is insufficient for a heavily loaded server, IMO. Even with the 512K Alteon boards, you have a minimum of about 200K, and probably more like 300K of cache for transmit and receive. The Alteon cards also need a certain amount of SRAM to run the firmware. Yep, thus the 200K-300K number. The minimum amount of buffer that the board will configure is 64K for transmit buffers and 128K for receive buffers. It looks at the size of the firmware and associated data structures, and allocates the rest of the card memory for transmit and receive buffer space. Both the Alteon and SysKonnect NICs are 64-bit PCI cards. (Actually, I'm pretty sure all of the PCI gigabit NICs are 64-bit.) Both kinds of cards can do jumbograms on FreeBSD. Also, both vendors have released pretty good hardware documentation, which makes them good choices for custom applications, if you're into that sort of thing. Alteon also provides firmware source, which can really come in handy. Do you know if SysKonnect has released firmware? Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Suggestions for Gigabit cards for -CURRENT
Of all the gin joints in all the towns in all the world, Kenneth D. Merry had to walk into mine and say: [ Thanks for the info Bill! ] No problemo. [...] Both the Alteon and SysKonnect NICs are 64-bit PCI cards. (Actually, I'm pretty sure all of the PCI gigabit NICs are 64-bit.) Both kinds of cards can do jumbograms on FreeBSD. Also, both vendors have released pretty good hardware documentation, which makes them good choices for custom applications, if you're into that sort of thing. Alteon also provides firmware source, which can really come in handy. Do you know if SysKonnect has released firmware? The SysKonnect GEnesis controller and the XaQti XMAC II chips are both static devices and do not require firmware. If you go to www.syskonnect.com and search their online knowledge base for the word "manual" you should be able to find the gigabit NIC programmer's manual. Similarly, XaQti has the full datasheet for the XMAC II at www.xaqti.com somewhere. (As I recall, you have to go through a brief registration procedure to get it, but once that's done you should be able to download it right away.) Talking of the XMAC II, there's one other thing I forgot to mention earlier. The FreeBSD sk driver does jumbo frames, but the SysKonnect drivers don't. At least, not yet. The XMAC II's receive FIFO is 8K. By default, the chip operates in 'store and forward' mode in order to perform error checking on received frames (it has to get the entire frame in the FIFO in order to do a CRC on it, I think). This is fine for normal frames, but if you want to handle jumbograms larger than 8192 bytes, you have to put the chip into 'streaming' mode, otherwise any frame larger than 8192 bytes will be truncated. To get 'streaming' mode to work, you have to disable all of the RX error checking. Also, the default TX FIFO threshold on the XMAC is very small (8 bytes, I think). The FreeBSD sk driver bumps this up a bit (to 512 bytes, if I remember correctly). This is to deal with the case where you have a dual port card and are pumping data through both XMAC chips at once: with the default FIFO threshold, I would often see TX FIFO underruns from one of the XMACs and performance on that port would get spotty. I think the total TX FIFO memory on the XMAC II is 2K. -Bill -- = -Bill Paul(212) 854-6020 | System Manager, Master of Unix-Fu Work: [EMAIL PROTECTED] | Center for Telecommunications Research Home: [EMAIL PROTECTED] | Columbia University, New York City = "It is not I who am crazy; it is I who am mad!" - Ren Hoek, "Space Madness" = To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Suggestions for Gigabit cards for -CURRENT
On Thu, Feb 03, 2000 at 23:09:46 -0500, Bill Paul wrote: Of all the gin joints in all the towns in all the world, Kenneth D. Merry had to walk into mine and say: Alteon also provides firmware source, which can really come in handy. Do you know if SysKonnect has released firmware? The SysKonnect GEnesis controller and the XaQti XMAC II chips are both static devices and do not require firmware. If you go to www.syskonnect.com and search their online knowledge base for the word "manual" you should be able to find the gigabit NIC programmer's manual. Similarly, XaQti has the full datasheet for the XMAC II at www.xaqti.com somewhere. (As I recall, you have to go through a brief registration procedure to get it, but once that's done you should be able to download it right away.) Well, all-hardware designs can be great when they're done right, and irritating when they're not. Thanks for the manual pointers. I've got the SysKonnect manual, and it seems pretty nifty. Talking of the XMAC II, there's one other thing I forgot to mention earlier. The FreeBSD sk driver does jumbo frames, but the SysKonnect drivers don't. At least, not yet. The XMAC II's receive FIFO is 8K. By default, the chip operates in 'store and forward' mode in order to perform error checking on received frames (it has to get the entire frame in the FIFO in order to do a CRC on it, I think). This is fine for normal frames, but if you want to handle jumbograms larger than 8192 bytes, you have to put the chip into 'streaming' mode, otherwise any frame larger than 8192 bytes will be truncated. To get 'streaming' mode to work, you have to disable all of the RX error checking. That is unfortunate, since it means you can't do checksum offloading with jumbo frames. FWIW, of the three gigabit ethernet implementations I've seen anything of (Alteon, Intel, SysKonnect), none have implemented all of the hooks necessary for a seamless zero copy receive implementation. Alteon comes the closest, but they don't support splitting out the headers (yet), which is a requirement for us. The only way to do zero copy receive with our VM architecture (that I know of) is page flipping, i.e. receive the page in the kernel, and then trade it for the user's page. You can't do it on anything less than page-sized granularity, and things have to be page aligned. (The IO-Lite stuff from Rice is an exception to all this.) The nice thing about the Alteon boards, though, is that you can modify the firmware, and so header splitting is an option there. It would even be possible to split the headers off of IPv6 packets, or any other protocol that you have knowlege of. Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Suggestions for Gigabit cards for -CURRENT
On Wed, Feb 02, 2000 at 13:03:09 -0500, Thomas Stromberg wrote: We're currently looking at upgrading several of our FreeBSD servers (dual PIII-600's, 66MHz PCI) and some Sun Ultra's to Gigabit Ethernet. We plan to hook these machines into our Cisco Catalyst 5000 server. They will most likely move to be running FreeBSD 4.x by the time that we actually get our budget approved. What experiences do you guys have with the cards? Currently we're looking at the ~$1000 range, specifically at Alteon 512k's ($1000) for the FreeBSD servers and Sun Gigabit 2.0's ($2000) for the Sun servers. I was interested in the Myrinet cards (for obvious reasons), but they appear to require a Myrinet switch (though I found myself slightly confused so I may be wrong) rather then being able to hook into our Catalyst 5000. The Intel PRO/1000 Gigabit cards look rather nice too, but I haven't seen drivers yet for FreeBSD (Linux yes). I'm pretty much purchasing on marketing and reputation rather then any experience here, so any help would be much appreciated. I would recommend getting Alteon boards. It is likely that the Sun boards are Alteon OEM, although I'm not positive. One thing to keep in mind is that both Netgear and 3Com are OEMing Alteon boards, and you'll get them much cheaper that way. The boards are pretty much identical to the Alteon branded boards (which have no identifying marks on them). The performance is the same, at least for the Netgear boards. (I don't have any 3Com boards.) The Netgear GA620 is a 512K Tigon 2 board, and generally goes for around $300 or so. The 3Com boards have 1MB of SRAM, but I'm not sure whether they're Tigon 1 or Tigon 2. You really want a Tigon 2 board. Maybe someone who has one can comment. The Intel cards may look nice, and there is a FreeBSD driver for them, but I wouldn't get one. The first problem with the Intel boards is that there are no docs for them. Supposedly they're using a Cisco chip, and the specs for the chip are top secret. The FreeBSD driver (written by Matt Jacob) is based on the Linux driver, which Intel wrote, and he hasn't yet managed to get decent throughput through the cards. (Maybe Matt will comment.) They also only have 64K of memory on board, which is insufficient for a heavily loaded server, IMO. Even with the 512K Alteon boards, you have a minimum of about 200K, and probably more like 300K of cache for transmit and receive. The Intel boards also don't have the features necessary to really support zero copy TCP receive. The Alteon boards, on the other hand, have most of the features necessary, and if I get some time, I may add the last feature (header splitting) to the firmware. The other alternative is SysKonnect, and that might actually be a good alternative. I haven't seen the boards, don't know how much they cost, etc. etc. You might want to ask Bill Paul about them, he wrote the driver. Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Suggestions for Gigabit cards for -CURRENT
"Kenneth D. Merry" wrote: On Wed, Feb 02, 2000 at 13:03:09 -0500, Thomas Stromberg wrote: We're currently looking at upgrading several of our FreeBSD servers (dual PIII-600's, 66MHz PCI) and some Sun Ultra's to Gigabit Ethernet. We plan to hook these machines into our Cisco Catalyst 5000 server. They will most likely move to be running FreeBSD 4.x by the time that we actually get our budget approved. What experiences do you guys have with the cards? I would recommend getting Alteon boards. It is likely that the Sun boards are Alteon OEM, although I'm not positive. One thing to keep in mind is that both Netgear and 3Com are OEMing Alteon boards, and you'll get them much cheaper that way. The boards are pretty much identical to the Alteon branded boards (which have no identifying marks on them). The performance is the same, at least for the Netgear boards. (I don't have any 3Com boards.) The Netgear GA620 is a 512K Tigon 2 board, and generally goes for around $300 or so. The 3Com boards have 1MB of SRAM, but I'm not sure whether they're Tigon 1 or Tigon 2. You really want a Tigon 2 board. Maybe someone who has one can comment. I have a couple of GA620's here, too, and they work quite well. They support both 66Mhz and 64-bit PCI bus. We paid $329 from warehouse.com about 4 months ago, quite a reasonable price. You might want to look around and see if anyone has Solaris drivers for the Tigon-2, if you're using PCI Sun machines. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC [EMAIL PROTECTED] http://softweyr.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Suggestions for Gigabit cards for -CURRENT
On Wed, Feb 02, 2000 at 15:11:16 -0700, Wes Peters wrote: "Kenneth D. Merry" wrote: On Wed, Feb 02, 2000 at 13:03:09 -0500, Thomas Stromberg wrote: We're currently looking at upgrading several of our FreeBSD servers (dual PIII-600's, 66MHz PCI) and some Sun Ultra's to Gigabit Ethernet. We plan to hook these machines into our Cisco Catalyst 5000 server. They will most likely move to be running FreeBSD 4.x by the time that we actually get our budget approved. What experiences do you guys have with the cards? I would recommend getting Alteon boards. It is likely that the Sun boards are Alteon OEM, although I'm not positive. One thing to keep in mind is that both Netgear and 3Com are OEMing Alteon boards, and you'll get them much cheaper that way. The boards are pretty much identical to the Alteon branded boards (which have no identifying marks on them). The performance is the same, at least for the Netgear boards. (I don't have any 3Com boards.) The Netgear GA620 is a 512K Tigon 2 board, and generally goes for around $300 or so. The 3Com boards have 1MB of SRAM, but I'm not sure whether they're Tigon 1 or Tigon 2. You really want a Tigon 2 board. Maybe someone who has one can comment. I have a couple of GA620's here, too, and they work quite well. They support both 66Mhz and 64-bit PCI bus. We paid $329 from warehouse.com about 4 months ago, quite a reasonable price. You might want to look around and see if anyone has Solaris drivers for the Tigon-2, if you're using PCI Sun machines. Alteon includes a Solaris driver in their driver development kit. It might be possible to modify the probe routine to recognize other Tigon boards like the Netgear or 3Com boards. Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Suggestions for Gigabit cards for -CURRENT
We're currently looking at upgrading several of our FreeBSD servers (dual PIII-600's, 66MHz PCI) and some Sun Ultra's to Gigabit Ethernet. We plan to hook these machines into our Cisco Catalyst 5000 server. They will most likely move to be running FreeBSD 4.x by the time that we actually get our budget approved. What experiences do you guys have with the cards? Currently we're looking at the ~$1000 range, specifically at Alteon 512k's ($1000) for the FreeBSD servers and Sun Gigabit 2.0's ($2000) for the Sun servers. I was interested in the Myrinet cards (for obvious reasons), but they appear to require a Myrinet switch (though I found myself slightly confused so I may be wrong) rather then being able to hook into our Catalyst 5000. The Intel PRO/1000 Gigabit cards look rather nice too, but I haven't seen drivers yet for FreeBSD (Linux yes). I'm pretty much purchasing on marketing and reputation rather then any experience here, so any help would be much appreciated. -- === Thomas R. StrombergAsst. IS Manager / Systems Guru FreeBSD Contrib, Security Geek, etc. Research Triangle Commerce, Inc. http://www.afterthought.org/ http://www.rtci.com/ [EMAIL PROTECTED] [EMAIL PROTECTED] --- MCSE: McDonald's Certified Service Engineer ===eof To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
