Re: [vchkpw] Vpopmail + NetApp still best practice?
Alex Borges wrote: While youre considering proprietary solutions and naturally, would like to pay for them, perhaps you should consider redhat's GFS thingie. Its GPL but redhat offers it with their AS for an extra $$ Ive seen it work and it seems like quite a scalable solution and tipically cheaper than buying a SAN. We have GFS (6.0). Its performance is mediocre - and horrible for some things (like doing du(1) on a GFS-directory) Also, I hate generating (well, trying to) packages for RHEL, where I could use the FreeBSD-port and have all the necessary patches and tuning included. Of the 1400 or so RPMs delivered by RHEL, I can barely use a handful for my toasters (some libraries, maybe). IMO, RHEL et.al. don't make any sense at all for this type of work. You're paying to have a supported linux-kernel + sshd updated regularly (because that's all that is left of the original after 5 years). And you've got more work adapting your software to your OS than elsewhere. Just try to get an equally modularized PHP4 or PHP5-RPM for RHEL that has support for as many modules as the FreeBSD-port. However, sans do offer plenty advantages on some environments (wann have the winboxes and linboxes scsi-plugged into the same san), if this is just for email, this can be a cheaper solution. We have a SAN (HP EVA 3000 with 6 TB raw cap.), it's nice. But it costs a lot of money all together (HBAs, FC-switches, FC-ports, cables etc.pp.) and my gut-feeling is that I can deliver the same performance and scalability (or even much more, in our case) with about the same level of reliability when going with a high-end NAS - and even save money in the end. Also, email is not just email unfortunately. Left without their email, our customers would just go to another ISP... With this kind of thing, what you get to do is plug three or more boxes with whatever storage they have and then store on all of them. This thing works with LVM2 so you can partition, snapshot and share it to your hearts content. Put a solid GB net on it with separate NICS (from the NICS youll be using to actually provide service) for best results. That's the theory, yes. In pratice, though, it seems that GFS6.0 (no tests with 6.1, yet) is not suitable for workloads where a lot of transactions occur in one directory (like mail-delivery). It creates a lock-file, everytime a file is changed in a maildir, doubling (at least) the I/Os of maildir-maildelivery). That's useless in this case because IMO qmail itself takes care that no locking-issues are race-conditions occur. cheers, Rainer
Re: [vchkpw] Vpopmail + NetApp still best practice?
Nicholas Harring wrote: Hi, when going the Maildir on NFS for clustering-route, is using NetApp Filers still considered state of the art or has something better emerged? There are plenty of other NAS options, see EMC for one vendor (also not cheap). Dell offers NAS, HP I believe does as well. Not sure how much clustering they offer, and what sort of feature set it has compared to NetApp. These are all basically W2K3-servers (Windows Storage Server) (EMC uses Windows even in the high-end gear, IIRC, but not necessarily WSS). I'm not going to gamble with the NFS-performance and the compatibility issues of Microsoft-flavoured NFS. From a price-point, I'd rather use FreeBSD, but the fact that there's no real volume-manager makes it unusable for our purposes. I've actually mailed Blue Arc about their hardware, but despite not being in the black, they didn't feel it necessary to answer my query. For a smaller cluster or one that doesn't have hard uptime commitments in the 4 or 5 9s range I'd say that a *nix solution would work just fine. If you laid something like Veritas Clustering on top of it then moving into the real HA range should also be quite possible and supportable. I've also thought about buying an X4100 and fitting it with a Dual QLA, then exporting the mailstorage via NFS from that (using our HP SAN as backend). But using a NetApp would allow have our hosting-operations being spread over two completely independent technologies (Web- HP EVA, Mail-NetApp), avoiding a complete loss of service should one of the two fail for whatever reason (like a competitor, who put all eggs in a single basket recently learned the hard way...). Does anybody have any sizing-information? NetApp offers a lot of hardware and even the entry-level stuff is not cheap. I'd like to know how many deliveries/h one can make e.g. with a small FAS 270. I'm running 8 servers (4 smtp, 4 pop/imap) on an F820c cluster doing around 600k messages daily. I don't have any hourly stats at the moment, but that load is spread with about 80% across 10-12 hours with the remainder spread evenly across the other 12-14. I'm currently upgrading my cluster to FAS3050s but not due to performance reasons, but rather storage consolidation throughout my network. 600k deliveries/day? How much room is there 'till the NetApp is maxed out? ... Hope that helps, Nick Yeah, it does. Thanks a lot. Rainer
RE: [vchkpw] Vpopmail + NetApp still best practice?
Nicholas Harring wrote: Hi, when going the Maildir on NFS for clustering-route, is using NetApp Filers still considered state of the art or has something better emerged? There are plenty of other NAS options, see EMC for one vendor (also not cheap). Dell offers NAS, HP I believe does as well. Not sure how much clustering they offer, and what sort of feature set it has compared to NetApp. These are all basically W2K3-servers (Windows Storage Server) (EMC uses Windows even in the high-end gear, IIRC, but not necessarily WSS). I'm not going to gamble with the NFS-performance and the compatibility issues of Microsoft-flavoured NFS. Yep, they're all based on the SAK and all have the same flaws. NFS being the big one of the bunch, but even stability isn't entirely where it should be compared to the competition. I figured I'd throw it out there because some people seem to be having good luck with them, and I wasn't sure how familiar you were or weren't with them. We came to the same conclusion before making our purchasing decision on the NetApp gear, for pretty much the same reasons you cited. From a price-point, I'd rather use FreeBSD, but the fact that there's no real volume-manager makes it unusable for our purposes. I've actually mailed Blue Arc about their hardware, but despite not being in the black, they didn't feel it necessary to answer my query. For a smaller cluster or one that doesn't have hard uptime commitments in the 4 or 5 9s range I'd say that a *nix solution would work just fine. If you laid something like Veritas Clustering on top of it then moving into the real HA range should also be quite possible and supportable. I've also thought about buying an X4100 and fitting it with a Dual QLA, then exporting the mailstorage via NFS from that (using our HP SAN as backend). But using a NetApp would allow have our hosting-operations being spread over two completely independent technologies (Web- HP EVA, Mail-NetApp), avoiding a complete loss of service should one of the two fail for whatever reason (like a competitor, who put all eggs in a single basket recently learned the hard way...). That sounds like a good division of labor, and mitigation of risk. While I normally don't advocate consolidation, the NetApps really do make it a safe and inviting option. Our filers currently have 797 days of uptime without a single hiccup in service (the only devices we own which have been more solid are a pair of Cisco Catalyst 6500 switches). This obviously means I've not been keeping up with ONTAP releases or firmware upgrades, but since none have addressed anything I'd need this has also been quite safe. Does anybody have any sizing-information? NetApp offers a lot of hardware and even the entry-level stuff is not cheap. I'd like to know how many deliveries/h one can make e.g. with a small FAS 270. I'm running 8 servers (4 smtp, 4 pop/imap) on an F820c cluster doing around 600k messages daily. I don't have any hourly stats at the moment, but that load is spread with about 80% across 10-12 hours with the remainder spread evenly across the other 12-14. I'm currently upgrading my cluster to FAS3050s but not due to performance reasons, but rather storage consolidation throughout my network. 600k deliveries/day? How much room is there 'till the NetApp is maxed out? I'm peaking at around 70% cpu during my busiest periods, doing around 8k nfs ops/second. To me this means I'm nearly tapped out, but only because I'm not willing to load these up to the point that a head failure would mean performance degradation when the other head took over. I believe the FAS line are all a good bit faster than the old F8xx line, so I'd expect even the lowest end to be able to handle more than this load without buckling. We did some back of the envelope figuring when buying the FAS3050, and figured that if we moved to GigE and bonded our Ethernet connections to get a 3Gig link from the filers to the core switches we could probably scale to ~8M deliveries/day, however that number should be taken with a big grain of salt because that's a lot of scaling without any empirical data between where we're at and where we'd end up. Cheers, Nick
Re: [vchkpw] Vpopmail + NetApp still best practice?
Rainer Duffner wrote: Hi, when going the Maildir on NFS for clustering-route, is using NetApp Filers still considered state of the art or has something better emerged? From a price-point, I'd rather use FreeBSD, but the fact that there's no real volume-manager makes it unusable for our purposes. I've actually mailed Blue Arc about their hardware, but despite not being in the black, they didn't feel it necessary to answer my query. Does anybody have any sizing-information? NetApp offers a lot of hardware and even the entry-level stuff is not cheap. I'd like to know how many deliveries/h one can make e.g. with a small FAS 270. No idea about a NetApp, but my backend is a Sun Enterprise. I connect to my toasters (three of them) via a 1gb network. I can deliver 8-13k messages an hour, per toaster. I have seen traffic at the level of 20-25k apiece. I can't give MTRG stats for the interface as the traffic includes webmail, spamc connections, mysql connections, etc. The Sun box has no trouble keeping up even though it is a older box with slow drives. Volume managers for FreeBSD, do you mean Vinum? CCD? GEOM? DAve
RE: [vchkpw] Vpopmail + NetApp still best practice?
Hi, when going the Maildir on NFS for clustering-route, is using NetApp Filers still considered state of the art or has something better emerged? There are plenty of other NAS options, see EMC for one vendor (also not cheap). Dell offers NAS, HP I believe does as well. Not sure how much clustering they offer, and what sort of feature set it has compared to NetApp. From a price-point, I'd rather use FreeBSD, but the fact that there's no real volume-manager makes it unusable for our purposes. I've actually mailed Blue Arc about their hardware, but despite not being in the black, they didn't feel it necessary to answer my query. For a smaller cluster or one that doesn't have hard uptime commitments in the 4 or 5 9s range I'd say that a *nix solution would work just fine. If you laid something like Veritas Clustering on top of it then moving into the real HA range should also be quite possible and supportable. Does anybody have any sizing-information? NetApp offers a lot of hardware and even the entry-level stuff is not cheap. I'd like to know how many deliveries/h one can make e.g. with a small FAS 270. I'm running 8 servers (4 smtp, 4 pop/imap) on an F820c cluster doing around 600k messages daily. I don't have any hourly stats at the moment, but that load is spread with about 80% across 10-12 hours with the remainder spread evenly across the other 12-14. I'm currently upgrading my cluster to FAS3050s but not due to performance reasons, but rather storage consolidation throughout my network. To see how that compares you might try using their spec_nfs numbers since they'd be roughly representative of the type of load you'd be using. cheers, Rainer I do have to say NetApp has been the easiest, most reliable vendor I've ever worked with in the IT realm. Their support is top notch, the only time there's been a hardware failure they knew about it before I did (I was out to lunch when the drive died and they called moments later). Even their VARs actually add value rather than just price. For sizing I initially worked with the VAR they use in the Chicago area (INCAT) and they were immensely helpful in determining what we needed and not overselling me. One other consideration when using NetApp is that they have a lot of features that would help with things like disaster recovery, backups, etc. These are the areas where NetApp and EMC typically clobber everyone else, and the reason you find them so frequently in high volume high availability data environments. Hope that helps, Nick
RE: [vchkpw] Vpopmail + NetApp still best practice?
While youre considering proprietary solutions and naturally, would like to pay for them, perhaps you should consider redhat's GFS thingie. Its GPL but redhat offers it with their AS for an extra $$ Ive seen it work and it seems like quite a scalable solution and tipically cheaper than buying a SAN. However, sans do offer plenty advantages on some environments (wann have the winboxes and linboxes scsi-plugged into the same san), if this is just for email, this can be a cheaper solution. With this kind of thing, what you get to do is plug three or more boxes with whatever storage they have and then store on all of them. This thing works with LVM2 so you can partition, snapshot and share it to your hearts content. Put a solid GB net on it with separate NICS (from the NICS youll be using to actually provide service) for best results. On Wed, 2006-02-22 at 14:20 -0600, Nicholas Harring wrote: Hi, when going the Maildir on NFS for clustering-route, is using NetApp Filers still considered state of the art or has something better emerged? There are plenty of other NAS options, see EMC for one vendor (also not cheap). Dell offers NAS, HP I believe does as well. Not sure how much clustering they offer, and what sort of feature set it has compared to NetApp. From a price-point, I'd rather use FreeBSD, but the fact that there's no real volume-manager makes it unusable for our purposes. I've actually mailed Blue Arc about their hardware, but despite not being in the black, they didn't feel it necessary to answer my query. For a smaller cluster or one that doesn't have hard uptime commitments in the 4 or 5 9s range I'd say that a *nix solution would work just fine. If you laid something like Veritas Clustering on top of it then moving into the real HA range should also be quite possible and supportable. Does anybody have any sizing-information? NetApp offers a lot of hardware and even the entry-level stuff is not cheap. I'd like to know how many deliveries/h one can make e.g. with a small FAS 270. I'm running 8 servers (4 smtp, 4 pop/imap) on an F820c cluster doing around 600k messages daily. I don't have any hourly stats at the moment, but that load is spread with about 80% across 10-12 hours with the remainder spread evenly across the other 12-14. I'm currently upgrading my cluster to FAS3050s but not due to performance reasons, but rather storage consolidation throughout my network. To see how that compares you might try using their spec_nfs numbers since they'd be roughly representative of the type of load you'd be using. cheers, Rainer I do have to say NetApp has been the easiest, most reliable vendor I've ever worked with in the IT realm. Their support is top notch, the only time there's been a hardware failure they knew about it before I did (I was out to lunch when the drive died and they called moments later). Even their VARs actually add value rather than just price. For sizing I initially worked with the VAR they use in the Chicago area (INCAT) and they were immensely helpful in determining what we needed and not overselling me. One other consideration when using NetApp is that they have a lot of features that would help with things like disaster recovery, backups, etc. These are the areas where NetApp and EMC typically clobber everyone else, and the reason you find them so frequently in high volume high availability data environments. Hope that helps, Nick
