Re: Speed up queue injection
On Tue, Aug 17, 2010 at 01:41:20PM -0500, Stan Hoeppner wrote: Anyway, if you had the time and inclination and were able to get your hands on a few units, it would be great to see some basic queue performance data from you on SSD vs a disk based test rig you use. All benchmarks are artificial, some are more artificial than others. It is rate on enterprise-grade kit to find MTAs that are disk I/O constrained. More typicall, the CPU cost of filtering or downstream throughput are the limiting factors. -- Viktor.
Re: Speed up queue injection
On Tue, Aug 17, 2010 at 01:41:20PM -0500, Stan Hoeppner wrote: Anyway, if you had the time and inclination and were able to get your hands on a few units, it would be great to see some basic queue performance data from you on SSD vs a disk based test rig you use. Victor Duchovni: All benchmarks are artificial, some are more artificial than others. It is rate on enterprise-grade kit to find MTAs that are disk I/O constrained. More typicall, the CPU cost of filtering or downstream throughput are the limiting factors. I already mentioned off-list that I work for a company (IBM) which sells hardware, and that it would not be appropriate for me to post performance measurements. Wietse
Re: Speed up queue injection
Hi! On Mon, Aug 16, 2010 at 9:02 PM, Stan Hoeppner s...@hardwarefreak.com wrote: Stan Hoeppner put forth on 8/16/2010 6:56 PM: Wietse Venema put forth on 8/16/2010 2:36 PM: Stan Hoeppner: Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Indeed, but then Google's scale of operations is not representative of most enterprises. Large companies (I work for one) can self-insure for small accidents, small companies can't. Wietse have you done any testing with SSDs? If not, would you like to? I'm sure various vendors would be glad to loan you some. Get a mix of consumer and enterprise SSDs. And make sure you get one of the Intel X25-E 80GB units. :) I should have made clear that they will do this for you, because you are, well, you. ;) I'm a nobody, so they won't loan me the hardware. :( I need to see if I can get a gig doing reviews for hardware sites. I kinda grew out of being a hardwarefreak a while back or I'd probably be doing reviews now. :( Ok, just do it! start your own technical blog, and do hardware reviews! Then: let us know to go there and put nice comments! You can start with whatever hardware you have, then, show your blog to your friends, and ask to review their hardware too, maybe, the college where you study or studied, there you could have some contact with the technical staff, and maybe you can get access to more hardware (after showing your site, and explaining politely). Also, don't you have a friend with a computer store? there you may get access to hardware for free (as long as you don't break it), and... more reviews! The more difficult part is: coming out with a nice domain name for the techblog :( . Ildefonso.
Re: Speed up queue injection
Hi! On Mon, Aug 23, 2010 at 12:30 PM, Wietse Venema wie...@porcupine.org wrote: On Tue, Aug 17, 2010 at 01:41:20PM -0500, Stan Hoeppner wrote: Anyway, if you had the time and inclination and were able to get your hands on a few units, it would be great to see some basic queue performance data from you on SSD vs a disk based test rig you use. Victor Duchovni: All benchmarks are artificial, some are more artificial than others. It is rate on enterprise-grade kit to find MTAs that are disk I/O constrained. More typicall, the CPU cost of filtering or downstream throughput are the limiting factors. I already mentioned off-list that I work for a company (IBM) which sells hardware, and that it would not be appropriate for me to post performance measurements. As a matter of fact, I don't think you can even publish any review without permission from your employee (even for IBM hardware). Last time I saw an IBM contract (around 5 years ago) it was basically a: We own you! kind of contract (they had ownership on anything one do, even on one's own free time). Now, on a side note: this thread is very interesting. I have always liked high-performance systems discussions. Ildefonso.
Re: Speed up queue injection
Stan Hoeppner: Wietse Venema put forth on 8/16/2010 2:36 PM: Stan Hoeppner: Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Indeed, but then Google's scale of operations is not representative of most enterprises. Large companies (I work for one) can self-insure for small accidents, small companies can't. Wietse have you done any testing with SSDs? Stan, don't f.. with me. I was addressing your comment about enterprise versus non-enterprise grade. Wietse
Re: Speed up queue injection
Wietse Venema put forth on 8/17/2010 6:11 AM: Stan Hoeppner: Wietse Venema put forth on 8/16/2010 2:36 PM: Stan Hoeppner: Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Indeed, but then Google's scale of operations is not representative of most enterprises. Large companies (I work for one) can self-insure for small accidents, small companies can't. Wietse have you done any testing with SSDs? Stan, don't f.. with me. I was addressing your comment about enterprise versus non-enterprise grade. Didn't realize you replied to the list as well. So here is a copy of my direct reply, lest anyone be left with the wrong idea: I wasn't messing with you Wietse. I was dead serious. I've not seen any real information out there regarding mail queue performance on SSD, merely guesses and conjecture. Given you are who you are, and I know you do at least some Postfix testing in the lab, I figured you wouldn't have any problems acquiring a few SSDs. The only thing I don't know is if you'd have time to mess with such a thing or the desire. I recall a few months ago you said you had two deliverables you had you get done and that you'd not have much time for Postfix development. Anyway, if you had the time and inclination and were able to get your hands on a few units, it would be great to see some basic queue performance data from you on SSD vs a disk based test rig you use. Like I said, you may not have the time or inclination, but if you did, your results would be invaluable to at least a few people. -- Stan
Re: Speed up queue injection
On Sun, 2010-08-15 at 17:35 +0200, J. Roeleveld wrote: On Friday 13 August 2010 19:58:38 Noel Jones wrote: On 8/13/2010 8:22 AM, J. Roeleveld wrote: On Friday 13 August 2010 14:23:51 Wietse Venema wrote: Ralf Hildebrandt: * Ramr...@netcore.co.in: Mail in plain text format , mime encoded message OK! Currenlty I get 40/s - 45/s That sounds normal. Any filtering (in these cases you should inject in a way that bypasses and filters) But I want it to be atleast 100/s Two machineS? relay boxes Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. Why not inject to the further relay boxes? Do I need to increase the hardware It could be :) Other options: increase input concurrency, or play with in_flow_delay. Note that increasing your input rates will cause output rates to drop. It's all about competing for disk access. Wietse Further options, I think: - Disable filtering (provided the only possible connections are related to these emails Presumably the client would be in mynetworks, which should bypass most or all restrictions, so this is unlikely to make much difference. Unless you're doing something silly like 1000 body_check rules or using a content_filter or milter. - put the queue on a ram-disk (8GB Ram, might leave 6GB for the queue, would this be sufficient?) Putting the queue on ramdisk is only for spammers who don't particularly care if their mail is lost. But putting the queue on an enterprise-quality SSD would almost certainly help. But Enterprise quality SSD's are so expensive. I can get an additional server and still save money. It seems I will have to break my app scatter the mail creation across multiple servers to acheieve higher injection. Thanks Ram
Re: Speed up queue injection
On 8/16/2010 9:36 AM, Stan Hoeppner wrote: Ram put forth on 8/16/2010 8:19 AM: But Enterprise quality SSD's are so expensive. I can get an additional server and still save money. I call BS: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167023 ... Yeah, it's fast. Whether you consider it enterprise quality or not, it's Intel, and it ain't gonna fail. If I was running an MX farm that needed maximum performance, I'd already have one of these in each server, many months ago. ;) The Intel X25-E series is enterprise-grade. The 64G model sells for $700~$800. Quite a bit more expensive than the consumer X25-M series, but better suited for server use, and still far less than a decent server. Can't say for sure without testing, but I wouldn't be surprised if the SSD is faster than two servers sharing the load. -- Noel Jones
Re: Speed up queue injection
Noel Jones put forth on 8/16/2010 10:03 AM: On 8/16/2010 9:36 AM, Stan Hoeppner wrote: Ram put forth on 8/16/2010 8:19 AM: But Enterprise quality SSD's are so expensive. I can get an additional server and still save money. I call BS: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167023 ... Yeah, it's fast. Whether you consider it enterprise quality or not, it's Intel, and it ain't gonna fail. If I was running an MX farm that needed maximum performance, I'd already have one of these in each server, many months ago. ;) The Intel X25-E series is enterprise-grade. The 64G model sells for $700~$800. Quite a bit more expensive than the consumer X25-M series, but better suited for server use, and still far less than a decent server. Can't say for sure without testing, but I wouldn't be surprised if the SSD is faster than two servers sharing the load. Enterprise grade or not, this 80GB SSD's 160GB big brother tests out at over 17,000 random write IO/s. I don't find test data for the 80GB model, but given Intel's spec claims for each and extrapolating, the 80GB model should be only 25% slower, equaling 12,750 random write IO/s. At that rate, it should easily give the OP a 10x increase in queue throughput assuming he's currently using a 4 x 15k RPM SAS drive RAID 0 stripe for his queue. If his queue is currently on a single 15k SAS drive, his throughput increase using this 80GB Intel SSD would be over 42x. Enterprise grade usually has far more to do with these things and little to do with performance or failure rates: 1. Warranty period 2. Longer availability period before EOL--longer spares availability 3. More extensive interoperability testing, or sometimes far less testing, but a certification that the device will work with other brand's model X devices. Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Enterprise is a marketing term, not a technical one. Too many people are cowed and convinced that they need Enterprise gear. This is far from the truth for well over 99% of server OPs. -- Stan
Re: Speed up queue injection
Stan Hoeppner: Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Indeed, but then Google's scale of operations is not representative of most enterprises. Large companies (I work for one) can self-insure for small accidents, small companies can't. Wietse
Re: Speed up queue injection
Wietse Venema put forth on 8/16/2010 2:36 PM: Stan Hoeppner: Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Indeed, but then Google's scale of operations is not representative of most enterprises. Large companies (I work for one) can self-insure for small accidents, small companies can't. Wietse have you done any testing with SSDs? If not, would you like to? I'm sure various vendors would be glad to loan you some. Get a mix of consumer and enterprise SSDs. And make sure you get one of the Intel X25-E 80GB units. :) -- Stan
Re: Speed up queue injection
Stan Hoeppner put forth on 8/16/2010 6:56 PM: Wietse Venema put forth on 8/16/2010 2:36 PM: Stan Hoeppner: Google uses less than 1/10th of 1% Enterprise grade hardware, using the typical definition of Enterprise grade, in their operations. And Google is the undisputed single largest operator of servers on the planet. I think that qualifies them as an Enterprise. ;) Indeed, but then Google's scale of operations is not representative of most enterprises. Large companies (I work for one) can self-insure for small accidents, small companies can't. Wietse have you done any testing with SSDs? If not, would you like to? I'm sure various vendors would be glad to loan you some. Get a mix of consumer and enterprise SSDs. And make sure you get one of the Intel X25-E 80GB units. :) I should have made clear that they will do this for you, because you are, well, you. ;) I'm a nobody, so they won't loan me the hardware. :( I need to see if I can get a gig doing reviews for hardware sites. I kinda grew out of being a hardwarefreak a while back or I'd probably be doing reviews now. :( -- Stan
Re: Speed up queue injection
On Friday 13 August 2010 19:58:38 Noel Jones wrote: On 8/13/2010 8:22 AM, J. Roeleveld wrote: On Friday 13 August 2010 14:23:51 Wietse Venema wrote: Ralf Hildebrandt: * Ramr...@netcore.co.in: Mail in plain text format , mime encoded message OK! Currenlty I get 40/s - 45/s That sounds normal. Any filtering (in these cases you should inject in a way that bypasses and filters) But I want it to be atleast 100/s Two machineS? relay boxes Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. Why not inject to the further relay boxes? Do I need to increase the hardware It could be :) Other options: increase input concurrency, or play with in_flow_delay. Note that increasing your input rates will cause output rates to drop. It's all about competing for disk access. Wietse Further options, I think: - Disable filtering (provided the only possible connections are related to these emails Presumably the client would be in mynetworks, which should bypass most or all restrictions, so this is unlikely to make much difference. Unless you're doing something silly like 1000 body_check rules or using a content_filter or milter. - put the queue on a ram-disk (8GB Ram, might leave 6GB for the queue, would this be sufficient?) Putting the queue on ramdisk is only for spammers who don't particularly care if their mail is lost. But putting the queue on an enterprise-quality SSD would almost certainly help. The OP did mention that if the email would be late, it wouldn't be of any use anymore. Research reports are useless after the market opens So I don't think he would mind loosing them if the power goes. But if he does, wouldn't a decent UPS with backup-to-disk of the queue during shutdown cover that? -- Joost
Re: Speed up queue injection
* Ram r...@netcore.co.in: We have a requirement to send some research analysis mails as quickly as possible. Everyday after the data is available my app generates the mails in eml format in a directory. What is eml format? Currently I have a perl script that makes parallel smtp connections on localhost and sends the mails. This sounds good! Should I send the mails on command line. No, using the postfix sendmail binary is actually slower. There are currently around 50k mails to be delivered ideally within 5-10 mins. How fast are you now? 50.000/10min = 5.000/min = 83/s = that's a lot 50.000/50min = 10.000/min = 186/s = that's even more -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Speed up queue injection
Hi , On Fri, 2010-08-13 at 09:39 +0200, Ralf Hildebrandt wrote: * Ram r...@netcore.co.in: We have a requirement to send some research analysis mails as quickly as possible. Everyday after the data is available my app generates the mails in eml format in a directory. What is eml format? Mail in plain text format , mime encoded message Currently I have a perl script that makes parallel smtp connections on localhost and sends the mails. This sounds good! Should I send the mails on command line. No, using the postfix sendmail binary is actually slower. There are currently around 50k mails to be delivered ideally within 5-10 mins. How fast are you now? 50.000/10min = 5.000/min = 83/s = that's a lot 50.000/50min = 10.000/min = 186/s = that's even more Currenlty I get 40/s - 45/s But I want it to be atleast 100/s Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. This is a 8GB Ram Centos 5.4 server with SAS discs Do I need to increase the hardware Thanks Ram
Re: Speed up queue injection
* Ram r...@netcore.co.in: Mail in plain text format , mime encoded message OK! Currenlty I get 40/s - 45/s That sounds normal. Any filtering (in these cases you should inject in a way that bypasses and filters) But I want it to be atleast 100/s Two machineS? relay boxes Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. Why not inject to the further relay boxes? Do I need to increase the hardware It could be :) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Speed up queue injection
Ralf Hildebrandt: * Ram r...@netcore.co.in: Mail in plain text format , mime encoded message OK! Currenlty I get 40/s - 45/s That sounds normal. Any filtering (in these cases you should inject in a way that bypasses and filters) But I want it to be atleast 100/s Two machineS? relay boxes Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. Why not inject to the further relay boxes? Do I need to increase the hardware It could be :) Other options: increase input concurrency, or play with in_flow_delay. Note that increasing your input rates will cause output rates to drop. It's all about competing for disk access. Wietse
Re: Speed up queue injection
On Friday 13 August 2010 14:23:51 Wietse Venema wrote: Ralf Hildebrandt: * Ram r...@netcore.co.in: Mail in plain text format , mime encoded message OK! Currenlty I get 40/s - 45/s That sounds normal. Any filtering (in these cases you should inject in a way that bypasses and filters) But I want it to be atleast 100/s Two machineS? relay boxes Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. Why not inject to the further relay boxes? Do I need to increase the hardware It could be :) Other options: increase input concurrency, or play with in_flow_delay. Note that increasing your input rates will cause output rates to drop. It's all about competing for disk access. Wietse Further options, I think: - Disable filtering (provided the only possible connections are related to these emails - put the queue on a ram-disk (8GB Ram, might leave 6GB for the queue, would this be sufficient?) These are theoretical, I have no idea if this is at all possible and if this can cause further issues elsewhere? -- Joost
Re: Speed up queue injection
On 8/13/2010 8:22 AM, J. Roeleveld wrote: On Friday 13 August 2010 14:23:51 Wietse Venema wrote: Ralf Hildebrandt: * Ramr...@netcore.co.in: Mail in plain text format , mime encoded message OK! Currenlty I get 40/s - 45/s That sounds normal. Any filtering (in these cases you should inject in a way that bypasses and filters) But I want it to be atleast 100/s Two machineS? relay boxes Delivery is not at all an issue , because postfix gives it to further relay boxes which are under our control again. Why not inject to the further relay boxes? Do I need to increase the hardware It could be :) Other options: increase input concurrency, or play with in_flow_delay. Note that increasing your input rates will cause output rates to drop. It's all about competing for disk access. Wietse Further options, I think: - Disable filtering (provided the only possible connections are related to these emails Presumably the client would be in mynetworks, which should bypass most or all restrictions, so this is unlikely to make much difference. Unless you're doing something silly like 1000 body_check rules or using a content_filter or milter. - put the queue on a ram-disk (8GB Ram, might leave 6GB for the queue, would this be sufficient?) Putting the queue on ramdisk is only for spammers who don't particularly care if their mail is lost. But putting the queue on an enterprise-quality SSD would almost certainly help. -- Noel Jones
Speed up queue injection
We have a requirement to send some research analysis mails as quickly as possible. Everyday after the data is available my app generates the mails in eml format in a directory. These are personalized mails with attachments and have to reach the recipients instantly ( in my customers lingo ... Research reports are useless after the market opens ) What is the quickest way of pushing EML files to postfix for delivery. Currently I have a perl script that makes parallel smtp connections on localhost and sends the mails. Should I send the mails on command line. There are currently around 50k mails to be delivered ideally within 5-10 mins. I am only bothered about sending to postfix because delievery from there is already taken care of. Is there a better way , other than sending mails on command line or SMTP. Something like and API to inject into postfix maildrop. Thanks Ram
