RE: [sniffer] Moving Sniffer to Declude/SmarterMail
Pete OK, I now have much more information on this problem with Declude/Sniffer/SmarterMail. It seems the current version of Declude does not have an Overflow Directory for SmarterMail, which therefore allows unlimited Declude processes to be spawned at any time. At our peak we were seeing a surge of more than 1,000 declude.exe instances running at the same time! This of course flattened the server, and seems the reason why Sniffer was dropping out of its perpetual mode, unfortunately compounding the problem when the server had least resources. On speaking with Declude, thankfully they have been working on a version to control the number of processes running, and have let us have a BETA version which allows us to set the number of processes to a setting of our choice - we have it at 30 and it's working fine! One thing we did whilst in the middle of this was to move all the log and spool files to a standalone disk instead of the RAID5 array for the main server, and we have seen a real reduction in the physical disk queue lengths, which, under significant load, helps. Worth knowing. So the migration is complete with all users running as normal on SmarterMail instead of iMail. Hope some people can learn from our pain! Nick -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: 14 March 2005 19:23 To: Nick Marshall Subject: Re: [sniffer] Moving Sniffer to Declude/SmarterMail On Monday, March 14, 2005, 12:47:33 PM, Nick wrote: NM Hi there NM We've just undergone a migration of a 1,000 domain iMail server to NM SmarterMail (for obvious reasons!), and using Declude and Sniffer on NM the new system. NM However, occasionally we see Sniffer jumping out of its perpetual NM mode by spawning thousands of threads which obviously make the NM server slow down considerably. Also, at this time, the Sniffer NM folder fills up with the equivalent temporary files. NM On changing the location of Sniffer in the Declude Global file and NM allowing the threads to disappear, when Sniffer is re-engaged it NM correctly goes back to perpetual mode working from the Service instance of Sniffer. NM Can any of the above be explained? This is extremely unusual and I've not heard of this kind of thing before. The closest connection I have seen is that on one very heavily loaded system (on Linux) a change in the rulebase file caused the persistent instance to pause for a few minutes while the message instances of SNF became impatient --- however in this case the system always recovered on it's own. In your case do you restart the persistent instance or does the system simply recover once you re-enable the clients? Please post the contents of your .stat file and let me know if these are typical values. Does your system have one or two CPUs? Hyperthreading? Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- [This e-mail was scanned for viruses by Giacom Anti-Virus] -- [This e-mail was scanned for viruses by Giacom Anti-Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Moving Sniffer to Declude/SmarterMail
Thanks John - I didn't know that, but it would explain things... Nick -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Tolmachoff (Lists) Sent: 16 March 2005 14:40 To: sniffer@SortMonster.com Subject: RE: [sniffer] Moving Sniffer to Declude/SmarterMail One thing we did whilst in the middle of this was to move all the log and spool files to a standalone disk instead of the RAID5 array for the main server, and we have seen a real reduction in the physical disk queue lengths, which, under significant load, helps. Worth knowing. Nick It is a well known and published fact (on the Imail list) that RAID5 should never ever be used for the spool directory or any other directory that has a high write activity. This is basic physics. RAID5 should really only be used for high read activity only, such as databases where most of the writing is done to transaction (log) files and at spaced intervals those transactions are committed to the database. RAID1 or even RAID0+1 is best for the spool and logs. John Tolmachoff Engineer/Consultant/Owner eServices For You This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- [This e-mail was scanned for viruses by Giacom Anti-Virus] -- [This e-mail was scanned for viruses by Giacom Anti-Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Moving Sniffer to Declude/SmarterMail
John, It is a well known and published fact (on the Imail list) that RAID5 should never ever be used for the spool directory or any other directory that has a high write activity. This is basic physics. RAID5 should really only be used for high read activity only, such as databases where most of the writing is done to transaction (log) files and at spaced intervals those transactions are committed to the database. RAID1 or even RAID0+1 is best for the spool and logs. I guess this is going against what I think should be happening. In a RAID 5 array the write to the drives is broken into many smaller writes along with the data protection/CRC info and then those writes are written to different drives. It seems to me that it should be faster to do a bunch of small writes rather than 1 big write. What am I missing? Goran Jovanovic The LAN Shoppe This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Moving Sniffer to Declude/SmarterMail
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: 16. marts 2005 17:43 Writing data to a raid 5 takes x+y+z amount of work where y is described above and z is calculating a CRC stripe which must now also be saved to a hard drive. And from my understanding, calculating the CRC requires knowing both the data in the sector overwritten and the previous CRC data, and thus 2 reads. So thats 2 reads, and 2 writes to write a single block of data. Regards, Kaj This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Moving Sniffer to Declude/SmarterMail
On Monday, March 14, 2005, 12:47:33 PM, Nick wrote: NM Hi there NM We've just undergone a migration of a 1,000 domain iMail server to NM SmarterMail (for obvious reasons!), and using Declude and Sniffer on the new NM system. NM However, occasionally we see Sniffer jumping out of its perpetual mode by NM spawning thousands of threads which obviously make the server slow down NM considerably. Also, at this time, the Sniffer folder fills up with the NM equivalent temporary files. NM On changing the location of Sniffer in the Declude Global file and allowing NM the threads to disappear, when Sniffer is re-engaged it correctly goes back NM to perpetual mode working from the Service instance of Sniffer. NM Can any of the above be explained? This is extremely unusual and I've not heard of this kind of thing before. The closest connection I have seen is that on one very heavily loaded system (on Linux) a change in the rulebase file caused the persistent instance to pause for a few minutes while the message instances of SNF became impatient --- however in this case the system always recovered on it's own. In your case do you restart the persistent instance or does the system simply recover once you re-enable the clients? Please post the contents of your .stat file and let me know if these are typical values. Does your system have one or two CPUs? Hyperthreading? Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
