[sniffer] Re: IP Change on rulebase delivery system
Pete
I thought the local gbudb got updates from the service or was that a future
enhancement?
Original message
Subject: [sniffer] Re: IP Change on rulebase delivery system
From: Richard Stupek rstu...@gmail.com javascript:_e({}, 'cvml',
'rstu...@gmail.com');
To: Message Sniffer Community sniffer@sortmonster.com javascript:_e({},
'cvml', 'sniffer@sortmonster.com');
CC:
Can you point me at the documentation for the truncate blacklist and its
usage?
On Thu, May 23, 2013 at 3:36 PM, Pete McNeil
madscient...@armresearch.comjavascript:_e({}, 'cvml',
'madscient...@armresearch.com');
wrote:
On 2013-05-23 15:22, Richard Stupek wrote:
Looks like I have this issue again (pegging 4 core cpu) and resetting
the process doesn't make a difference. Not sure what is causing it but it
does slow down spam detection to 40-50 seconds for many emails. Any ideas
what I can look at or do to resolve this?
Check the message sizes. As part of the newest spam storms we've noticed
that a lot of the messages are huge (65536++). I suspect this might impact
throughput as large buffers are allocated and moved around to handle these
messages. This kind of thing has also been known to cause NTFS to crawl.
Please let us know what you find.
If you are not already doing it -- you should consider blocking
connections using the truncate blacklist. No sense taking on some of these
messages if they can be eliminated up front.
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller
##**##**#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com javascript:_e({}, 'cvml',
'sniffer@sortmonster.com');.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.comjavascript:_e({},
'cvml', 'sniffer-...@sortmonster.com');
To switch to the DIGEST mode, E-mail to
sniffer-digest@sortmonster.**comjavascript:_e({}, 'cvml',
'sniffer-dig...@sortmonster.com');
To switch to the INDEX mode, E-mail to
sniffer-in...@sortmonster.comjavascript:_e({}, 'cvml',
'sniffer-in...@sortmonster.com');
**
Send administrative queries to
sniffer-request@sortmonster.**comjavascript:_e({}, 'cvml',
'sniffer-requ...@sortmonster.com');
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-24 08:38, Richard Stupek wrote: Pete I thought the local gbudb got updates from the service or was that a future enhancement? That's true right now. GBUdb is part of a distributed machine learning system. There is a conversation going on between all SNF nodes where they share their point of view on IP reputations. This happens approximately once per minute, out of band. Each node alerts the system that they have new activity on a given IP. Then, via the SYNC server(s), each node receives a reflection of the consensus on that IP. So, when an IP is new to a node it will be updated within about a minute with the consensus reputation from the other nodes. As there are more interactions, the consensus matters less and the local experiences matter more -- but the conversation continues so the each node is always influencing the other nodes about any active IPs. The conversation protocols are intelligent so that there is just enough traffic to accomplish the learning goals and so that a hostile / compromised node cannot poison the system; and so that each node can maintain it's own point of view about each IP. For example: Say node A regularly corresponds with an ISP in blackhatistan. So, node A sees a mixture of good and bad messages. Node B only gets bad messages from the same ISP. Node A will have a local reputation for the ISP that is good enough to let messages through on that system, but node B will have a local reputation for the ISP that blocks most messages. The consensus of all GBUdb nodes will be somewhere in between. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Sorry, but CommTouch is not a viable cost effective solution for most of us anymore. If you are a long time Imail user/admin you should be aware of this. -Original Message- From: DFN systems off...@dfn.com Sent: Thursday, May 23, 2013 5:30pm To: Message Sniffer Community sniffer@sortmonster.com Subject: [sniffer] Re: IP Change on rulebase delivery system Commtouch Ryan Bair Original message Subject: [sniffer] Re: IP Change on rulebase delivery system From: Richard Stupek rstu...@gmail.com To: Message Sniffer Community sniffer@sortmonster.com CC: Can you point me at the documentation for the truncate blacklist and its usage? On Thu, May 23, 2013 at 3:36 PM, Pete McNeil madscient...@armresearch.com wrote: On 2013-05-23 15:22, Richard Stupek wrote: Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? Check the message sizes. As part of the newest spam storms we've noticed that a lot of the messages are huge (65536++). I suspect this might impact throughput as large buffers are allocated and moved around to handle these messages. This kind of thing has also been known to cause NTFS to crawl. Please let us know what you find. If you are not already doing it -- you should consider blocking connections using the truncate blacklist. No sense taking on some of these messages if they can be eliminated up front. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? On Fri, Mar 29, 2013 at 12:27 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-29 12:59, Richard Stupek wrote: well when all else fails restarting snf seems to have corrected the issue for now. In that case, it is likely that RAM fragmentation was involved. Dropping the process allowed the fragmentation to be cleared. (theory). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
I've been blocking subnets to the mail server manually for the past 10 days or so. Scan the logs and look at common IP sources for spam. PITA but I've got it under control. One of the earlier schemes I noticed was from .pw and .in top level domains. What I'm seeing now are messages coming from assorted domains but from a common subnet and hosting company - some US based. I've had mail queued up for 20-30 mins before delivery before adding some firewall rules. My mail server is an i5 running Windows Server. -- Original Message -- From: Richard Stupek rstu...@gmail.com Reply-To: Message Sniffer Community sniffer@sortmonster.com Date: Thu, 23 May 2013 14:22:59 -0500 Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? On Fri, Mar 29, 2013 at 12:27 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-29 12:59, Richard Stupek wrote: well when all else fails restarting snf seems to have corrected the issue for now. In that case, it is likely that RAM fragmentation was involved. Dropping the process allowed the fragmentation to be cleared. (theory). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com -- Thanks, Greg AllureTech/CoffeyNet www.atwy.net 1546 E Burlington Ave Casper, WY 82601 307.473.2323 -- # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-23 15:22, Richard Stupek wrote: Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? Check the message sizes. As part of the newest spam storms we've noticed that a lot of the messages are huge (65536++). I suspect this might impact throughput as large buffers are allocated and moved around to handle these messages. This kind of thing has also been known to cause NTFS to crawl. Please let us know what you find. If you are not already doing it -- you should consider blocking connections using the truncate blacklist. No sense taking on some of these messages if they can be eliminated up front. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Can you point me at the documentation for the truncate blacklist and its usage? On Thu, May 23, 2013 at 3:36 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-05-23 15:22, Richard Stupek wrote: Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? Check the message sizes. As part of the newest spam storms we've noticed that a lot of the messages are huge (65536++). I suspect this might impact throughput as large buffers are allocated and moved around to handle these messages. This kind of thing has also been known to cause NTFS to crawl. Please let us know what you find. If you are not already doing it -- you should consider blocking connections using the truncate blacklist. No sense taking on some of these messages if they can be eliminated up front. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-23 16:41, Richard Stupek wrote: Can you point me at the documentation for the truncate blacklist and its usage? http://gbudb.com/truncate/index.jsp It's an ordinary ip4 dnsbl. Most email systems have some mechanism for blocking connections based on this kind of blacklist. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Would this: http://armresearch.com/support/articles/software/snfServer/xci/gbudb.jsp yield the same results as using the ip4 blocklist? On Thu, May 23, 2013 at 4:11 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-05-23 16:41, Richard Stupek wrote: Can you point me at the documentation for the truncate blacklist and its usage? http://gbudb.com/truncate/**index.jsphttp://gbudb.com/truncate/index.jsp It's an ordinary ip4 dnsbl. Most email systems have some mechanism for blocking connections based on this kind of blacklist. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-23 17:21, Richard Stupek wrote: Would this: http://armresearch.com/support/articles/software/snfServer/xci/gbudb.jsp yield the same results as using the ip4 blocklist? No. Asking your local GBUdb about an IP will only give you a local perspective. The truncate blacklist contains the currently active worst-of-the-worst as seen by all SNF nodes working together. Also -- getting your MTA to pay attention to your local GBUdb is nontrivial since no MTA software (that I know of) can speak XCI yet. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
well when all else fails restarting snf seems to have corrected the issue for now. On Thu, Mar 28, 2013 at 11:50 AM, Darin Cox dc...@4cweb.com wrote: Richard, Do you have any directories with a large number of files (4k)? We had a similar problem a few months back with sniffer scans taking much longer to complete and sniffer temporary files being left over. We finally traced the performance issues to a frequently accessed directory with thousands of files. We’ve also seen issues in the past with directories with a large number of files being very poor performing. Darin. *From:* Richard Stupek rstu...@gmail.com *Sent:* Thursday, March 28, 2013 12:10 PM *To:* Message Sniffer Community sniffer@sortmonster.com *Subject:* [sniffer] Re: IP Change on rulebase delivery system Ok looking at the log I see quite a few messages taking over a second to process (samples below): s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0' p s='1172' t='1109' l='72697' d='127'/ g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/ /s s u='20130328155506' m='\temp\1332407477336.msg' s='60' r='5113015' m s='60' r='5113015' i='235' e='280' f='m'/ m s='60' r='4346940' i='16722' e='16812' f='m'/ p s='1141' t='937' l='16658' d='129'/ g o='0' i='192.210.233.215' t='u' c='0.360316' p='0.575758' r='Normal'/ /s s u='20130328155513' m='\temp\1332407477360.msg' s='52' r='5470216' m s='52' r='5470216' i='235' e='295' f='m'/ m s='52' r='5471910' i='949' e='1009' f='m'/ m s='52' r='5431546' i='1074' e='1200' f='m'/ m s='52' r='5479780' i='1857' e='1933' f='m'/ m s='62' r='5303955' i='82' e='2688' f='m'/ m s='52' r='5400681' i='1818' e='9143' f='m'/ p s='1031' t='750' l='8538' d='130'/ g o='0' i='192.210.134.21' t='u' c='0.545993' p='0.82' r='Black'/ /s s u='20130328155622' m=\temp\1332407477655.msg' s='60' r='5538969' m s='60' r='5538969' i='221' e='236' f='m'/ m s='61' r='5448415' i='2283' e='2297' f='m'/ m s='61' r='5438936' i='2247' e='2337' f='m'/ m s='60' r='5404555' i='15832' e='15850' f='m'/ m s='60' r='5539002' i='16033' e='16074' f='m'/ m s='62' r='5437246' i='30967' e='30985' f='m'/ p s='1219' t='1312' l='17171' d='135'/ g o='0' i='205.234.138.240' t='u' c='0.634697' p='0.763214' r='Normal'/ /s On Wed, Mar 27, 2013 at 4:42 PM, Pete McNeil madscient...@armresearch.com wrote: On 2013-03-27 17:16, Richard Stupek wrote: The spikes aren't as prolonged at the present. Interesting. A short spike like that might be expected if the message was longer than usual, but on average SNF should be very light-weight. One thing you can check is the performance data in your logs. That will show how much time in cpu milleseconds it is taking for each scan and how long the scans are in bytes. This might shed some light. http://www.armresearch.com/**support/articles/software/** snfServer/logFiles/**activityLogs.jsphttp://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp Look for something like p s='10' t='8' l='3294' d='84'/ in each scan. From the documentation: sp//s - Scan Performance Monitoring (performance='yes') p:s = Setup time in milliseconds p:t = Scan time in milliseconds p:l = Scan length in bytes p:d = Scan depth (peak evaluator count) Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 866-770-1044%20x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-29 12:59, Richard Stupek wrote: well when all else fails restarting snf seems to have corrected the issue for now. In that case, it is likely that RAM fragmentation was involved. Dropping the process allowed the fragmentation to be cleared. (theory). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-28 12:10, Richard Stupek wrote: Ok looking at the log I see quite a few messages taking over a second to process (samples below): s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0' p s='1172' t='1109' l='72697' d='127'/ g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/ /s Great! Now we're getting somewhere. It seems likely that your system is bound in 2 ways: * Allocating RAM * Reading / writing from the hard drive. The setup time s='1172' indicates that it took more than a second to allocate a 72K buffer and read the message. That is the only work done during setup. The scan time t='1109' indicates the amount of time it took to complete the rest of the process. I'm guessing that since the setup took more than a second that writing the message back with injected headers probably took a while too. A scan depth of 127 is nominal. The data you sent indicates this is a problem when the messages are fairly large. There is likely a boundary condition between smaller messages and larger ones that allow the smaller messages to be handled more efficiently. Since you are indicating that CPU utilization is high during these events and since you're not mentioning other performance issues, it seems likely that RAM fragmentation or RAM starvation might be the problem here. During the setup, SNF allocates a buffer large enough for the entire message and then reads it all at once. This is most efficient because there is a single system call for each of these events - so the OS has complete control over the entire step and it only happens once. After the scanning process is done, SNF will typically allocate another buffer large enough to include the message and all of it's headers. This new version of the message is constructed in the buffer and then written all at once to the file system. If the problem is RAM fragmentation or starvation then it could be taking as much as a second for the OS to allocate a 72K buffer --- that's the kind of thing that should be nearly undetectable, but in these cases where high CPU loads are reported with this kind of log data I have seen that happen. If the problem were simply IO then it is likely that CPU utilization would be low while the CPUs wait for the IO operations to happen. Of course it could be a combination of things -- it's hard to tell what's happening in the internals of the OS. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Richard, Do you have any directories with a large number of files (4k)? We had a similar problem a few months back with sniffer scans taking much longer to complete and sniffer temporary files being left over. We finally traced the performance issues to a frequently accessed directory with thousands of files. We’ve also seen issues in the past with directories with a large number of files being very poor performing. Darin. From: Richard Stupek Sent: Thursday, March 28, 2013 12:10 PM To: Message Sniffer Community Subject: [sniffer] Re: IP Change on rulebase delivery system Ok looking at the log I see quite a few messages taking over a second to process (samples below): s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0' p s='1172' t='1109' l='72697' d='127'/ g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/ /s s u='20130328155506' m='\temp\1332407477336.msg' s='60' r='5113015' m s='60' r='5113015' i='235' e='280' f='m'/ m s='60' r='4346940' i='16722' e='16812' f='m'/ p s='1141' t='937' l='16658' d='129'/ g o='0' i='192.210.233.215' t='u' c='0.360316' p='0.575758' r='Normal'/ /s s u='20130328155513' m='\temp\1332407477360.msg' s='52' r='5470216' m s='52' r='5470216' i='235' e='295' f='m'/ m s='52' r='5471910' i='949' e='1009' f='m'/ m s='52' r='5431546' i='1074' e='1200' f='m'/ m s='52' r='5479780' i='1857' e='1933' f='m'/ m s='62' r='5303955' i='82' e='2688' f='m'/ m s='52' r='5400681' i='1818' e='9143' f='m'/ p s='1031' t='750' l='8538' d='130'/ g o='0' i='192.210.134.21' t='u' c='0.545993' p='0.82' r='Black'/ /s s u='20130328155622' m=\temp\1332407477655.msg' s='60' r='5538969' m s='60' r='5538969' i='221' e='236' f='m'/ m s='61' r='5448415' i='2283' e='2297' f='m'/ m s='61' r='5438936' i='2247' e='2337' f='m'/ m s='60' r='5404555' i='15832' e='15850' f='m'/ m s='60' r='5539002' i='16033' e='16074' f='m'/ m s='62' r='5437246' i='30967' e='30985' f='m'/ p s='1219' t='1312' l='17171' d='135'/ g o='0' i='205.234.138.240' t='u' c='0.634697' p='0.763214' r='Normal'/ /s On Wed, Mar 27, 2013 at 4:42 PM, Pete McNeil madscient...@armresearch.com wrote: On 2013-03-27 17:16, Richard Stupek wrote: The spikes aren't as prolonged at the present. Interesting. A short spike like that might be expected if the message was longer than usual, but on average SNF should be very light-weight. One thing you can check is the performance data in your logs. That will show how much time in cpu milleseconds it is taking for each scan and how long the scans are in bytes. This might shed some light. http://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp Look for something like p s='10' t='8' l='3294' d='84'/ in each scan. From the documentation: sp//s - Scan Performance Monitoring (performance='yes') p:s = Setup time in milliseconds p:t = Scan time in milliseconds p:l = Scan length in bytes p:d = Scan depth (peak evaluator count) Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Not sure if its related but since yesterday SNFserver CPU utilization has been inordinately high (50%) for the middle of the day with not any additional volume in mail being received. On Mon, Mar 25, 2013 at 9:13 AM, Pete McNeil madscient...@armresearch.comwrote: Hi Sniffer Folks, We are about to change the IP of the rulebase delivery system. This change should be completely transparent and you should not need to take any action; however if you do notice anything unusual please let us know. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Probably unrelated... and due to a significant increase in spam over the past few days. Darin. From: Richard Stupek Sent: Wednesday, March 27, 2013 2:18 PM To: Message Sniffer Community Subject: [sniffer] Re: IP Change on rulebase delivery system Not sure if its related but since yesterday SNFserver CPU utilization has been inordinately high (50%) for the middle of the day with not any additional volume in mail being received. On Mon, Mar 25, 2013 at 9:13 AM, Pete McNeil madscient...@armresearch.com wrote: Hi Sniffer Folks, We are about to change the IP of the rulebase delivery system. This change should be completely transparent and you should not need to take any action; however if you do notice anything unusual please let us know. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-27 14:38, Darin Cox wrote: Probably unrelated... and due to a significant increase in spam over the past few days. I agree with that -- our inbound spamtrap pre-processor has seen 4x increase over the past few days so that's likely to be related. Also, Richard, I took a quick look at your telemetry and verified that your rulebase file(s) are up to date. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Its odd because the number of messags snf is processing isn't more than usual and the % of spam being detected through snf is actually lower than typical yet is is routinely maxing out 4 processors at 100%. On Wed, Mar 27, 2013 at 3:20 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-27 14:38, Darin Cox wrote: Probably unrelated... and due to a significant increase in spam over the past few days. I agree with that -- our inbound spamtrap pre-processor has seen 4x increase over the past few days so that's likely to be related. Also, Richard, I took a quick look at your telemetry and verified that your rulebase file(s) are up to date. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-27 16:49, Richard Stupek wrote: Its odd because the number of messags snf is processing isn't more than usual and the % of spam being detected through snf is actually lower than typical yet is is routinely maxing out 4 processors at 100%. You're saying that SNF is maxing out 4 processors? ... or is the combination of operations on your server maxing out 4 processors? We're using the same engine and ruelbase in our CGP server and humming along nicely at between 2000 - 8000 msg/minute with nominal CPU loads. I don't see anything unusual in your telemetry and I haven't heard any other complaints, so I can't explain why SNF would act differently on your system. I hate a mystery though -- so I would love to get to the bottom of it. Do you see anything else that might be causing the CPU load? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
It would be SNF routinely showing 80% utilization spikes for a 4 cpu system. I hadn't ever seen it do that before which was why I sent the message. Don't believe the load is any higher than normal. The spikes aren't as prolonged at the present. On Wed, Mar 27, 2013 at 4:08 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-27 16:49, Richard Stupek wrote: Its odd because the number of messags snf is processing isn't more than usual and the % of spam being detected through snf is actually lower than typical yet is is routinely maxing out 4 processors at 100%. You're saying that SNF is maxing out 4 processors? ... or is the combination of operations on your server maxing out 4 processors? We're using the same engine and ruelbase in our CGP server and humming along nicely at between 2000 - 8000 msg/minute with nominal CPU loads. I don't see anything unusual in your telemetry and I haven't heard any other complaints, so I can't explain why SNF would act differently on your system. I hate a mystery though -- so I would love to get to the bottom of it. Do you see anything else that might be causing the CPU load? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-27 17:16, Richard Stupek wrote: The spikes aren't as prolonged at the present. Interesting. A short spike like that might be expected if the message was longer than usual, but on average SNF should be very light-weight. One thing you can check is the performance data in your logs. That will show how much time in cpu milleseconds it is taking for each scan and how long the scans are in bytes. This might shed some light. http://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp Look for something like p s='10' t='8' l='3294' d='84'/ in each scan. From the documentation: sp//s - Scan Performance Monitoring (performance='yes') p:s = Setup time in milliseconds p:t = Scan time in milliseconds p:l = Scan length in bytes p:d = Scan depth (peak evaluator count) Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
