[sniffer] Re: IP Change on rulebase delivery system

2013-05-24 Thread Richard Stupek
Pete
I thought the local gbudb got updates from the service or was that a future
enhancement?

  Original message 
 Subject: [sniffer] Re: IP Change on rulebase delivery system
 From: Richard Stupek rstu...@gmail.com javascript:_e({}, 'cvml',
 'rstu...@gmail.com');
 To: Message Sniffer Community sniffer@sortmonster.com javascript:_e({},
 'cvml', 'sniffer@sortmonster.com');
 CC:


 Can you point me at the documentation for the truncate blacklist and its
 usage?


 On Thu, May 23, 2013 at 3:36 PM, Pete McNeil 
 madscient...@armresearch.comjavascript:_e({}, 'cvml', 
 'madscient...@armresearch.com');
  wrote:

 On 2013-05-23 15:22, Richard Stupek wrote:

 Looks like I have this issue again (pegging 4 core cpu) and resetting
 the process doesn't make a difference.  Not sure what is causing it but it
 does slow down spam detection to 40-50 seconds for many emails.  Any ideas
 what I can look at or do to resolve this?


 Check the message sizes. As part of the newest spam storms we've noticed
 that a lot of the messages are huge (65536++). I suspect this might impact
 throughput as large buffers are allocated and moved around to handle these
 messages. This kind of thing has also been known to cause NTFS to crawl.

 Please let us know what you find.

 If you are not already doing it -- you should consider blocking
 connections using the truncate blacklist. No sense taking on some of these
 messages if they can be eliminated up front.


 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com javascript:_e({}, 'cvml',
 'sniffer@sortmonster.com');.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.comjavascript:_e({}, 
 'cvml', 'sniffer-...@sortmonster.com');
 
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comjavascript:_e({}, 'cvml', 
 'sniffer-dig...@sortmonster.com');
 
 To switch to the INDEX mode, E-mail to 
 sniffer-in...@sortmonster.comjavascript:_e({}, 'cvml', 
 'sniffer-in...@sortmonster.com');
 **
 Send administrative queries to  
 sniffer-request@sortmonster.**comjavascript:_e({}, 'cvml', 
 'sniffer-requ...@sortmonster.com');
 





[sniffer] Re: IP Change on rulebase delivery system

2013-05-24 Thread Pete McNeil

On 2013-05-24 08:38, Richard Stupek wrote:

Pete
I thought the local gbudb got updates from the service or was that a 
future enhancement?


That's true right now. GBUdb is part of a distributed machine learning 
system. There is a conversation going on between all SNF nodes where 
they share their point of view on IP reputations. This happens 
approximately once per minute, out of band.


Each node alerts the system that they have new activity on a given IP. 
Then, via the SYNC server(s), each node receives a reflection of the 
consensus on that IP. So, when an IP is new to a node it will be updated 
within about a minute with the consensus reputation from the other 
nodes. As there are more interactions, the consensus matters less and 
the local experiences matter more -- but the conversation continues so 
the each node is always influencing the other nodes about any active IPs.


The conversation protocols are intelligent so that there is just enough 
traffic to accomplish the learning goals and so that a hostile / 
compromised node cannot poison the system; and so that each node can 
maintain it's own point of view about each IP.


For example: Say node A regularly corresponds with an ISP in 
blackhatistan. So, node A sees a mixture of good and bad messages. Node 
B only gets bad messages from the same ISP. Node A will have a local 
reputation for the ISP that is good enough to let messages through on 
that system, but node B will have a local reputation for the ISP that 
blocks most messages. The consensus of all GBUdb nodes will be somewhere 
in between.


Hope this helps,

_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-24 Thread johnlist
Sorry, but CommTouch is not a viable cost effective solution for most of us 
anymore.

If you are a long time Imail user/admin you should be aware of this.

-Original Message-
From: DFN systems off...@dfn.com
Sent: Thursday, May 23, 2013 5:30pm
To: Message Sniffer Community sniffer@sortmonster.com
Subject: [sniffer] Re: IP Change on rulebase delivery system

Commtouch


Ryan Bair

 Original message 
Subject: [sniffer] Re: IP Change on rulebase delivery system 
From: Richard Stupek rstu...@gmail.com 
To: Message Sniffer Community sniffer@sortmonster.com 
CC:  

Can you point me at the documentation for the truncate blacklist and its usage?


On Thu, May 23, 2013 at 3:36 PM, Pete McNeil madscient...@armresearch.com 
wrote:
On 2013-05-23 15:22, Richard Stupek wrote:
Looks like I have this issue again (pegging 4 core cpu) and resetting the 
process doesn't make a difference.  Not sure what is causing it but it does 
slow down spam detection to 40-50 seconds for many emails.  Any ideas what I 
can look at or do to resolve this?

Check the message sizes. As part of the newest spam storms we've noticed that a 
lot of the messages are huge (65536++). I suspect this might impact throughput 
as large buffers are allocated and moved around to handle these messages. This 
kind of thing has also been known to cause NTFS to crawl.

Please let us know what you find.

If you are not already doing it -- you should consider blocking connections 
using the truncate blacklist. No sense taking on some of these messages if they 
can be eliminated up front.


_M

-- 
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com





#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Richard Stupek
Looks like I have this issue again (pegging 4 core cpu) and resetting the
process doesn't make a difference.  Not sure what is causing it but it does
slow down spam detection to 40-50 seconds for many emails.  Any ideas what
I can look at or do to resolve this?


On Fri, Mar 29, 2013 at 12:27 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-03-29 12:59, Richard Stupek wrote:

 well when all else fails restarting snf seems to have corrected the issue
 for now.


 In that case, it is likely that RAM fragmentation was involved. Dropping
 the process allowed the fragmentation to be cleared. (theory).


 Best,
 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Greg Coffey
I've been blocking subnets to the mail server manually for the past 10 days or 
so.  Scan the logs and look at common IP sources for spam.  PITA but I've got 
it under control.  One of the earlier schemes I noticed was from .pw and .in 
top level domains.  What I'm seeing now are messages coming from assorted 
domains but from a common subnet and hosting company - some US based.  I've had 
mail queued up for 20-30 mins before delivery before adding some firewall 
rules.  My mail server is an i5 running Windows Server.  

-- Original Message --
From: Richard Stupek rstu...@gmail.com
Reply-To: Message Sniffer Community sniffer@sortmonster.com
Date:  Thu, 23 May 2013 14:22:59 -0500

Looks like I have this issue again (pegging 4 core cpu) and resetting the
process doesn't make a difference.  Not sure what is causing it but it does
slow down spam detection to 40-50 seconds for many emails.  Any ideas what
I can look at or do to resolve this?


On Fri, Mar 29, 2013 at 12:27 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-03-29 12:59, Richard Stupek wrote:

 well when all else fails restarting snf seems to have corrected the issue
 for now.


 In that case, it is likely that RAM fragmentation was involved. Dropping
 the process allowed the fragmentation to be cleared. (theory).


 Best,
 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 





--
Thanks, Greg

AllureTech/CoffeyNet  www.atwy.net
1546 E Burlington Ave
Casper, WY  82601 307.473.2323
--

#
This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Pete McNeil

On 2013-05-23 15:22, Richard Stupek wrote:
Looks like I have this issue again (pegging 4 core cpu) and resetting 
the process doesn't make a difference.  Not sure what is causing it 
but it does slow down spam detection to 40-50 seconds for many emails. 
 Any ideas what I can look at or do to resolve this?


Check the message sizes. As part of the newest spam storms we've noticed 
that a lot of the messages are huge (65536++). I suspect this might 
impact throughput as large buffers are allocated and moved around to 
handle these messages. This kind of thing has also been known to cause 
NTFS to crawl.


Please let us know what you find.

If you are not already doing it -- you should consider blocking 
connections using the truncate blacklist. No sense taking on some of 
these messages if they can be eliminated up front.


_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Richard Stupek
Can you point me at the documentation for the truncate blacklist and its
usage?


On Thu, May 23, 2013 at 3:36 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-05-23 15:22, Richard Stupek wrote:

 Looks like I have this issue again (pegging 4 core cpu) and resetting the
 process doesn't make a difference.  Not sure what is causing it but it does
 slow down spam detection to 40-50 seconds for many emails.  Any ideas what
 I can look at or do to resolve this?


 Check the message sizes. As part of the newest spam storms we've noticed
 that a lot of the messages are huge (65536++). I suspect this might impact
 throughput as large buffers are allocated and moved around to handle these
 messages. This kind of thing has also been known to cause NTFS to crawl.

 Please let us know what you find.

 If you are not already doing it -- you should consider blocking
 connections using the truncate blacklist. No sense taking on some of these
 messages if they can be eliminated up front.


 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Pete McNeil

On 2013-05-23 16:41, Richard Stupek wrote:
Can you point me at the documentation for the truncate blacklist and 
its usage?

http://gbudb.com/truncate/index.jsp

It's an ordinary ip4 dnsbl.

Most email systems have some mechanism for blocking connections based on 
this kind of blacklist.


Hope this helps,

_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Richard Stupek
Would this:
http://armresearch.com/support/articles/software/snfServer/xci/gbudb.jsp yield
the same results as using the ip4 blocklist?


On Thu, May 23, 2013 at 4:11 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-05-23 16:41, Richard Stupek wrote:

 Can you point me at the documentation for the truncate blacklist and its
 usage?

 http://gbudb.com/truncate/**index.jsphttp://gbudb.com/truncate/index.jsp

 It's an ordinary ip4 dnsbl.

 Most email systems have some mechanism for blocking connections based on
 this kind of blacklist.

 Hope this helps,


 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Pete McNeil

On 2013-05-23 17:21, Richard Stupek wrote:
Would this: 
http://armresearch.com/support/articles/software/snfServer/xci/gbudb.jsp yield 
the same results as using the ip4 blocklist?


No. Asking your local GBUdb about an IP will only give you a local 
perspective.


The truncate blacklist contains the currently active worst-of-the-worst 
as seen by all SNF nodes working together.


Also -- getting your MTA to pay attention to your local GBUdb is 
nontrivial since no MTA software (that I know of) can speak XCI yet.


_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Richard Stupek
Thanks for the info.  Mine can speak XCI, its custom.


On Thu, May 23, 2013 at 4:31 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-05-23 17:21, Richard Stupek wrote:

 Would this: http://armresearch.com/**support/articles/software/**
 snfServer/xci/gbudb.jsphttp://armresearch.com/support/articles/software/snfServer/xci/gbudb.jspyield
  the same results as using the ip4 blocklist?


 No. Asking your local GBUdb about an IP will only give you a local
 perspective.

 The truncate blacklist contains the currently active worst-of-the-worst as
 seen by all SNF nodes working together.

 Also -- getting your MTA to pay attention to your local GBUdb is
 nontrivial since no MTA software (that I know of) can speak XCI yet.


 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread Pete McNeil

On 2013-05-23 17:37, Richard Stupek wrote:

Mine can speak XCI, its custom.


Kewl -- then you can use GBUdb for local IP reputation data whenever you 
like. That could be very useful.


Anything you can do with SNFClient you can do with XCI -- SNFClient is 
just a command line translator.


Since you can do that, one thing you might consider doing is to use 
GBUdb for targeted gray-listing.


_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-05-23 Thread DFN systems
Commtouch


Ryan Bair

 Original message 
Subject: [sniffer] Re: IP Change on rulebase delivery system 
From: Richard Stupek rstu...@gmail.com 
To: Message Sniffer Community sniffer@sortmonster.com 
CC:  

Can you point me at the documentation for the truncate blacklist and its usage?


On Thu, May 23, 2013 at 3:36 PM, Pete McNeil madscient...@armresearch.com 
wrote:
On 2013-05-23 15:22, Richard Stupek wrote:
Looks like I have this issue again (pegging 4 core cpu) and resetting the 
process doesn't make a difference.  Not sure what is causing it but it does 
slow down spam detection to 40-50 seconds for many emails.  Any ideas what I 
can look at or do to resolve this?

Check the message sizes. As part of the newest spam storms we've noticed that a 
lot of the messages are huge (65536++). I suspect this might impact throughput 
as large buffers are allocated and moved around to handle these messages. This 
kind of thing has also been known to cause NTFS to crawl.

Please let us know what you find.

If you are not already doing it -- you should consider blocking connections 
using the truncate blacklist. No sense taking on some of these messages if they 
can be eliminated up front.


_M

-- 
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com




[sniffer] Re: IP Change on rulebase delivery system

2013-03-29 Thread Richard Stupek
well when all else fails restarting snf seems to have corrected the issue
for now.


On Thu, Mar 28, 2013 at 11:50 AM, Darin Cox dc...@4cweb.com wrote:

   Richard,

 Do you have any directories with a large number of files (4k)?  We had a
 similar problem a few months back with sniffer scans taking much longer to
 complete and sniffer temporary files being left over.  We finally traced
 the performance issues to a frequently accessed directory with thousands of
 files.  We’ve also seen issues in the past with directories with a large
 number of files being very poor performing.

 Darin.


  *From:* Richard Stupek rstu...@gmail.com
 *Sent:* Thursday, March 28, 2013 12:10 PM
 *To:* Message Sniffer Community sniffer@sortmonster.com
 *Subject:* [sniffer] Re: IP Change on rulebase delivery system

  Ok looking at the log I see quite a few messages taking over a second to
 process (samples below):

  s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0'
 p s='1172' t='1109' l='72697' d='127'/
 g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/
 /s

  s u='20130328155506' m='\temp\1332407477336.msg' s='60' r='5113015'
 m s='60' r='5113015' i='235' e='280' f='m'/
 m s='60' r='4346940' i='16722' e='16812' f='m'/
 p s='1141' t='937' l='16658' d='129'/
 g o='0' i='192.210.233.215' t='u' c='0.360316' p='0.575758'
 r='Normal'/
 /s

  s u='20130328155513' m='\temp\1332407477360.msg' s='52' r='5470216'
 m s='52' r='5470216' i='235' e='295' f='m'/
 m s='52' r='5471910' i='949' e='1009' f='m'/
 m s='52' r='5431546' i='1074' e='1200' f='m'/
 m s='52' r='5479780' i='1857' e='1933' f='m'/
 m s='62' r='5303955' i='82' e='2688' f='m'/
 m s='52' r='5400681' i='1818' e='9143' f='m'/
 p s='1031' t='750' l='8538' d='130'/
 g o='0' i='192.210.134.21' t='u' c='0.545993' p='0.82' r='Black'/
 /s

  s u='20130328155622' m=\temp\1332407477655.msg' s='60' r='5538969'
 m s='60' r='5538969' i='221' e='236' f='m'/
 m s='61' r='5448415' i='2283' e='2297' f='m'/
 m s='61' r='5438936' i='2247' e='2337' f='m'/
 m s='60' r='5404555' i='15832' e='15850' f='m'/
 m s='60' r='5539002' i='16033' e='16074' f='m'/
 m s='62' r='5437246' i='30967' e='30985' f='m'/
 p s='1219' t='1312' l='17171' d='135'/
 g o='0' i='205.234.138.240' t='u' c='0.634697' p='0.763214'
 r='Normal'/
 /s



 On Wed, Mar 27, 2013 at 4:42 PM, Pete McNeil madscient...@armresearch.com
  wrote:

 On 2013-03-27 17:16, Richard Stupek wrote:

 The spikes aren't as prolonged at the present.


 Interesting. A short spike like that might be expected if the message was
 longer than usual, but on average SNF should be very light-weight.

 One thing you can check is the performance data in your logs. That will
 show how much time in cpu milleseconds it is taking for each scan and how
 long the scans are in bytes. This might shed some light.

 http://www.armresearch.com/**support/articles/software/**
 snfServer/logFiles/**activityLogs.jsphttp://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp

 Look for something like p s='10' t='8' l='3294' d='84'/ in each scan.

 From the documentation:

 sp//s - Scan Performance Monitoring (performance='yes')
 p:s = Setup time in milliseconds
 p:t = Scan time in milliseconds
 p:l = Scan length in bytes
 p:d = Scan depth (peak evaluator count)


 Best,


 _M


 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010 866-770-1044%20x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 





[sniffer] Re: IP Change on rulebase delivery system

2013-03-29 Thread Pete McNeil

On 2013-03-29 12:59, Richard Stupek wrote:
well when all else fails restarting snf seems to have corrected the 
issue for now.


In that case, it is likely that RAM fragmentation was involved. Dropping 
the process allowed the fragmentation to be cleared. (theory).


Best,
_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-03-28 Thread Pete McNeil

On 2013-03-28 12:10, Richard Stupek wrote:
Ok looking at the log I see quite a few messages taking over a second 
to process (samples below):


s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0'
p s='1172' t='1109' l='72697' d='127'/
g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/
/s


Great! Now we're getting somewhere.
It seems likely that your system is bound in 2 ways:

* Allocating RAM
* Reading / writing from the hard drive.

The setup time s='1172' indicates that it took more than a second to 
allocate a 72K buffer and read the message. That is the only work done 
during setup.


The scan time t='1109' indicates the amount of time it took to complete 
the rest of the process. I'm guessing that since the setup took more 
than a second that writing the message back with injected headers 
probably took a while too.


A scan depth of 127 is nominal.

The data you sent indicates this is a problem when the messages are 
fairly large. There is likely a boundary condition between smaller 
messages and larger ones that allow the smaller messages to be handled 
more efficiently.


Since you are indicating that CPU utilization is high during these 
events and since you're not mentioning other performance issues, it 
seems likely that RAM fragmentation or RAM starvation might be the 
problem here.


During the setup, SNF allocates a buffer large enough for the entire 
message and then reads it all at once. This is most efficient because 
there is a single system call for each of these events - so the OS has 
complete control over the entire step and it only happens once.


After the scanning process is done, SNF will typically allocate another 
buffer large enough to include the message and all of it's headers. This 
new version of the message is constructed in the buffer and then written 
all at once to the file system.


If the problem is RAM fragmentation or starvation then it could be 
taking as much as a second for the OS to allocate a 72K buffer --- 
that's the kind of thing that should be nearly undetectable, but in 
these cases where high CPU loads are reported with this kind of log data 
I have seen that happen.


If the problem were simply IO then it is likely that CPU utilization 
would be low while the CPUs wait for the IO operations to happen.


Of course it could be a combination of things -- it's hard to tell 
what's happening in the internals of the OS.


Hope this helps,

_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-03-28 Thread Darin Cox
Richard,

Do you have any directories with a large number of files (4k)?  We had a 
similar problem a few months back with sniffer scans taking much longer to 
complete and sniffer temporary files being left over.  We finally traced the 
performance issues to a frequently accessed directory with thousands of 
files.  We’ve also seen issues in the past with directories with a large 
number of files being very poor performing.

Darin.



From: Richard Stupek
Sent: Thursday, March 28, 2013 12:10 PM
To: Message Sniffer Community
Subject: [sniffer] Re: IP Change on rulebase delivery system

Ok looking at the log I see quite a few messages taking over a second to 
process (samples below):

s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0'
p s='1172' t='1109' l='72697' d='127'/
g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/
/s

s u='20130328155506' m='\temp\1332407477336.msg' s='60' r='5113015'
m s='60' r='5113015' i='235' e='280' f='m'/
m s='60' r='4346940' i='16722' e='16812' f='m'/
p s='1141' t='937' l='16658' d='129'/
g o='0' i='192.210.233.215' t='u' c='0.360316' p='0.575758' 
r='Normal'/
/s

s u='20130328155513' m='\temp\1332407477360.msg' s='52' r='5470216'
m s='52' r='5470216' i='235' e='295' f='m'/
m s='52' r='5471910' i='949' e='1009' f='m'/
m s='52' r='5431546' i='1074' e='1200' f='m'/
m s='52' r='5479780' i='1857' e='1933' f='m'/
m s='62' r='5303955' i='82' e='2688' f='m'/
m s='52' r='5400681' i='1818' e='9143' f='m'/
p s='1031' t='750' l='8538' d='130'/
g o='0' i='192.210.134.21' t='u' c='0.545993' p='0.82' r='Black'/
/s

s u='20130328155622' m=\temp\1332407477655.msg' s='60' r='5538969'
m s='60' r='5538969' i='221' e='236' f='m'/
m s='61' r='5448415' i='2283' e='2297' f='m'/
m s='61' r='5438936' i='2247' e='2337' f='m'/
m s='60' r='5404555' i='15832' e='15850' f='m'/
m s='60' r='5539002' i='16033' e='16074' f='m'/
m s='62' r='5437246' i='30967' e='30985' f='m'/
p s='1219' t='1312' l='17171' d='135'/
g o='0' i='205.234.138.240' t='u' c='0.634697' p='0.763214' 
r='Normal'/
/s




On Wed, Mar 27, 2013 at 4:42 PM, Pete McNeil madscient...@armresearch.com 
wrote:

  On 2013-03-27 17:16, Richard Stupek wrote:

The spikes aren't as prolonged at the present.



  Interesting. A short spike like that might be expected if the message was 
longer than usual, but on average SNF should be very light-weight.

  One thing you can check is the performance data in your logs. That will 
show how much time in cpu milleseconds it is taking for each scan and how 
long the scans are in bytes. This might shed some light.

  
http://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp

  Look for something like p s='10' t='8' l='3294' d='84'/ in each scan.

  From the documentation:


sp//s - Scan Performance Monitoring (performance='yes')
p:s = Setup time in milliseconds
p:t = Scan time in milliseconds
p:l = Scan length in bytes
p:d = Scan depth (peak evaluator count)



  Best,


  _M


  -- 
  Pete McNeil
  Chief Scientist
  ARM Research Labs, LLC
  www.armresearch.com
  866-770-1044 x7010
  twitter/codedweller


  #
  This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
  This list is for discussing Message Sniffer,
  Anti-spam, Anti-Malware, and related email topics.
  For More information see http://www.armresearch.com
  To unsubscribe, E-mail to: sniffer-...@sortmonster.com
  To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
  To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
  Send administrative queries to  sniffer-requ...@sortmonster.com




[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Richard Stupek
Not sure if its related but since yesterday SNFserver CPU utilization has
been inordinately high (50%) for the middle of the day with not any
additional volume in mail being received.

On Mon, Mar 25, 2013 at 9:13 AM, Pete McNeil
madscient...@armresearch.comwrote:

 Hi Sniffer Folks,

 We are about to change the IP of the rulebase delivery system. This change
 should be completely transparent and you should not need to take any
 action; however if you do notice anything unusual please let us know.

 Thanks,

 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Darin Cox
Probably unrelated... and due to a significant increase in spam over the 
past few days.

Darin.



From: Richard Stupek
Sent: Wednesday, March 27, 2013 2:18 PM
To: Message Sniffer Community
Subject: [sniffer] Re: IP Change on rulebase delivery system

Not sure if its related but since yesterday SNFserver CPU utilization has 
been inordinately high (50%) for the middle of the day with not any 
additional volume in mail being received.


On Mon, Mar 25, 2013 at 9:13 AM, Pete McNeil madscient...@armresearch.com 
wrote:

  Hi Sniffer Folks,

  We are about to change the IP of the rulebase delivery system. This change 
should be completely transparent and you should not need to take any action; 
however if you do notice anything unusual please let us know.

  Thanks,

  _M

  -- 
  Pete McNeil
  Chief Scientist
  ARM Research Labs, LLC
  www.armresearch.com
  866-770-1044 x7010
  twitter/codedweller


  #
  This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
  This list is for discussing Message Sniffer,
  Anti-spam, Anti-Malware, and related email topics.
  For More information see http://www.armresearch.com
  To unsubscribe, E-mail to: sniffer-...@sortmonster.com
  To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
  To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
  Send administrative queries to  sniffer-requ...@sortmonster.com




[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Pete McNeil

On 2013-03-27 14:38, Darin Cox wrote:
Probably unrelated... and due to a significant increase in spam over 
the past few days.


I agree with that -- our inbound spamtrap pre-processor has seen 4x 
increase over the past few days so that's likely to be related.


Also, Richard, I took a quick look at your telemetry and verified that 
your rulebase file(s) are up to date.


Best,

_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Richard Stupek
Its odd because the number of messags snf is processing isn't more than
usual and the % of spam being detected through snf is actually lower than
typical yet is is routinely maxing out 4 processors at 100%.

On Wed, Mar 27, 2013 at 3:20 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-03-27 14:38, Darin Cox wrote:

 Probably unrelated... and due to a significant increase in spam over the
 past few days.


 I agree with that -- our inbound spamtrap pre-processor has seen 4x
 increase over the past few days so that's likely to be related.

 Also, Richard, I took a quick look at your telemetry and verified that
 your rulebase file(s) are up to date.

 Best,


 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Pete McNeil

On 2013-03-27 16:49, Richard Stupek wrote:
Its odd because the number of messags snf is processing isn't more 
than usual and the % of spam being detected through snf is actually 
lower than typical yet is is routinely maxing out 4 processors at 100%.


You're saying that SNF is maxing out 4 processors? ... or is the 
combination of operations on your server maxing out 4 processors?


We're using the same engine and ruelbase in our CGP server and humming 
along nicely at between 2000 - 8000 msg/minute with nominal CPU loads.


I don't see anything unusual in your telemetry and I haven't heard any 
other complaints, so I can't explain why SNF would act differently on 
your system. I hate a mystery though -- so I would love to get to the 
bottom of it.


Do you see anything else that might be causing the CPU load?

_M

--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com



[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Richard Stupek
It would be SNF routinely showing 80% utilization spikes for a 4 cpu
system. I hadn't ever seen it do that before which was why I sent the
message.  Don't believe the load is any higher than normal.  The spikes
aren't as prolonged at the present.

On Wed, Mar 27, 2013 at 4:08 PM, Pete McNeil
madscient...@armresearch.comwrote:

 On 2013-03-27 16:49, Richard Stupek wrote:

 Its odd because the number of messags snf is processing isn't more than
 usual and the % of spam being detected through snf is actually lower than
 typical yet is is routinely maxing out 4 processors at 100%.


 You're saying that SNF is maxing out 4 processors? ... or is the
 combination of operations on your server maxing out 4 processors?

 We're using the same engine and ruelbase in our CGP server and humming
 along nicely at between 2000 - 8000 msg/minute with nominal CPU loads.

 I don't see anything unusual in your telemetry and I haven't heard any
 other complaints, so I can't explain why SNF would act differently on your
 system. I hate a mystery though -- so I would love to get to the bottom of
 it.

 Do you see anything else that might be causing the CPU load?


 _M

 --
 Pete McNeil
 Chief Scientist
 ARM Research Labs, LLC
 www.armresearch.com
 866-770-1044 x7010
 twitter/codedweller


 ##**##**#
 This message is sent to you because you are subscribed to
  the mailing list sniffer@sortmonster.com.
 This list is for discussing Message Sniffer,
 Anti-spam, Anti-Malware, and related email topics.
 For More information see http://www.armresearch.com
 To unsubscribe, E-mail to: sniffer-...@sortmonster.com
 To switch to the DIGEST mode, E-mail to 
 sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com
 
 To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com**
 Send administrative queries to  
 sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
 




[sniffer] Re: IP Change on rulebase delivery system

2013-03-27 Thread Pete McNeil

On 2013-03-27 17:16, Richard Stupek wrote:

The spikes aren't as prolonged at the present.


Interesting. A short spike like that might be expected if the message 
was longer than usual, but on average SNF should be very light-weight.


One thing you can check is the performance data in your logs. That will 
show how much time in cpu milleseconds it is taking for each scan and 
how long the scans are in bytes. This might shed some light.


http://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp

Look for something like p s='10' t='8' l='3294' d='84'/ in each scan.

From the documentation:


sp//s - Scan Performance Monitoring (performance='yes')
p:s = Setup time in milliseconds
p:t = Scan time in milliseconds
p:l = Scan length in bytes
p:d = Scan depth (peak evaluator count)



Best,

_M


--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller


#
This message is sent to you because you are subscribed to
 the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to  sniffer-requ...@sortmonster.com