[sniffer] GBUdb
Is the GBUdb currently sharing information as described in the documentation? Do the GBUdb XCI commands detailed within snf_xci.xml work through the tcp interface?
[sniffer] Re: GBUdb
Ok. We are seeing a large amount of spam lately that is not being picked up through snf and most of it has the from and the to set the same. Are you seeing anything similar? On Thu, Dec 4, 2008 at 2:37 PM, Pete McNeil [EMAIL PROTECTED]wrote: Hello Richard, Thursday, December 4, 2008, 3:27:51 PM, you wrote: Is the GBUdb currently sharing information as described in the documentation? Yes. Do the GBUdb XCI commands detailed within snf_xci.xml work through the tcp interface? Yes. The SNFClient utility simply translates your command line parameters into XCI requests. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] GBUdb
Does the snf XML command interface for GBUdb work? I was considering pumping in bad IPs as I find them into the GBUdb and also short-circuiting spam processing by calling the GBUdb to determine the status of an IP to reduce workload. Is this something that sounds like a workable idea?
[sniffer] xci scanner command
Which of the 2 scan commands should we use to scan a message? Does sending the IP address help improve scanning? snfxciscannerscan file='filepath'//scanner/xci/snf OR snfxciscannerscan file='filepath' xhdr='no' log='no' ip='12.34.56.78'//scanner/xci/snf
[sniffer] Re: xci scanner command
So there would not be a real benefit to passing the IP over when it is the is already in the mail having been added by the mail server? On Fri, Feb 13, 2009 at 2:56 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: Which of the 2 scan commands should we use to scan a message? Does sending the IP address help improve scanning? snfxciscannerscan file='filepath'//scanner/xci/snf OR snfxciscannerscan file='filepath' xhdr='no' log='no' ip='12.34.56.78'//scanner/xci/snf That depends on your needs. If you want SNF + GBUdb to learn IPs by reading through the Received headers then you would NOT include the ip= attribute. If you want to tell SNF + GBUdb what the source IP was for the message explicitly then you DO include the ip= attribute. See: http://www.armresearch.com/support/articles/software/snfServer/xci/scanner.jsp The ip='12.34.56.78' attribute is optional and indicates that the source IP for the message has already been determined as 12.34.56.78. If this option is used then GBUdb training directives may not function as expected. This option is best used when the calling application has already determined the correct source IP for the message and wishes to force the source IP value rather than have SNF+GBUdb discover it from Received headers in the message. Note: It is often best to let SNF + GBUdb determine the source IP for a given message based on the Received headers. If the connecting IP is not already represented in the top Received header for the message then the calling application should create one in the top of the temporary file SNF is going to scan. Hope this helps, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
A question on GBUDB utilization. I show a current utilization of 95% (from the log file) which I assume means the amount of memory used from what is set aside for gbudb entries. Is that correct? What happens when more entries are added? Does the GBUdb grow or does it get pruned out? Will gbudb XCI commands (like bad) show up in a log file? On Fri, Feb 13, 2009 at 3:27 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: So there would not be a real benefit to passing the IP over when it is the is already in the mail having been added by the mail server? Correct. The vast majority of the time a properly configured SNF + GBUdb can learn the original source of the IP even if you have multiple gateways in your infrastructure. You can even teach SNF + GBUdb to learn to see past the infrastructure of other ISPs in many cases. For example you might teach it to see past a DSL provider's outbound servers so that it can map IP reputations based on individual message sources on their network provided they include Received headers you can understand and predict (to some extent). This way GBUdb can provide pinpoint accuracy instead of a rough average of every source on that network. That said, there are still some times where you might want to explicitly define the source IP even if it is present in the Received headers. For example, one of our larger customers has a complex infrastructure. They found that it was easier to explicitly provide the source IP than to train SNF + GBUdb to understand their structure and the inevitable changes that go on through time. Another large customer has developed a very complex system for determining the precise original source for a message even when it is relayed through many large ISPs. They chose to provide that IP rather than have SNF + GBUdb attempt to duplicate that effort. _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
Thanks for the info. Is there any diagnostic information available when a gbudb sync occurs? On Tue, Feb 17, 2009 at 4:35 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: A question on GBUDB utilization. I show a current utilization of 95% (from the log file) which I assume means the amount of memory used from what is set aside for gbudb entries. Is that correct? Yes. What happens when more entries are added? Does the GBUdb grow or does it get pruned out? Both. When more space is needed the ram allocated to GBUdb will grow to accommodate the need. This should happen infrequently. When entries are no longer active they are dropped to make room for additional entries. In practice the GBUdb data size stabilizes quickly and then doesn't change much. Once per day or as otherwise specified by GBUdb condensation triggers the GBUdb data will be condensed. This means that all of the good and bad event counts are divided in half. This has the effect of retaining the ratios (probability of spam) while reducing the event counts (confidence figure). Any ugly entries that drop to zero are dropped from the system (forgotten). The remaining live entries are placed in a new GBUdb and the old one is thrown away. If the GBUdb size can shrink during a condensation run then it will. Will gbudb XCI commands (like bad) show up in a log file? No. These are administrative entries so they don't get reported in scan activity. We may change this later but at present there are no requests for it. Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: xci scanner command
A question about using the XCI bad command. Assume an email passes through sniffer and does not trigger any rules, I then run it through and determine it is in fact spam. I send a bad command to let sniffer know the IP address had a bad event. Won't the good event that would occur due the spam passing through nullify the bad event? Should I post 2 bad events for each mail that is caught after sniffer? On Tue, Feb 17, 2009 at 5:02 PM, Pete McNeil madscient...@armresearch.comwrote: Richard Stupek wrote: Thanks for the info. Is there any diagnostic information available when a gbudb sync occurs? You can always see the current status of GBUdb in your status.* files. If you append these logs you can follow the state of the system through time using pre-compiled statistics including the size of GBUdb. This can be done with one entry per minute (status.minute) or one entry per hour (status.hour) or both. In the same status log you can see when the last GBUdb condensation event occurred. No other useful data is produced by a condensation run so none is recorded. If there is something you would like to see then please let us know and we will consider adding features to support your request(s). Best, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] curl couldn't connect to host
I just started seeing this error for the getrulebase.cmd script. Is there an issue going on?
[sniffer] Re: 3 million rules and counting.
Congratulations. Keep up the good work!
[sniffer] Re: IP Change on rulebase delivery system
Not sure if its related but since yesterday SNFserver CPU utilization has been inordinately high (50%) for the middle of the day with not any additional volume in mail being received. On Mon, Mar 25, 2013 at 9:13 AM, Pete McNeil madscient...@armresearch.comwrote: Hi Sniffer Folks, We are about to change the IP of the rulebase delivery system. This change should be completely transparent and you should not need to take any action; however if you do notice anything unusual please let us know. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Its odd because the number of messags snf is processing isn't more than usual and the % of spam being detected through snf is actually lower than typical yet is is routinely maxing out 4 processors at 100%. On Wed, Mar 27, 2013 at 3:20 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-27 14:38, Darin Cox wrote: Probably unrelated... and due to a significant increase in spam over the past few days. I agree with that -- our inbound spamtrap pre-processor has seen 4x increase over the past few days so that's likely to be related. Also, Richard, I took a quick look at your telemetry and verified that your rulebase file(s) are up to date. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
It would be SNF routinely showing 80% utilization spikes for a 4 cpu system. I hadn't ever seen it do that before which was why I sent the message. Don't believe the load is any higher than normal. The spikes aren't as prolonged at the present. On Wed, Mar 27, 2013 at 4:08 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-27 16:49, Richard Stupek wrote: Its odd because the number of messags snf is processing isn't more than usual and the % of spam being detected through snf is actually lower than typical yet is is routinely maxing out 4 processors at 100%. You're saying that SNF is maxing out 4 processors? ... or is the combination of operations on your server maxing out 4 processors? We're using the same engine and ruelbase in our CGP server and humming along nicely at between 2000 - 8000 msg/minute with nominal CPU loads. I don't see anything unusual in your telemetry and I haven't heard any other complaints, so I can't explain why SNF would act differently on your system. I hate a mystery though -- so I would love to get to the bottom of it. Do you see anything else that might be causing the CPU load? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
well when all else fails restarting snf seems to have corrected the issue for now. On Thu, Mar 28, 2013 at 11:50 AM, Darin Cox dc...@4cweb.com wrote: Richard, Do you have any directories with a large number of files (4k)? We had a similar problem a few months back with sniffer scans taking much longer to complete and sniffer temporary files being left over. We finally traced the performance issues to a frequently accessed directory with thousands of files. We’ve also seen issues in the past with directories with a large number of files being very poor performing. Darin. *From:* Richard Stupek rstu...@gmail.com *Sent:* Thursday, March 28, 2013 12:10 PM *To:* Message Sniffer Community sniffer@sortmonster.com *Subject:* [sniffer] Re: IP Change on rulebase delivery system Ok looking at the log I see quite a few messages taking over a second to process (samples below): s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0' p s='1172' t='1109' l='72697' d='127'/ g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/ /s s u='20130328155506' m='\temp\1332407477336.msg' s='60' r='5113015' m s='60' r='5113015' i='235' e='280' f='m'/ m s='60' r='4346940' i='16722' e='16812' f='m'/ p s='1141' t='937' l='16658' d='129'/ g o='0' i='192.210.233.215' t='u' c='0.360316' p='0.575758' r='Normal'/ /s s u='20130328155513' m='\temp\1332407477360.msg' s='52' r='5470216' m s='52' r='5470216' i='235' e='295' f='m'/ m s='52' r='5471910' i='949' e='1009' f='m'/ m s='52' r='5431546' i='1074' e='1200' f='m'/ m s='52' r='5479780' i='1857' e='1933' f='m'/ m s='62' r='5303955' i='82' e='2688' f='m'/ m s='52' r='5400681' i='1818' e='9143' f='m'/ p s='1031' t='750' l='8538' d='130'/ g o='0' i='192.210.134.21' t='u' c='0.545993' p='0.82' r='Black'/ /s s u='20130328155622' m=\temp\1332407477655.msg' s='60' r='5538969' m s='60' r='5538969' i='221' e='236' f='m'/ m s='61' r='5448415' i='2283' e='2297' f='m'/ m s='61' r='5438936' i='2247' e='2337' f='m'/ m s='60' r='5404555' i='15832' e='15850' f='m'/ m s='60' r='5539002' i='16033' e='16074' f='m'/ m s='62' r='5437246' i='30967' e='30985' f='m'/ p s='1219' t='1312' l='17171' d='135'/ g o='0' i='205.234.138.240' t='u' c='0.634697' p='0.763214' r='Normal'/ /s On Wed, Mar 27, 2013 at 4:42 PM, Pete McNeil madscient...@armresearch.com wrote: On 2013-03-27 17:16, Richard Stupek wrote: The spikes aren't as prolonged at the present. Interesting. A short spike like that might be expected if the message was longer than usual, but on average SNF should be very light-weight. One thing you can check is the performance data in your logs. That will show how much time in cpu milleseconds it is taking for each scan and how long the scans are in bytes. This might shed some light. http://www.armresearch.com/**support/articles/software/** snfServer/logFiles/**activityLogs.jsphttp://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp Look for something like p s='10' t='8' l='3294' d='84'/ in each scan. From the documentation: sp//s - Scan Performance Monitoring (performance='yes') p:s = Setup time in milliseconds p:t = Scan time in milliseconds p:l = Scan length in bytes p:d = Scan depth (peak evaluator count) Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 866-770-1044%20x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? On Fri, Mar 29, 2013 at 12:27 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-03-29 12:59, Richard Stupek wrote: well when all else fails restarting snf seems to have corrected the issue for now. In that case, it is likely that RAM fragmentation was involved. Dropping the process allowed the fragmentation to be cleared. (theory). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Can you point me at the documentation for the truncate blacklist and its usage? On Thu, May 23, 2013 at 3:36 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-05-23 15:22, Richard Stupek wrote: Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? Check the message sizes. As part of the newest spam storms we've noticed that a lot of the messages are huge (65536++). I suspect this might impact throughput as large buffers are allocated and moved around to handle these messages. This kind of thing has also been known to cause NTFS to crawl. Please let us know what you find. If you are not already doing it -- you should consider blocking connections using the truncate blacklist. No sense taking on some of these messages if they can be eliminated up front. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Would this: http://armresearch.com/support/articles/software/snfServer/xci/gbudb.jsp yield the same results as using the ip4 blocklist? On Thu, May 23, 2013 at 4:11 PM, Pete McNeil madscient...@armresearch.comwrote: On 2013-05-23 16:41, Richard Stupek wrote: Can you point me at the documentation for the truncate blacklist and its usage? http://gbudb.com/truncate/**index.jsphttp://gbudb.com/truncate/index.jsp It's an ordinary ip4 dnsbl. Most email systems have some mechanism for blocking connections based on this kind of blacklist. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller ##**##**# This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-digest@sortmonster.**comsniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com** Send administrative queries to sniffer-request@sortmonster.**comsniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
Pete
I thought the local gbudb got updates from the service or was that a future
enhancement?
Original message
Subject: [sniffer] Re: IP Change on rulebase delivery system
From: Richard Stupek rstu...@gmail.com javascript:_e({}, 'cvml',
'rstu...@gmail.com');
To: Message Sniffer Community sniffer@sortmonster.com javascript:_e({},
'cvml', 'sniffer@sortmonster.com');
CC:
Can you point me at the documentation for the truncate blacklist and its
usage?
On Thu, May 23, 2013 at 3:36 PM, Pete McNeil
madscient...@armresearch.comjavascript:_e({}, 'cvml',
'madscient...@armresearch.com');
wrote:
On 2013-05-23 15:22, Richard Stupek wrote:
Looks like I have this issue again (pegging 4 core cpu) and resetting
the process doesn't make a difference. Not sure what is causing it but it
does slow down spam detection to 40-50 seconds for many emails. Any ideas
what I can look at or do to resolve this?
Check the message sizes. As part of the newest spam storms we've noticed
that a lot of the messages are huge (65536++). I suspect this might impact
throughput as large buffers are allocated and moved around to handle these
messages. This kind of thing has also been known to cause NTFS to crawl.
Please let us know what you find.
If you are not already doing it -- you should consider blocking
connections using the truncate blacklist. No sense taking on some of these
messages if they can be eliminated up front.
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller
##**##**#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com javascript:_e({}, 'cvml',
'sniffer@sortmonster.com');.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.comjavascript:_e({},
'cvml', 'sniffer-...@sortmonster.com');
To switch to the DIGEST mode, E-mail to
sniffer-digest@sortmonster.**comjavascript:_e({}, 'cvml',
'sniffer-dig...@sortmonster.com');
To switch to the INDEX mode, E-mail to
sniffer-in...@sortmonster.comjavascript:_e({}, 'cvml',
'sniffer-in...@sortmonster.com');
**
Send administrative queries to
sniffer-request@sortmonster.**comjavascript:_e({}, 'cvml',
'sniffer-requ...@sortmonster.com');
