[sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin
Sorry, forgot to CC all:
No the weight=1 issue is not yet resolved. In fact, I have been able to
determine that snf4sa is actually querying snfserver properly. I
removed the old plugin so only snf4sa is loaded by SA. I then tailed
the sniffer log and see items like this continuing to scroll by:
s u='20090515130114' m='/tmp/snf4sa/dL5Q6vQZ9G' s='52' r='2266218'
m s='52' r='2266218' i='906' e='949' f='m'/
p s='0' t='44' l='65536' d='56'/
g o='0' i='67.23.34.175' t='u' c='0.936317' p='-0.0474465'
r='Normal'/
/s
Note the path to the temp file /tmp/snf4sa/
That tells me that everything is working properly except the returning
of the score to SA.
I have tried running test messages through SA manually and the SNF4SA
headers get inserted properly, but I haven't yet run through a message
that sniffer identified as spam. I will attempt to get one of those and
run it through SA manually to see if SNF4SA returns the correct weight
when it identifies the spam.
I will also join the amavisd-new list and see if anyone there can shed
some light.
Dan Horne
TAIS
Director of Operations
www.taisweb.net
supp...@taisweb.net
828.252.TAIS (8247)
-Original Message-
From: Pete McNeil [mailto:madscient...@armresearch.com]
Sent: Thursday, May 14, 2009 6:27 PM
To: Alban Deniz
Cc: Dan Horne
Subject: Re: [sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin
for
SpamAssassin
Alban Deniz wrote:
snip/
1) I'll look at the SA3 and SNF4SA plugins to see if I can
determine the
reason for the timeout, and a solution. Pete mentioned that one
major
difference is that SNF4SA uses a TCP connection to communicate
with
SNFServer, while SA3 uses SNFClient.
The only possibility I can think of is that the snf4sa plugin
doesn't
wait long enough when running under amavisd-new. The timeout in
snf4sa
is set to 1 second, which is long enough when snf4sa is run by the
spamassassin command line. It might not be long enough when running
under amavisd-new. I don't think this is the problem. However, if
you
don't mind trying a longer timeout, here's how to change it: Edit
snf4sa.pm, changing line 72 from
$self-{SNF_Timeout} = 1;
to
$self-{SNF_Timeout} = 10;
Of course, a 10 second delay to process an email is unacceptable;
this
would simply point us in the right direction. Please let me know if
can try this.
Hey guys...
The timeout used in the SNFClient is on the order of 30 seconds--- 10
to
get a connection, 20 more to get an answer. When a system is busy it
can
take a few seconds for other requests that have already started to be
processed. The overall throughput is much higher than the individual
message timeout may suggest.
I recommend allowing at least 10 seconds -- though 30 might be more
appropriate.
Note also that I've seen SA itself take as long as 10-15 seconds to
process a message (depending on conditions) and it is roughly nominal
to
see it take 1 - 3 seconds per message in many configurations. SNF is
usually much quicker -- but we can't make assumptions about what else
may be happening on the system at any moment -- especially during
start-up conditions where incoming messages might be queued elsewhere
and ready to cause a rush.
Also -- isn't it reasonable that if SNF4SA does timeout it should
provide a 0 weight instead of 1 ??
Is that issues resolved?
Thanks for keeping me in the loop.
_M
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin
OK, I found a message that Sniffer identified as spam and ran it through SA manually and following are results: [mail:/home/vmail/taisweb.net/archive_received/Maildir] 9:22am# spamassassin --siteconfigpath=/usr/local/etc/mail/spamassassin -x -t .jlee/new/1237155804.M27154P10624V005CI0051B175_0.mail.taisweb.net,S =3981 Return-Path: sys...@blogsuccess.com X-Spam-Checker-Version: SpamAssassin 3.2.1 (2007-05-02) on mail.taisweb.net X-Spam-GBUdb-Analysis: 2, 67.131.25.27, Ugly c=0 p=0 Source New X-Spam-Status: No, score=-1.8 required=5.0 tests=HABEAS_ACCREDITED_COI,SNF4SA, URIBL_GREY autolearn=disabled version=3.2.1 X-Spam-SNF-Result: 62 (Obfuscation Techniques) X-Spam-DCC: CollegeOfNewCaledonia: mail.taisweb.net 1189; Body=1 Fuz1=1 Fuz2=1 X-Spam-Level: X-Spam-MessageSniffer-Rules: 62-469556-2307-2317-m 62-469556-4261-4271-m 62-469556-0-5994-f X-Spam-MessageSniffer-Scan-Result: X-Original-To: archive_received+j...@taisweb.net Delivered-To: archive_received+j...@taisweb.net Received: from localhost (localhost.taisweb.net [127.0.0.1]) by mail.taisweb.net (Postfix) with ESMTP id D7B292B2C87 for j...@taisweb.net; Sun, 15 Mar 2009 18:23:23 -0400 (EDT) X-Virus-Scanned: amavisd-new at taisweb.net Received: from mx1.rmslink.net (mx1.rmslink.net [68.118.154.10]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.taisweb.net (Postfix) with ESMTP id 65A522B2C92 for j...@taisweb.net; Sun, 15 Mar 2009 18:23:20 -0400 (EDT) Received: from platinum-smtp.infusionsoft.com (blogsuccess.platinum-smtp.infusionsoft.com [67.131.25.27]) by mx1.rmslink.net (Postfix) with ESMTP id 1EBDC39824 for j...@taisweb.net; Sun, 15 Mar 2009 18:23:19 -0400 (EDT) Received: from gil (unknown [10.3.0.124]) by smtp29.infusionsoft.com (Postfix) with ESMTP id 1B41B20841874 for j...@taisweb.net; Sun, 15 Mar 2009 18:23:19 -0400 (EDT) Date: Sun, 15 Mar 2009 18:23:19 -0400 (EDT) From: Jack Humphrey listrespo...@blogsuccess.com Sender: sys...@blogsuccess.com To: j...@taisweb.net Message-ID: 1429329783.1408551237155799111.javamail.tom...@gil Subject: J, this is BIG news! Errors-To: sys...@blogsuccess.com MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 7bit BatchId: 27269 X-BatchId: 27269 X-campaignid: infusion_blogsuccess27269 X-InfApp: blogsuccess X-BBounce: blogsuccess_3812781 X-InfContact: 235195 X-InfSent: 3812781 Package: platinum X-inf-package: platinum X-inf-source: MailBatchFulfillRequest X-MinStatusFlags: Double Opt-In X-MaxStatusFlags: Double Opt-In X-inf-uflags: Double Opt-In X-inf-iflags: Double Opt-In X-Virus-Scanned: ClamAV 0.94.2/9110/Sun Mar 15 01:06:44 2009 on mx1.rmslink.net X-Virus-Status: Clean [SNIP.../] Content preview: J, I have some news to share with you. Some BIG news Mike Filsaime has announced that he is GIVING AWAY 5000 Home Study courses of Butterfly Marketing. [...] Content analysis details: (-1.8 points, 5.0 required) pts rule name description -- -- -8.0 HABEAS_ACCREDITED_COI RBL: Habeas Accredited Confirmed Opt-In or Better [67.131.25.27 listed in sa-accredit.habeas.com] 6.0 SNF4SA Message Sniffer 0.2 URIBL_GREY Contains an URL listed in the URIBL greylist [URIs: infusionsoft.com] So the SNF4SA plugin is correctly returning the weight when run manually through SA. I will report this to the amavisd-new list to see if anyone has any ideas. Dan Horne TAIS Director of Operations www.taisweb.net supp...@taisweb.net 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Dan Horne Sent: Friday, May 15, 2009 9:23 AM To: Message Sniffer Community Subject: [sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin Sorry, forgot to CC all: No the weight=1 issue is not yet resolved. In fact, I have been able to determine that snf4sa is actually querying snfserver properly. I removed the old plugin so only snf4sa is loaded by SA. I then tailed the sniffer log and see items like this continuing to scroll by: s u='20090515130114' m='/tmp/snf4sa/dL5Q6vQZ9G' s='52' r='2266218' m s='52' r='2266218' i='906' e='949' f='m'/ p s='0' t='44' l='65536' d='56'/ g o='0' i='67.23.34.175' t='u' c='0.936317' p='-0.0474465' r='Normal'/ /s Note the path to the temp file /tmp/snf4sa/ That tells me that everything is working properly except the returning of the score to SA. I have tried running test messages through SA manually and the SNF4SA headers get inserted properly, but I haven't yet run through a message that sniffer identified as spam. I will attempt to get one of those and run it through SA manually
[sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin
I'm not getting any sniffer headers inserted, but then I'm running SA via amavisd-new, so I never get SA headers, only amavisd-new headers. Could this implementation have anything to do with the problem? Amavisd-new calls spamassassin directly via perl for each message and doesn't use spamd. Dan Horne From: Pete McNeil [mailto:madscient...@armresearch.com] Sent: Wednesday, May 13, 2009 5:37 PM To: Alban Deniz Cc: Dan Horne Subject: Re: [sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin Alban, I don't think this note made it to Dan or the list... I'm copying him on this. Alban Deniz wrote: Hi, When I ran the snf4sa plugin without SNFServer running, I got the following headers injected into the email: X-Spam-GBUdb-Analysis: _SNFGBUDBANALYSIS_ X-Spam-SNF-Result: _SNFRESULTTAG_ X-Spam-MessageSniffer-Rules: _SNFMESSAGESNIFFERRULES_ X-Spam-MessageSniffer-Scan-Result: _SNFMESSAGESNIFFERSCANRESULT_ If SNFServer were running, the _SNFXXX_ would have been replaced with the scan results. Also, the score was something like 8.3, which was from the other tests. the snf4sa plugin doesn't add to the score when SNFServer isn't running. I ran this using the spamassassin command, and got the following error (when SNFServer isn't running): [25199] warn: rules: failed to run SNF4SA test, skipping: [25199] warn: (Snf4sa: Error from SNFServer: cannot connect to socket (Connection refused) at /etc/spamassassin/snf4sa.pm line 466. [25199] warn: ) Received: from localhost by skidmark with SpamAssassin (version 3.2.5); Wed, 13 May 2009 17:13:41 -0400 Pete mentioned to me that your SNFServer is running. So, I think there might be some other problem (perhaps SNFServer isn't sending the messages the snf4sa plugin is expecting). What do you get when you pass the message through the spamassassin command? Thanks, Alban On Wednesday 13 May 2009 04:44:56 pm Pete McNeil wrote: Dan Horne wrote: Oh, yeah, I should also include this from the mail logs. It doesn't look like SNF4SA is being run successfully, but it still results in a SA weight of 1 rather than the sa_score configured: Interesting -- I wonder why snf4sa did not connect with SNFServer --- Am I correct that the later reference SNIFFER=6 indicates that a different SNF implementation was successful? When you run SNFClient -status.second do you get XML status data? In any case-- if the plugin was unable to connect it should return a zero score I'm passing this on to Alban. Thanks! _M
[sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin
Hi Pete. I've loaded SNF4SA.cf as instructed into /usr/local/etc/mail/spamassassin (the correct location on my FreeBSD server), but when I do spamassassin -D --lint I get the following lines: [1382] warn: config: failed to parse line, skipping, in /usr/local/etc/mail/spamassassin/snf4sa.cf: GBUdb_max_weight 3.0 [1382] warn: config: failed to parse line, skipping, in /usr/local/etc/mail/spamassassin/snf4sa.cf: snf_result 1 sa_score -5.0 short_circuit_no [1382] warn: config: failed to parse line, skipping, in /usr/local/etc/mail/spamassassin/snf4sa.cf: snf_result 20 sa_score 6.0 short_circuit_yes [1382] warn: config: failed to parse line, skipping, in /usr/local/etc/mail/spamassassin/snf4sa.cf: snf_result 40 sa_score 2.5 short_circuit_no [1382] warn: config: failed to parse line, skipping, in /usr/local/etc/mail/spamassassin/snf4sa.cf: snf_result 47-62 sa_score 4.0 short_circuit_no [1382] warn: config: failed to parse line, skipping, in /usr/local/etc/mail/spamassassin/snf4sa.cf: snf_result 63 sa_score 3.5 short_circuit_no [1382] dbg: config: fixed relative path: /usr/local/etc/mail/spamassassin/snf4sa.pm [1382] dbg: plugin: loading Snf4sa from /usr/local/etc/mail/spamassassin/snf4sa.pm ... and later in the output ... [1382] dbg: plugin: Snf4sa=HASH(0x986ba38) implements 'have_shortcircuited', priority 0 Please advise regarding the failed to parse line, skipping warnings. Does this mean this isn't working properly? Dan Horne TAIS Director of Operations www.taisweb.net supp...@taisweb.net 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Pete McNeil Sent: Tuesday, May 12, 2009 3:07 PM To: Message Sniffer Community Subject: [sniffer] SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin Hello Sniffer Folks, We have just released a MUCH improved plugin for SpamAssassin. Our new plugin makes full use of the SpamAssassin Plugin API to provide features like: * Add weights for specific scan result codes. * Add (or subtract) additional weight based on IP reputation statistics. * Optionally skip other tests. * Inject SNF headers. The SNF4SA plugin is included in the latest *nix distribution of SNF on our products page: http://www.armresearch.com/products/index.jsp Also we have packaged the SNF4SA plugin separately for those of you running SpamAssassin on Windows machines -- or if you already have SNF up and running and just want to switch to the latest SpamAssassin plugin. Here is a link for more information on SNF4SA: http://www.armresearch.com/products/SNF4SA.jsp We look forward to your feedback! Thanks, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin
Yes, SNIFFER is the old SA plugin, SNF4SA is the new one. I'm installing the new SNF4SA now and will report back with results. Dan Horne TAIS Director of Operations www.taisweb.net supp...@taisweb.net 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Pete McNeil Sent: Wednesday, May 13, 2009 4:45 PM To: Message Sniffer Community Subject: [sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin Dan Horne wrote: Oh, yeah, I should also include this from the mail logs. It doesn't look like SNF4SA is being run successfully, but it still results in a SA weight of 1 rather than the sa_score configured: Interesting -- I wonder why snf4sa did not connect with SNFServer --- Am I correct that the later reference SNIFFER=6 indicates that a different SNF implementation was successful? When you run SNFClient -status.second do you get XML status data? In any case-- if the plugin was unable to connect it should return a zero score I'm passing this on to Alban. Thanks! _M Remainder for reference... May 13 16:04:32 mail amavis[1051]: (01051-07) _WARN: rules: failed to run SNF4SA test, skipping:\n\t(Snf4sa: Timeout waiting for response from SNFServer at /usr/local/etc/mail/spamassassin/snf4sa.pm line 721.\n)\n May 13 16:04:33 mail amavis[1051]: (01051-07) spam_scan: score=25.451 autolearn=disabled tests=[DCC_CHECK=1.37,DIGEST_MULTIPLE=0.001,FB_INDEPEND_RWD=3.599,FH_FRO M_CASH=2.996,HTML_MESSAGE=0.001,HTML_MIME_NO_HTML_TAG=1.052,HTML_TAG_BAL ANCE_BODY=0.807,MIME_HTML_ONLY=1.672,RAZOR2_CF_RANGE_51_100=0.5,RAZOR2_C F_RANGE_E4_51_100=1.5,RAZOR2_CF_RANGE_E8_51_100=1.5,RAZOR2_CHECK=0.5,SNF 4SA=1,SNIFFER=6,SPF_FAIL=0.992,URIBL_BLACK=1.961] May 13 16:04:34 mail amavis[1051]: (01051-07) SPAM, bounce-muxikqvilwlv...@topspotbrands.com - dwil...@wilcoxtravel.com, Yes, score=25.451 tag=-999 tag2=6 kill=6 tests=[DCC_CHECK=1.37, DIGEST_MULTIPLE=0.001, FB_INDEPEND_RWD=3.599, FH_FROM_CASH=2.996, HTML_MESSAGE=0.001, HTML_MIME_NO_HTML_TAG=1.052, HTML_TAG_BALANCE_BODY=0.807, MIME_HTML_ONLY=1.672, RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E4_51_100=1.5, RAZOR2_CF_RANGE_E8_51_100=1.5, RAZOR2_CHECK=0.5, SNF4SA=1, SNIFFER=6, SPF_FAIL=0.992, URIBL_BLACK=1.961], autolearn=disabled # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin
OK, the new version fixes the spamassassin -D --lint warnings, but it still seems to be operating incorrectly. SNF4SA seems to be hitting on everything and I've still got the error in the log about timing out, but only once when spamd started. Otherwise, everything still seems to be getting a 1 weight for SNF4SA. About the command, the executables for SNFClient snd SNFServer compiled with .exe extensions by default, and their location (/var/spool/snfilter) isn't in the path, so I had to enter the following command to get any response: # /var/spool/snfilter/SNFClient.exe -status.second !-- Status Report -- stats nodeid='oknrwfg5' basetime='20090513210522' elapsed='1002' class='second' version engineSNFMulti Engine Version 3.0 Build: Aug 4 2008 11:29:01/engine platformSNF Server Version 3.0.1 Build: Aug 4 2008 11:29:21/platform /version timers run started='20090513205211' elapsed='792'/ sync latest='20090513210447' elapsed='36'/ save latest='20080805123428' elapsed='24309055'/ condense latest='1970010100' elapsed='1242248723'/ /timers gbudb size bytes='8388608'/ records count='114'/ utilization percent='0.57373'/ /gbudb counters /counters rates m s='0' m='12.7023' h='246.818' d='5923.64'/ s s='0' m='7.25843' h='143.717' d='3449.21'/ h s='0' m='5.44382' h='103.101' d='2474.43'/ w s='0' m='0' h='0' d='0'/ c s='0' m='0' h='0' d='0'/ b s='0' m='0' h='0' d='0'/ t s='0' m='0' h='0' d='0'/ a s='0' m='0' h='0' d='0'/ r s='0' m='0' h='0' d='0'/ /rates results /results rules rulebase utc='20090507193558'/ active utc='20090507193558'/ update ready='yes' utc='20090513194922'/ latest rule='2447774'/ /rules panics /panics /stats It also looks like the auto-rulebase updating isn't working either, but I'll worry about that separately. Dan Horne TAIS Director of Operations www.taisweb.net supp...@taisweb.net 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Pete McNeil Sent: Wednesday, May 13, 2009 4:45 PM To: Message Sniffer Community Subject: [sniffer] Re: SNF4SA - Message Sniffer Antispam Plugin for SpamAssassin Dan Horne wrote: Oh, yeah, I should also include this from the mail logs. It doesn't look like SNF4SA is being run successfully, but it still results in a SA weight of 1 rather than the sa_score configured: Interesting -- I wonder why snf4sa did not connect with SNFServer --- Am I correct that the later reference SNIFFER=6 indicates that a different SNF implementation was successful? When you run SNFClient -status.second do you get XML status data? In any case-- if the plugin was unable to connect it should return a zero score I'm passing this on to Alban. Thanks! _M Remainder for reference... May 13 16:04:32 mail amavis[1051]: (01051-07) _WARN: rules: failed to run SNF4SA test, skipping:\n\t(Snf4sa: Timeout waiting for response from SNFServer at /usr/local/etc/mail/spamassassin/snf4sa.pm line 721.\n)\n May 13 16:04:33 mail amavis[1051]: (01051-07) spam_scan: score=25.451 autolearn=disabled tests=[DCC_CHECK=1.37,DIGEST_MULTIPLE=0.001,FB_INDEPEND_RWD=3.599,FH_FRO M_CASH=2.996,HTML_MESSAGE=0.001,HTML_MIME_NO_HTML_TAG=1.052,HTML_TAG_BAL ANCE_BODY=0.807,MIME_HTML_ONLY=1.672,RAZOR2_CF_RANGE_51_100=0.5,RAZOR2_C F_RANGE_E4_51_100=1.5,RAZOR2_CF_RANGE_E8_51_100=1.5,RAZOR2_CHECK=0.5,SNF 4SA=1,SNIFFER=6,SPF_FAIL=0.992,URIBL_BLACK=1.961] May 13 16:04:34 mail amavis[1051]: (01051-07) SPAM, bounce-muxikqvilwlv...@topspotbrands.com - dwil...@wilcoxtravel.com, Yes, score=25.451 tag=-999 tag2=6 kill=6 tests=[DCC_CHECK=1.37, DIGEST_MULTIPLE=0.001, FB_INDEPEND_RWD=3.599, FH_FROM_CASH=2.996, HTML_MESSAGE=0.001, HTML_MIME_NO_HTML_TAG=1.052, HTML_TAG_BALANCE_BODY=0.807, MIME_HTML_ONLY=1.672, RAZOR2_CF_RANGE_51_100=0.5, RAZOR2_CF_RANGE_E4_51_100=1.5, RAZOR2_CF_RANGE_E8_51_100=1.5, RAZOR2_CHECK=0.5, SNF4SA=1, SNIFFER=6, SPF_FAIL=0.992, URIBL_BLACK=1.961], autolearn=disabled # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you
[sniffer] Re: favorite / best *nix distributions in the Sniffer community.
+1 for FreeBSD 6 and 7. Rock-solid stable for many years. We started with IMGate, then went further until it wasn't really recognizable as IMGate anymore, then we rebuilt from scratch with our own highly researched config. Dan Horne TAIS Director of Operations www.taisweb.net supp...@taisweb.net 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Harry Palmer Sent: Friday, December 12, 2008 1:36 PM To: Message Sniffer Community Subject: [sniffer] Re: favorite / best *nix distributions in the Sniffer community. We are also running Sniffer on FreeBSD with IMGate Advanced. It is probably a good idea for Sniffer to support FreeBSD rev 6.x and 7.x with the FreeBSD pkg and port systems. With pkg update, it is very easy to maintain applications. Thanks, Harry -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of David Fletcher Sent: Friday, December 12, 2008 12:25 PM To: Message Sniffer Community Subject: [sniffer] Re: favorite / best *nix distributions in the Sniffer community. We have our mail gateway running Sniffer on FreeBSD, but when we rebuild it we will either go with Ubuntu linux or a Windows based solution. This is not to bash FreeBSD. We just don't have the expertise in house to support it. David -Original Message- From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Pete McNeil Sent: Friday, December 12, 2008 12:10 PM To: Message Sniffer Community Subject: [sniffer] favorite / best *nix distributions in the Sniffer community. Hello Sniffer Folks, We are nearing completion of a significantly upgraded set of SNF distributions for *nix systems (BSD, OSX, Linux, etc.) We will soon be releasing Client/Server and Milter distributions built with autotools to simplify the installation process and make things more normal and less tricky for each platform. While RedHat has a high profile and market share in the server realm, it is often a challenge to cope with how far behind it is in software versions. I often wonder: Is RH perhaps it is too stable? I would really like to know your opinions on which distributions are most popular in our community and why. What about maintenance? Support from hosting providers? Other issues that matter more to you folks? This discussion will help us fine tune our next releases and might also help some of us get a handle on what really is best practice on these platforms these days. Thanks! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: What's in a name - or - objects in mirror.exe are bigger than they appear
Just want to chime in here. We use SNF on FreeBSD and more than once, when a newbie tech was troubleshooting the system, he'd remark that we seem to have the Windows version of Sniffer installed because of the extension. Files with a .exe extension just LOOK like Windows progs and can cause confusion when troubleshooting. Dan Horne TAIS Director of Operations www.taisweb.net [EMAIL PROTECTED] 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Saturday, September 06, 2008 4:38 PM To: Message Sniffer Community Subject: [sniffer] Re: What's in a name - or - objects in mirror.exe are bigger than they appear Hello Sanford, Friday, September 5, 2008, 2:21:38 PM, you wrote: I say, yes, remove the extensions. For when one deals with multi-boot or VM environments, seeing a Windows-style name can make you think it's just showing through from another filesystem or somethin' or was accidentally dumped there. Obvs. direct SNF admins wouldn't be likely to have this confusion, but other people on the box could. Feel you should leave the names in mixed case; that isn't non-*nix IMO. Forcing lowercase on case-preserving + case-sensitive systems is like pretending they're not cp/cs (easier to remember, sure, but not using the power, etc.). Thanks! This is what we've decided to do. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Stampede - amazing!
I've nothing of value to add, I just want to say thanks for posting things like this. It is very interesting to get these behind the scenes views of what the spammers are doing. It also gives me a valid explanation to give to my bosses when they complain that they're suddenly getting all kinds of spam. Dan Horne TAIS Director of Operations www.taisweb.net [EMAIL PROTECTED] 828.252.TAIS (8247) -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Thursday, August 28, 2008 5:13 AM To: Message Sniffer Community Subject: [sniffer] Stampede - amazing! Hello Sniffer Folks, I had been wondering why the blackhats had been pushing so hard for new bots these last few weeks. Then the other day I saw something very strange in the SNF telemetry. A storm came in that seemed to stop all other traffic. For more than an hour I really thought something was broken -- but I wasn't sure I'd really seen it. Just a short time ago our SortMonster on duty (Mitchell Skull) called all-hands for a new spam storm. This was another of the new penis spams. We coded the rules quickly and as they went out I saw it again: T rates fell to zero on many systems and close to that on all of the others. This means that virtually all of the IPs were brand-new. At the same time traffic spiked on all systems and capture rates went off-scale high as the new rules tagged virtually every message. This is not an entirely new tactic by the blackhats-- I've talked about it before. It is essentially a high-amplitude burst - where a new campaign is pre-tested against all known filters and then launched on a large number of new bots that are unknown to IP reputation systems. What is new is the purity of these recent events. When we've seen them before they were mixed in with a lot of other traffic from other bot nets and even other campaigns from the same bot net. While there was still a trickle of this activity, the purity of this burst was astounding. This was a stampede where essentially all visible bots started running in a single new direction. T rates have recovered now by and large -- so the new bots are already largely recognized by GBUdb, but the wild swing in telemetry across the network was amazing to watch -- as is the new telemetry showing dramatically increased traffic and capture rates indicating a nearly pure stream of spam from this new herd. Theories, comments, and observations welcome. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: NEW Version 2-9b1.1 Wide Beta
http://www.armresearch.com/message-sniffer/download/SNF2- 9b1.4.Source.zip [DH] Does this require a minimum version of gcc to compile? I am running gcc 3.4.4 on FreeBSD6 and I am getting errors when compiling: [mail:/root/SNF2-9b1.4.Source/ClientSource] 10:50am# ./compile In file included from main.cpp:26: networking.hpp:177: error: field `Address' has incomplete type In file included from networking.hpp:482, from main.cpp:26: networking.inline.hpp: In member function `void SocketAddress::clear()': networking.inline.hpp:128: error: `Address' undeclared (first use this function) networking.inline.hpp:128: error: (Each undeclared identifier is reported only once for each function it appears in.) networking.inline.hpp:130: error: `INADDR_ANY' undeclared (first use this function) networking.inline.hpp: In member function `sockaddr_in* SocketAddress::getPtr_sockaddr_in()': networking.inline.hpp:139: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `sockaddr* SocketAddress::getPtr_sockaddr()': networking.inline.hpp:143: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `socklen_t SocketAddress::getAddressSize()': networking.inline.hpp:148: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `void SocketAddress::setAddress(long unsigned int)': networking.inline.hpp:152: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `void SocketAddress::setAddress(char*)': networking.inline.hpp:156: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `long unsigned int SocketAddress::getAddress()': networking.inline.hpp:160: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `void SocketAddress::setPort(short unsigned int)': networking.inline.hpp:164: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `short unsigned int SocketAddress::getPort()': networking.inline.hpp:172: error: `Address' undeclared (first use this function) In file included from networking.cpp:25: networking.hpp:177: error: field `Address' has incomplete type In file included from networking.hpp:482, from networking.cpp:25: networking.inline.hpp: In member function `void SocketAddress::clear()': networking.inline.hpp:128: error: `Address' undeclared (first use this function) networking.inline.hpp:128: error: (Each undeclared identifier is reported only once for each function it appears in.) networking.inline.hpp:130: error: `INADDR_ANY' undeclared (first use this function) networking.inline.hpp: In member function `sockaddr_in* SocketAddress::getPtr_sockaddr_in()': networking.inline.hpp:139: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `sockaddr* SocketAddress::getPtr_sockaddr()': networking.inline.hpp:143: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `socklen_t SocketAddress::getAddressSize()': networking.inline.hpp:148: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `void SocketAddress::setAddress(long unsigned int)': networking.inline.hpp:152: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `void SocketAddress::setAddress(char*)': networking.inline.hpp:156: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `long unsigned int SocketAddress::getAddress()': networking.inline.hpp:160: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `void SocketAddress::setPort(short unsigned int)': networking.inline.hpp:164: error: `Address' undeclared (first use this function) networking.inline.hpp: In member function `short unsigned int SocketAddress::getPort()': networking.inline.hpp:172: error: `Address' undeclared (first use this function) networking.cpp: In member function `virtual void TCPListener::open()': networking.cpp:148: error: `IPPROTO_TCP' undeclared (first use this function) networking.cpp: In member function `virtual void TCPHost::open()': networking.cpp:267: error: `IPPROTO_TCP' undeclared (first use this function) # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: NEW Version 2-9b1.1 Wide Beta
When do you expect the source distribution to be available? I use sniffer as a spamassassin plugin on my freebsd mail server. -DH -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Thursday, October 04, 2007 8:51 PM To: Message Sniffer Community Subject: [sniffer] NEW Version 2-9b1.1 Wide Beta Hello Sniffer Folks, At your earliest convenience, please follow the following link to read about the newest version of Message Sniffer which has just been released for wide beta testing. http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarte d.Distributions#NEW_SNF_V2-9_Wide_Beta The command line client/server version is available now. It is a drop-in replacement for folks who have been running the current command line version (2-3.5) with a persistent instance on Winx platforms. The version in the posted distribution file requires a P3 or better. MDaemon and *nix (source) distributions will be coming shortly. This new engine has been in testing on a number of production systems from the very big to the very small for quite some time. There are no known bugs at this time. None the less, please be careful :-) and read carefully! A GREAT BIG THANK-YOU goes out to the folks who have helped us alpha test and refine this version over the previous months and weeks through scores of alpha iterations! We really appreciate the help. Over the next few days/weeks we will be adding documentation and answering questions to help folks explore and make the most use of the new features. We will also be looking for any last minute tweaks that might be needed; and we will be building a list of any additional features and/or refinements that come to light so we can get them into the production release, or at the very least the .1 that will follow. As always, your comments, questions, and feedback will help guide our efforts. The value of the discussions we share both privately and on this list cannot be overstated. Thanks for your patience, trust, and participation! Enjoy, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to sniffer- [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Version 2-3.5 Release -- Faster Engine
Thanks, Pete, I have it compiled and running on FreeBSD 6.0 as a spamassassin plugin. Logs show it is working as expected. Kudos. -Dan Horne -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, October 23, 2006 12:26 PM To: Message Sniffer Community Subject: [sniffer] Version 2-3.5 Release -- Faster Engine Hello SNF Folks, The plan was to hold off until the next major release, however in light of recent increases in spam traffic we are pushing out a new version with our faster engine included. All other upgrades are will wait for the major release ;-) The scanning engine upgrade results in a 2x speed increase that hopefully will help with the higher volumes we are seeing now. Version 2-3.5 also rolls up 2-3.2i1 which included the timing and file locking upgrades. You can find version 2-3.5 here: http://kb.armresearch.com/index.php?title=Message_Sniffer.Gett ingStarted.Distributions Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. SPAM-FREE 1.0(2476) # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Snf2check.exe on FreeBSD
Thanks, I will try the perl update script and see how it works. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, June 19, 2006 5:51 PM To: Message Sniffer Community Subject: [sniffer] Re: Snf2check.exe on FreeBSD Hello Dan, Monday, June 19, 2006, 5:30:15 PM, you wrote: I'm using sniffer on FreeBSD, plugging into Spamassassin. I am trying to write a good autoupdate cron script that works as well on my FreeBSD box as did the one I used to have on my Imail box. I can download the Sniffer DB, but I can't use snf2check.exe in my cron script. When I manually run the script logged in as root, and it gets to the line: /var/spool/snfilter/snf2check.exe /var/spool/snfilter/filename.snf authcodexxx The file checks out OK, however when it runs from cron (as root) it always gets ERROR RULE AUTH. Does anyone have an autoupdate script that is meant to run on a *nix-type system? Or does anyone know a solution to my problem? There is no reason I can think of for this not to work except perhaps for a permissions problem. Error rule auth would generally indicate that the file was corrupt, or that the authentication string is incorrect. All update scripts should use snf2check.exe before pressing the new rulebase file into production or else you may cripple your scanner with a bad file. (the SNF scanner does a less comprehensive check to maintain speed). All that said, on this page you can find PerlAutoUpdates and a few others which might help: http://kb.armresearch.com/index.php?title=Message_Sniffer.Tech nicalDetails.SubmittedScripts Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. SPAM-FREE 1.0(2476) # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
RE: [sniffer] Last chance to renew at the old price!
We've been using Sniffer for almost 5 years now and the price hasn't increased in that time. It's overdue, really. Fox, Thomas wrote on Tuesday, December 27, 2005 2:03 PM: I said the same thing, and the response was, basically, We haven't raised the price in a long time, we need the money, like it or lump it. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dave Koontz Sent: Tuesday, December 27, 2005 1:57 PM To: sniffer@SortMonster.com Subject: RE: [sniffer] Last chance to renew at the old price! Pete, why over a 50% increase? That seems rather drastic -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, December 27, 2005 12:42 PM To: sniffer@sortmonster.com Subject: [sniffer] Last chance to renew at the old price! Hello Sniffer folks, This is just a friendly reminder that prices will be going up January 1. You can add a year to your SNF subscription at the current price if you renew before January 1. Details are here: https://www.armresearch.com/message-sniffer/forms/form-renewal.asp Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- [This E-mail scanned for viruses by Declude Virus] --- [This E-mail scanned for viruses by Declude Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] auto update tmp files
Bonno Bloksma wrote on Friday, September 23, 2005 2:44 AM: C:\IMail\spool\tmp6C40.tmp As you can see the %1 is a complete path. So just Del %1 should do the trick. Wow, thanks. I never thought of actually checking to see what the value of %1 was. I just assumed (I know...) that it was just the file name. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] auto update tmp files
Pete McNeil wrote on Thursday, September 22, 2005 11:24 AM: On Thursday, September 22, 2005, 9:51:31 AM, John wrote: Sorry I'm late. I had trouble for a while with the del %1 functionality, but I had a problem with the script running in the wrong directory. I believe I added a cd \sniffer2 type line and it worked thereafter like a charm. This is a common problem with program aliases in IMail. It is always best to set the working directory at the top of any scripts that run as a program alias so that there is no question where they are running. I've learned that one the hard way a couple of times ;-) _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html Yeah, my script does explicitly enter the sniffer directory, and the line to delete the file is explicit as well: Del s:\imail\spool\%1 ...but that never worked. Maybe if I cd into the spool first it might work, but it is working with current directive, which is: Del s:\imail\spool\*.tmp ...so I really don't have a compelling reason to test it. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] auto update tmp files
I have tried to delete %1, but it never seemed to work. I ended up putting a "del *.tmp" at the end of my script and haven't had any problems. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bonno BloksmaSent: Monday, September 19, 2005 9:22 AMTo: sniffer@SortMonster.comSubject: [sniffer] auto update tmp files Hi, Ok, I had auto update pretty much in the air. Seems all I needed was a program alias that fired the script. ;-) There's just one thing, I end up with alot of "tmpID.tmp" files in my spool directory. Any way of deleting those automagically? I could simply delete all tmp.tmp files in my midnight run. Would that be a problem? The only program alias I have is the sniffer update. Met vriendelijke groet, Bonno Bloksma hoofd systeembeheer tio hogeschool toerisme en hospitality julianalaan 9 / 7553 ab hengelo t 074 255 06 10 / f 074 255 06 16 [EMAIL PROTECTED] / www.tio.nl
RE: Re[2]: [sniffer] Sniffer taking a long time?
Thanks, I will do that. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Wednesday, August 03, 2005 3:17 AM To: sniffer@SortMonster.com Subject: RE: Re[2]: [sniffer] Sniffer taking a long time? So basically, what you are saying is that my volume is really too low to take advantage of the persistent sniffer (and such may actually decrease my performance), and I should stick with the non-service version. Is that right? That is about what I thought (without the details of how sniffer works, I just wanted to be sure). Well, Dan, for the inevitable rush of traffic, I'd stick with the persistent sniffer implementation now that you have it working. If the 2 second wait time galls you, then use your **.cfg file and specify the MaxPollTime: 500 value at 500 ms or whatever you'd like your maximum wait time to be instead of 2 seconds (2000 ms). Andrew 8) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Sniffer taking a long time?
OK, I have managed to get SOMETHING working, but it still seems too slow and something is still not right. I originally set up the persistent sniffer using the instructions from this post: http://www.mail-archive.com/sniffer@sortmonster.com/msg00169.html This uses SRVANY.exe. I conjectured that possibly the service needed a home directory, so I added an AppDirectory value to the sniffer service's Parameters key in the registry. This value is set to the directory sniffer resides in. I also (based on my reading of the srvany.exe documentation) added another value to the same key called AppParameters. This is set to my auth code followed by a space, followed by the word persistent. Now when I start the service, the time spent processing a single message goes down to something around 2 seconds, but is still far longer than the non-service version. I also still had no .stat file in my sniffer directory. I did get a *.SVR file, which I never got before. So then I'm thinking, let's just make sure that I have the latest version of sniffer. I downloaded that, did the necessary renaming of the files and then started the service. NOW there is a *.persistent.stat file. However, the scan time is still at around 2 seconds. Average Scan times (based on average scan times of 5 emails each): Without sniffer service running: .033 seconds With sniffer service running: 2.244 seconds The *.persistent.stat file has the following contents: TicToc: 1122990610 Loop: 512 Poll: 445 Jobs: 34 Secs: 303 Msg/Min: 6.73267 Current-Load: 8.69565 Average-Load: 10.6371 Any suggestions? Thanks, Dan Horne This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Sniffer taking a long time?
I removed the AppParameters value and put the authcode and persistent back in the Application value where it was before. It didn't make any difference at all in the processing time, still right around 2 seconds. I don't know how your setup is working without at least the AppDirectory value, because mine didn't start working until I put that in, but if it is, I can't argue. My server load isn't anywhere near yours, so I don't see what the problem could be with mine. Oh well, unless Pete responds with a suggestion, I guess I'll just keep using the non-service version. Thanks anyway. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Tuesday, August 02, 2005 2:37 PMTo: sniffer@SortMonster.comSubject: Re: [sniffer] Sniffer taking a long time? Dan,I seem to recall trying to use the AppParameters key and having difficulty with it. I think that you might want to try removing that key and putting everything in the Parameters key, or at least that works for me. If you change HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Sniffer\Parameters in RegEdit to the following it might fix the issue that you are having: C:\IMail\Declude\Sniffer\***RULEBASE-NAME***.exe ***AUTH-CODE*** persistentYou should of course adjust the path and service name as well.The directions that I provided are working perfectly on my server so far as I can tell. I'm running dual 3.2 Ghz 1 MB cache Xeons with 5 x 15,000 RPM drives in RAID 5. The following three debug log entries shows between 300 ms and 550 ms per message: 08/02/2005 14:19:47.113 QB93D976201222A43 [2616] SNIFFER-IP: External program started: C:\IMail\Declude\Sniffer\executable.exe auth-code F:\\DB93D976201222A43.SMD08/02/2005 14:19:47.676 QB93D976201222A43 [2616] SNIFFER-IP: External program reports exit code of 61-08/02/2005 14:19:47.488 QB9418A4800EC2A49 [6196] SNIFFER-IP: External program started: C:\IMail\Declude\Sniffer\executable.exe auth-code F:\\DB9418A4800EC2A49.SMD08/02/2005 14:19:47.770 QB9418A4800EC2A49 [6196] SNIFFER-IP: External program reports exit code of 51-08/02/2005 14:19:49.879 QB943711501382A4D [6388] SNIFFER-IP: External program started: C:\IMail\Declude\Sniffer\executable.exe auth-code F:\\DB943711501382A4D.SMD08/02/2005 14:19:50.176 QB943711501382A4D [6388] SNIFFER-IP: External program reports exit code of 59My stat file shows the following: TicToc: 1122992104Loop: 154Poll: 0Jobs: 118392Secs: 155137Msg/Min: 45.7887Current-Load: 24.4275 Average-Load: 23.8719 I'm not sure why people use FireDaemon for this. My experience with SRVANY.exe has been absolutely flawless since I integrated this, and it has worked on both Win2k and Windows 2003.MattDan Horne wrote: OK, I have managed to get SOMETHING working, but it still seems too slow and something is still not right. I originally set up the persistent sniffer using the instructions from this post: http://www.mail-archive.com/sniffer@sortmonster.com/msg00169.html This uses SRVANY.exe. I conjectured that possibly the service needed a home directory, so I added an AppDirectory value to the sniffer service's "Parameters" key in the registry. This value is set to the directory sniffer resides in. I also (based on my reading of the srvany.exe documentation) added another value to the same key called AppParameters. This is set to my auth code followed by a space, followed by the word persistent. Now when I start the service, the time spent processing a single message goes down to something around 2 seconds, but is still far longer than the non-service version. I also still had no .stat file in my sniffer directory. I did get a *.SVR file, which I never got before. So then I'm thinking, let's just make sure that I have the latest version of sniffer. I downloaded that, did the necessary renaming of the files and then started the service. NOW there is a *.persistent.stat file. However, the scan time is still at around 2 seconds. Average Scan times (based on average scan times of 5 emails each): Without sniffer service running: .033 seconds With sniffer service running: 2.244 seconds The *.persistent.stat file has the following contents: TicToc: 1122990610 Loop: 512 Poll: 445 Jobs: 34 Secs: 303 Msg/Min: 6.73267 Current-Load: 8.69565 Average-Load: 10.6371 Any suggestions? Thanks, Dan Horne This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
RE: [sniffer] Sniffer taking a long time?
et up the persistent sniffer using the instructions from this post: http://www.mail-archive.com/sniffer@sortmonster.com/msg00169.html This uses SRVANY.exe. I conjectured that possibly the service needed a home directory, so I added an AppDirectory value to the sniffer service's "Parameters" key in the registry. This value is set to the directory sniffer resides in. I also (based on my reading of the srvany.exe documentation) added another value to the same key called AppParameters. This is set to my auth code followed by a space, followed by the word persistent. Now when I start the service, the time spent processing a single message goes down to something around 2 seconds, but is still far longer than the non-service version. I also still had no .stat file in my sniffer directory. I did get a *.SVR file, which I never got before. So then I'm thinking, let's just make sure that I have the latest version of sniffer. I downloaded that, did the necessary renaming of the files and then started the service. NOW there is a *.persistent.stat file. However, the scan time is still at around 2 seconds. Average Scan times (based on average scan times of 5 emails each): Without sniffer service running: .033 seconds With sniffer service running: 2.244 seconds The *.persistent.stat file has the following contents: TicToc: 1122990610 Loop: 512 Poll: 445 Jobs: 34 Secs: 303 Msg/Min: 6.73267 Current-Load: 8.69565 Average-Load: 10.6371 Any suggestions? Thanks, Dan Horne This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =-- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
RE: Re[2]: [sniffer] Sniffer taking a long time?
So basically, what you are saying is that my volume is really too low to take advantage of the persistent sniffer (and such may actually decrease my performance), and I should stick with the non-service version. Is that right? That is about what I thought (without the details of how sniffer works, I just wanted to be sure). Thanks, Pete. Dan Horne -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, August 02, 2005 4:09 PM To: Dan Horne Subject: Re[2]: [sniffer] Sniffer taking a long time? After following through all of this and looking at the .stat file, I think I see what's going on. Now that it is running and producing a .stat file, the flow rate is very low. According to the stat data, about 6 msgs / minute. Note the poll and loop times are in the 450 - 550 ms range. SNF with the persistent engine is built for high throughput, but it's also built to play nice. The maximum poll time gets up to 2 seconds or so (sound familiar?) If there are no messages for a while, then everything slows down until the first message goes through. For that first message, the SNF client will probably wait about 2 seconds before looking for it's result because that's what the stat file will tell it to do. Since the next message probably won't come around for a few seconds, that next message will probably wait about 2 seconds also. If you were doing 6 messages a second then all of the times would be much lower and so would the individual delays. When you turn off the persistent instance, each new message causes a client to look and see if there are any other peers acting a servers... Since the messages are far and few between, the client will elect to be a server (momentarily), will find no work but it's own, will process it's own message and leave. -- This is the automatic peer-server mode. It will always work like this unless more than one message is being processed at the same moment. In peer-server mode, since there is nothing else going on and no persistent instance to coordinate the operations, each message will get processed as fast as the rulebase can be loaded and then the program will drop. When the persistent instance is introduced, it sets the pace - and sicne there are no other messages, each client will wait about 2 seconds (or half a second or so with the .stat file contents you show) before it begins looking for it's results. The server instance will also wait a bit before looking for new jobs so that the file system isn't constantly being scanned. Of course, if a burst of messages come through then the pacing will speed up as much as necessary to keep up with the volume. Hope this helps, _M On Tuesday, August 2, 2005, 3:38:52 PM, Dan wrote: DH No, I followed your instructions exactly (and not for the first DH time). I didn't add those extra values until today. Prior to DH adding the AppDirectory value, the service was taking a minute to DH scan emails; after adding it the scan time went to around 2 DH seconds. I can't get it any lower than that. Initially mine was DH set up exactly as you said, with only Application containing the DH path, authcode and persistent. Today after hearing no suggestions DH from the list, and based on recent list messages mentioning the home DH directory for the service, I looked at the srvany.exe doco to find DH out how to give it a home directory. DH That's when I added AppDirectory. I also saw and added DH AppParameters at the same time and added those as well, though they DH seem not to be needed. DH DH Prior to adding the AppDirectory value, I never got any .stat file DH or any .SVR file in my sniffer dir. After adding that value and DH starting the service those files appeared. DH DH DH From: [EMAIL PROTECTED] DH [mailto:[EMAIL PROTECTED] On Behalf Of Matt DH Sent: Tuesday, August 02, 2005 3:24 PM DH To: sniffer@SortMonster.com DH Subject: Re: [sniffer] Sniffer taking a long time? DH Dan, DH There is no AppDirectory value on my servereither. The DH Parameters key has only one value under it besides Default DH which is Application, and it contains exactly what I provided DH below. Could it be that you tried to hard to get everything DH right by tweaking theseadditional keys? DH Something else. Did you make sure that theSniffer DH service that you created was started? No doubt it will work if DH you follow those directions to a T, and there aren't any issues DH with yourserver apart from this. DH Matt DH Dan Horne wrote: DH I removed the AppParameters value and put the authcode DH and persistent back in the Application value where it was before. DH It didn't make any difference at all in the processing time, DH still right around 2
[sniffer] Sniffer taking a long time?
OK, based on another thread on the Declude Junkmail list, I've taken a look at the Declude debug logs for a couple of messages. I am seeing this: 08/01/2005 11:32:51.747 Q40a201cc1a59 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D40a201cc1a59.SMD 08/01/2005 11:33:46.751 Q40a201cc1a59 SNIFFER: External program reports exit code of 61 Am I reading this right (I must be) that this log snip shows sniffer taking almost a full minute to scan this message? Here are more: 08/01/2005 11:30:53.757 Q402b01b61a28 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D402b01b61a28.SMD 08/01/2005 11:31:48.210 Q402b01b61a28 SNIFFER: External program reports exit code of 52 08/01/2005 11:30:56.561 Q402a01cc1a27 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D402a01cc1a27.SMD 08/01/2005 11:31:51.074 Q402a01cc1a27 SNIFFER: External program reports exit code of 0 If so, I think I've found my bottleneck, and I guess I need help figuring out why it is taking so long to scan. These messages are always in this order (meaning the program started line is always right before the program reports line). This means (if I understand Declude's logging correctly) that Declude started sniffer, then sat back and waited almost a full minute for each email. I am running persistent sniffer (assuming I set it up correctly). Where can I look to find out why it is taking so long to scan? This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Sniffer taking a long time?
Here are the sniffer log entries for each of the messages, if that helps any: 08/01/2005 11:32:51.747 Q40a201cc1a59 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D40a201cc1a59.SMD 08/01/2005 11:33:46.751 Q40a201cc1a59 SNIFFER: External program reports exit code of 61 20050801153252 D40a201cc1a59.SMD 70 20 Match 266707 61 343 358 50 20050801153252 D40a201cc1a59.SMD 70 20 Match 426427 61 1915192950 20050801153252 D40a201cc1a59.SMD 70 20 Final 266707 61 0 502050 08/01/2005 11:30:53.757 Q402b01b61a28 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D402b01b61a28.SMD 08/01/2005 11:31:48.210 Q402b01b61a28 SNIFFER: External program reports exit code of 52 20050801153054 D402b01b61a28.SMD 80 10 Match 372669 52 2745286060 20050801153054 D402b01b61a28.SMD 80 10 Match 423177 61 2695303660 20050801153054 D402b01b61a28.SMD 80 10 Match 372652 61 2695313860 20050801153054 D402b01b61a28.SMD 80 10 Final 372669 52 0 495260 08/01/2005 11:30:56.561 Q402a01cc1a27 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D402a01cc1a27.SMD 08/01/2005 11:31:51.074 Q402a01cc1a27 SNIFFER: External program reports exit code of 0 20050801153056 D402a01cc1a27.SMD 190 40 White 137999 0 2256228544 20050801153056 D402a01cc1a27.SMD 190 40 Final 137999 0 0 24419 44 This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Sniffer taking a long time?
More info: When I stop the Sniffer service, processing time goes to milliseconds. Start the service back and it is back up to a minute. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dan Horne Sent: Monday, August 01, 2005 11:58 AM To: sniffer@SortMonster.com Subject: RE: [sniffer] Sniffer taking a long time? Here are the sniffer log entries for each of the messages, if that helps any: 08/01/2005 11:32:51.747 Q40a201cc1a59 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D40a201cc1a59.SMD 08/01/2005 11:33:46.751 Q40a201cc1a59 SNIFFER: External program reports exit code of 61 20050801153252D40a201cc1a59.SMD 70 20 Match 266707 61343 358 50 20050801153252D40a201cc1a59.SMD 70 20 Match 426427 611915192950 20050801153252D40a201cc1a59.SMD 70 20 Final 266707 610 502050 08/01/2005 11:30:53.757 Q402b01b61a28 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D402b01b61a28.SMD 08/01/2005 11:31:48.210 Q402b01b61a28 SNIFFER: External program reports exit code of 52 20050801153054D402b01b61a28.SMD 80 10 Match 372669 522745286060 20050801153054D402b01b61a28.SMD 80 10 Match 423177 612695303660 20050801153054D402b01b61a28.SMD 80 10 Match 372652 612695313860 20050801153054D402b01b61a28.SMD 80 10 Final 372669 520 495260 08/01/2005 11:30:56.561 Q402a01cc1a27 SNIFFER: External program started: M:\IMail\Sniffer2\Distribution\Winx\mysniffer.exe mysnifferauthcode S:\imail\spool\D402a01cc1a27.SMD 08/01/2005 11:31:51.074 Q402a01cc1a27 SNIFFER: External program reports exit code of 0 20050801153056D402a01cc1a27.SMD 190 40 White 137999 0 2256228544 20050801153056D402a01cc1a27.SMD 190 40 Final 137999 0 0 24419 44 This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Sniffer taking a long time?
I replied to an off-list message from Pete, but for completeness, I will repost it to the list. We can keep it on the list, Pete, if that does ya'. It looks like Pete is probably right in that the service is probably not loading correctly for some reason. There is no .stat file in my sniffer directory. Here are my responses to Pete's questions: Can you please tell me the content of your .stat file. There is no .stat file in my sniffer directory. No file ending with .stat, either. Can you estimate the number of messages per minute that you are processing? Fairly low volume, I guess, around 10 messages per minute. Do you have a lot of extra files in your sniffer directory? Yes, there are tons of old *.FIN files, *.WRK files, *.XXX files, *.ERR files, and a few *.ABT files. However they are mostly old files. Sorting by date, I can see several *.FIN files, but they don't hang around long. There are several still there from each day though (I assume due to daily scheduled reboots according to the timestamp). The last occurrences of the other files by extension are: *.XXX - 7/24/2005 *.ERR - 4/27/2005 *.ABT - 2/4/2005 *.WRK - 12/14/2004 I assume it is ok to delete all these? Does you have a lot of fragmentation in your file system? How do you mitigate the fragmentation you do have? No, we defrag daily after hours using Diskeeper's smart scheduling. This information will help. Thanks, _M NP. I'm sure you saw my other posts to the list, but I'll recap. When I stop the service, processing time goes down to milliseconds. Reenabling the sniffer service (installed per the archived instructions using srvany.exe) causes the processing time to go back up into the minute per message range. I have the service disabled for now. We moved our Imail/Declude install off to a weaker machine a couple weeks ago in prep for replacing it with Suse Linux ES running postfix (and sniffer, of course) on the more powerful hardware. Because the current computer is not as powerful and has become backed up a few times, I was looking at ways to lower the CPU cost per message when I found this. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Declude and Sniffer
I weight sniffer high enough to hold the message on its own. We use it as our blacklist and it works great. We get some false positives, but we whitelist those and move on. Our users forward all spam received in their inbox to an email address that the sniffer system checks automatically. It adds rules to our sniffer rulebase for all the messages it downloads. So next time it comes in it gets held automatically. It has been working wonderfully. Dan Horne -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Carter Sent: Wednesday, July 20, 2005 9:16 AM To: sniffer@sortmonster.com Subject: [sniffer] Declude and Sniffer To other Declude users with Sniffer: I currently tag subject lines at 10 and delete at 20. Sniffer results are scored at 9. No two tests currently result in more than 18 and therefore it takes three failed tests to delete. I am considering moving Sniffer to 10. This would tag the subjects based on Sniffer alone, but still required three failed tests to delete. Question: Do any of you tag subject lines based on Sniffer alone? My main problem is that some of my users delete based on the tagged subject line. Thanks, John This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
