A silly logging question
This is probably a dumb question, but my looking through the docs is just confusing me. Can I get SpamAssassin to fully log what it is doing? The best I can ever get is something like this; Mon Aug 3 06:27:57 2009 [4290] info: logger: removing stderr method Mon Aug 3 06:27:58 2009 [4292] info: spamd: server started on port 783/tcp (running version 3.2.5) Mon Aug 3 06:27:58 2009 [4292] info: spamd: server pid: 4292 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4293 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4294 Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: SI Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: II Mon Aug 3 06:34:31 2009 [4292] info: spamd: server killed by SIGTERM, shutting down But never any actual information on the mail scanning process. The problem, from my perspective, is I reject with a milter at the SMTP stage - so I never get to see any blocked messages. I have a few in my logs that Fetchmail has picked up from a pop account, before dumping them into Postfix - and I have no Spamassassin log I can view to see what they caught on and if I need to take any action. I had a look at: Mail::SpamAssassin::Logger - SpamAssassin logging module But I'm not entirely sure how to invoke that or if I can get it to give me what I need. It is very terse in information. Perhaps I can start SA differently to produce a log of the scanning it is doing? I'm sure this is a beginners question and I feel very stupid having to ask - but I cannot find the obvious answer. -- --- C Werclick
Re: how to speed up scans of really large text-only emails?
On 09/08/2009 01:50 PM, Jason Haar wrote: We're having problems with a particular class of email. 400K in size, text-only. spamd takes 40-80sec to process it, and spamc is set with a 30sec timeout. The long processing time isn't network-related: it's all those body searches that are causing the hang. On 08.09.09 16:27, Jason Haar wrote: Just to follow myself up, it looks like spamc -t 30 means if spamd doesn't return in 30sec, spamc will simply output the email to stdout and exit. spamd carries on processing the email and I can see the final spamd syslog report when it finishes in 30sec. Unfortunately we re-invoke spamc and it just goes through the same problem again :-( It's a pity spamd can't keep a small cache of checksum'ed previous messages and their scores, so that if it sees the same message again within (say) 10-30 min, it just throws up the cached score? That could cause other troubles. Note that at second scan of the same message the result can be much different mostlu because of network checks. It would also make spamd more complicated for no good reason. Simply use spamc -t 120 or 180, I think up to 240 is safe at SMTP level unless you are using other time-consuming test (data phase should end in 5 minutes otherwise client may assumt the connection hanged and close it). -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Silvester Stallone: Father of the RISC concept.
whitelist_from_dkim
SA list, I still don't seem to be getting more friendly with whitelist_from_dkim - could someone please try feeding this email through your SA setup: http://jessen.ch/files/community36.eml with this enabled: whitelist_from_dkim *...@community36.net The actual author is 'keine-antw...@community36.net'; I have run it through SA with debug on and I see it being added to whitelist entries. Still when it is checked by DIM, it reports author keine-antw...@community36.net, not in any dkim whitelist. /Per Jessen, Zürich
Re: how to speed up scans of really large text-only emails?
On 09/08/2009 07:54 PM, Matus UHLAR - fantomas wrote: It would also make spamd more complicated for no good reason. Simply use spamc -t 120 or 180, I think up to 240 is safe at SMTP level unless you are using other time-consuming test (data phase should end in 5 minutes otherwise client may assumt the connection hanged and close it). As spamc normally runs in 1-5 sec, 30sec was considered by us to be a sensible timeout. However, a longer timeout of 120s should totally get around this sort of problem without really altering anything else. The end of DATA phase was of concern to me - but I think the RFCs states the client has to wait around for either 10 or 20 minutes - either way it'd be fine. So yup - will do! :-) Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: A silly logging question
This is probably a dumb question, but my looking through the docs is just confusing me. Can I get SpamAssassin to fully log what it is doing? The best I can ever get is something like this; Mon Aug 3 06:27:57 2009 [4290] info: logger: removing stderr method Mon Aug 3 06:27:58 2009 [4292] info: spamd: server started on port 783/tcp (running version 3.2.5) Mon Aug 3 06:27:58 2009 [4292] info: spamd: server pid: 4292 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4293 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4294 Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: SI Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: II Mon Aug 3 06:34:31 2009 [4292] info: spamd: server killed by SIGTERM, shutting down But never any actual information on the mail scanning process. The problem, from my perspective, is I reject with a milter at the SMTP stage - so I never get to see any blocked messages. I have a few in my logs that Fetchmail has picked up from a pop account, before dumping them into Postfix - and I have no Spamassassin log I can view to see what they caught on and if I need to take any action. I had a look at: Mail::SpamAssassin::Logger - SpamAssassin logging module But I'm not entirely sure how to invoke that or if I can get it to give me what I need. It is very terse in information. Perhaps I can start SA differently to produce a log of the scanning it is doing? I'm sure this is a beginners question and I feel very stupid having to ask - but I cannot find the obvious answer. SA logs to local3.info, not mail.info, in my configuration. I do not recall changing that, it may be default. Does your logger catch local3.info?
Re: whitelist_from_dkim
On 08.09.09 10:04, Per Jessen wrote: I still don't seem to be getting more friendly with whitelist_from_dkim - could someone please try feeding this email through your SA setup: http://jessen.ch/files/community36.eml with this enabled: whitelist_from_dkim *...@community36.net The actual author is 'keine-antw...@community36.net'; I have run it through SA with debug on and I see it being added to whitelist entries. Still when it is checked by DIM, it reports author keine-antw...@community36.net, not in any dkim whitelist. I don't see DKIM record set for community36.net. If it's not set, it can't match. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. 2B|!2B, that's a question!
Filtering depending mail header
Hi,
I want to know if it would be possible a spamassassing configuration
that allows me execute spamassassing just in case a header mail exists
with a defined value.
System configuration is the following:
Spamassassing: /etc/spamassassin/
rewrite_header Subject *SPAM*
report_safe 0
required_score 3.5
use_bayes 1
bayes_auto_learn 1
bayes_learn_during_report 1
bayes_auto_expire 1
bayes_expiry_max_db_size 15
bayes_journal_max_size 102400
bayes_use_hapaxes 1
use_pyzor 0
use_razor2 1
allow_user_rules 1
Postfix: /etc/postfix/spamassassin
smtp inet n - n - 80 smtpd -o
content_filter=spamassassin
pickupfifo n - - 60 1 pickup -o
receive_override_options=no_header_body_checks
cleanup unix n - - - 0 cleanup
qmgr fifo n - n 300 1 qmgr
tlsmgrunix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounceunix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verifyunix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - - - - smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay unix - - - - - smtp
-o fallback_relay=
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scacheunix - - - - 1 scache
spamassassinunix- n n - - pipe
user=spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender}
-- ${recipient}
policyunix - n n - - spawn
user=nobody argv=/usr/bin/perl /usr/sbin/postfix-policyd-spf-perl
With this configuration, each mail that receives postfix is scanned
before sending to each user. My idea is that spamassassin scans only in
case mail received has a header added by another SMTP. My SMTP receives
mails that have been received before for the Enterprise SMTP. That
Enterprise SMTP add a header called X-imss-result:. Depending on the
value of this header, I would like to configure SpamAssassing for
execute scan just in case this header has a value equal to
Default_Triggered; in other cases, mail will be sent to user with no scan.
Any idea?
Re: whitelist_from_dkim
Matus UHLAR - fantomas wrote: On 08.09.09 10:04, Per Jessen wrote: I still don't seem to be getting more friendly with whitelist_from_dkim - could someone please try feeding this email through your SA setup: http://jessen.ch/files/community36.eml with this enabled: whitelist_from_dkim *...@community36.net The actual author is 'keine-antw...@community36.net'; I have run it through SA with debug on and I see it being added to whitelist entries. Still when it is checked by DIM, it reports author keine-antw...@community36.net, not in any dkim whitelist. I don't see DKIM record set for community36.net. If it's not set, it can't match. Okay. Well, SA still reports DKIM_VERIFIED, so the signature is fine, but maybe I'm whitelisting on the wrong thing? I also tried this: whitelist_from_dkim *=community36@mcsv129.net same result, i.e. not whitelisted. /Per Jessen, Zürich PS: the email is now available again, sorry about the HTTP 500 that some people got.
Re: whitelist_from_dkim
Per, http://jessen.ch/files/community36.eml whitelist_from_dkim *...@community36.net The actual author is 'keine-antw...@community36.net'; I have run it through SA with debug on and I see it being added to whitelist entries. Still when it is checked by DIM, it reports author keine-antw...@community36.net, not in any dkim whitelist. I don't see DKIM record set for community36.net. If it's not set, it can't match. Okay. Well, SA still reports DKIM_VERIFIED, so the signature is fine, but maybe I'm whitelisting on the wrong thing? I also tried this: Without the second argument to whitelist_from_dkim, it checks for author signatures, as documented. In your case the mail carries a signature by domain mcsv129.net, so you have a third-party signature there. If you want to whitelist an author by some third party signature, you need to tell in the second argument which signing domain is acceptable. whitelist_from_dkim *...@community36.net mcsv129.net Mark
Re: whitelist_from_dkim
Mark Martinec wrote: Per, Without the second argument to whitelist_from_dkim, it checks for author signatures, as documented. In your case the mail carries a signature by domain mcsv129.net, so you have a third-party signature there. If you want to whitelist an author by some third party signature, you need to tell in the second argument which signing domain is acceptable. whitelist_from_dkim *...@community36.net mcsv129.net Mark Aha. Thanks Marc! Much appreciated. /Per Jessen, Zürich
Re: whitelist_from_dkim [solved]
Mark Martinec wrote: Per, [snip] whitelist_from_dkim *...@community36.net mcsv129.net Just to confirm that it works: dkim: author keine-antw...@community36.net, WHITELISTED by whitelist_from_dkim /Per Jessen, Zürich
user prefs from sql problem
Hi, I am trying to load user preferences from SQL db (mysql). Setup was done according to doc/spamassassin/sql/README.gz, but user preferences are still loaded from files. No error message is raised into log file in debug mode. DB-based bayes and awl works fine. Debian GNU/Linux 5.0.3, spamassassin 3.2.5, mysql 5.0.51a. Spamassassin is invoked by spamc in /etc/procmailrc. spamd --max-children 2 --helper-home-dir --setuid-with-sql -d --pidfile=x What is going wrong? Thanks a lot, --kb -- Karel Beneš be...@is.svitavy.cz RádobyFotoBlog: http://kaaia.rohozna.cz/rfb/
Re: Filtering depending mail header
There's no way to do that with SpamAssassin itself. Once you send something to SA, it will do the whole process (there's short circuiting, but that's not really what you want here). It sounds like you're trying to not filter internal mail but filter external mail, so I would recommend two things: a) Ideally, have your MTA listen on two different IPs, one internal and one external. Apply different rules depending on which IP is being used. You mentioned using Postfix, and doing this is fairly trivial. b) Send the mail through something like procmail. It can make lightweight decisions about if this header exists do X. On Tue, Sep 8, 2009 at 5:17 AM, Daniel Ruiz Molinadaniel.r...@caos.uab.es wrote: I want to know if it would be possible a spamassassing configuration that allows me execute spamassassing just in case a header mail exists with a defined value. [...] With this configuration, each mail that receives postfix is scanned before sending to each user. My idea is that spamassassin scans only in case mail received has a header added by another SMTP. My SMTP receives mails that have been received before for the Enterprise SMTP. That Enterprise SMTP add a header called X-imss-result:. Depending on the value of this header, I would like to configure SpamAssassing for execute scan just in case this header has a value equal to Default_Triggered; in other cases, mail will be sent to user with no scan.
Re: how to speed up scans of really large text-only emails?
On Tue, 2009-09-08 at 13:50 +1200, Jason Haar wrote:
[...] Allowing spamd to only scan the first 50KB of text attachments
would do the trick. I can't think of a way that could be misused by
spammers? (ie they aren't going to send text-spam where the first 50KB
is bayes killer and the final bit is the spam - potential customers
won't scroll past the first couple of screens to find the spam).
I can.
It is indeed trivial to construct large messages, where the actual
payload is way down the end -- and the user will never see the bulk
above.
--
char *t=\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: whitelist_from_dkim
Benny, Still when it is checked by DIM, it reports author keine-antw...@community36.net, not in any dkim whitelist. correct it happends here aswell [22718] dbg: dkim: VALID third-party signature by id keine-antwort=3dcommunity36@mcsv129.net, author keine-antw...@community36.net, no valid matches [22718] dbg: dkim: author keine-antw...@community36.net, not in any dkim whitelist third party domain need to be whitelisted ? (Mail-DKIM 0.31, sa 3.2.5) Sure, if you want it to be be whitelisted. In absence of the second parameter, whitelist_from_dkim whitelists only on author signatures. $ man Mail::SpamAssassin::Plugin::DKIM whitelist_from_dkim aut...@example.com [signing-identity] [...] If no signing identity parameter is specified, the only acceptable signature will be a first-party signature, i.e. the so called author signature, which is a signature where the signing identity of a signature matches the author address (i.e. the address in a From header field). Mark
Re: A silly logging question
On Tue, 2009-09-08 at 09:08 -0700, John Hardin wrote: On Tue, 8 Sep 2009, Clunk Werclick wrote: Can I get SpamAssassin to fully log what it is doing? The best I can ever get is something like this; Mon Aug 3 06:27:57 2009 [4290] info: logger: removing stderr method Mon Aug 3 06:27:58 2009 [4292] info: spamd: server started on port 783/tcp (running version 3.2.5) Mon Aug 3 06:27:58 2009 [4292] info: spamd: server pid: 4292 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4293 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4294 Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: SI Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: II Mon Aug 3 06:34:31 2009 [4292] info: spamd: server killed by SIGTERM, shutting down That looks like the system log file (/var/log/messages). Have you looked in the mail log file (typically /var/log/maillog) ? I have it now - the only disappointment for me is it does not log the 'to' or 'from' or client ip. I understand this is not a function of Spamassassin, but it would have been so useful. It's an odd situation perhaps. Blocking at the SMTP stage using a milter in response to a Fetchmail hash feeding Postfix. The result is you see nothing very useful in the logs to identify the blocked message. I trust the set up entirely, but I love my logs too :-) It's not the fault of SpamAssassin - I'm clear on that, but as it is scanning the entire message retrieved by Fetchmail, it would be really useful to get it to log the client ip, to and from as well. There may even be a hack to do this someone is aware of? -- --- C Werclick .Lot Technical incompetent Loyal Order Of The Teapot. This e-mail and its attachments is intended only to be used as an e-mail and an attachment. Any use of it for other purposes other than as an e-mail and an attachment will not be covered by any warranty that may or may not form part of this e-mail and attachment.
Re: whitelist_from_dkim
On Tue 08 Sep 2009 06:25:49 PM CEST, Mark Martinec wrote Sure, if you want it to be be whitelisted. tidy give me 95 warns on the html part :) In absence of the second parameter, whitelist_from_dkim whitelists only on author signatures. this makes it simple to dump address books from horde into whitelist_from_dkim, telling users to add senders there if thay want to whitelist senders, the signing identity makes it work only for some senders :/ [snip] worse is when dkim and spf is not same domain, but this can be tested in sa with a meta so if whitelist_from_auth is used only one score will be added, whitelist_from_auth works both on spf and dkim, but problem comes when both is pass then sender would be giving ham scores from both looking forward to see sa 3.3.x on gentoo -- xpoint
Re: whitelist_from_dkim
On Tue 08 Sep 2009 10:04:21 AM CEST, Per Jessen wrote Still when it is checked by DIM, it reports author keine-antw...@community36.net, not in any dkim whitelist. correct it happends here aswell [22718] dbg: dkim: performing public key lookup and signature verification [22718] dbg: dkim: signing identity: keine-antwort=3dcommunity36@mcsv129.net, d=mcsv129.net, a=rsa-sha1, c=relaxed/relaxed [22718] dbg: dkim: signing identity: keine-antwort=community36@mcsv129.net, d=mcsv129.net, a=rsa-sha1, c=nofws [22718] dbg: dkim: signature verification result: PASS [22718] dbg: dkim: VALID third-party signature by id keine-antwort=3dcommunity36@mcsv129.net, author keine-antw...@community36.net, no valid matches [22718] dbg: dkim: VALID third-party signature by id keine-antwort=community36@mcsv129.net, author keine-antw...@community36.net, no valid matches [22718] dbg: dkim: author keine-antw...@community36.net, not in any dkim whitelist [22718] dbg: dkim: policy: performing lookup [22718] dbg: dkim: policy result neutral: o=~ [22718] info: async: ignoring response, mismatched id 62389/*.4e4546b0aaca3c9ec9cfbca060ff135c.al.dkim-reputation.org/TXT/IN, expected 12688/*.4e4546b0aaca3c9ec9cfbca060ff135c.al.dkim-reputation.org/TXT/IN [22718] info: dkimrep: id=community36.net$keine-antwort, mcsv129.net$community36.net, mcsv129.net rep=undef info=none third party domain need to be whitelisted ? Mail-DKIM 0.31 sa 3.2.5 -- xpoint
Re: A silly logging question
On Tue, 2009-09-08 at 09:34 -0700, John Hardin wrote: On Tue, 8 Sep 2009, Clunk Werclick wrote: I have it now - the only disappointment for me is it does not log the 'to' or 'from' or client ip. You may be able to determine that if you correlate more than one log. SA logs the message-ID, and the MTA log should give you enough information to determine the client IP address given the message-ID. Sadly, no. As Fetchmail is polling a remote POP3 server, the only part of the system to see *all* of the information, is Spamassassin. The MTA only sees 'localhost' from Fetchmail. Postfix parses out some information, but the client IP is missing. If I could change the way Spamassassin logs and what it logs, I would be - how do you put it - 'cooking on gas'. -- --- C Werclick .Lot Technical incompetent Loyal Order Of The Teapot. This e-mail and its attachments is intended only to be used as an e-mail and an attachment. Any use of it for other purposes other than as an e-mail and an attachment will not be covered by any warranty that may or may not form part of this e-mail and attachment.
Re: A silly logging question
On Tue, 8 Sep 2009, Clunk Werclick wrote: Can I get SpamAssassin to fully log what it is doing? The best I can ever get is something like this; Mon Aug 3 06:27:57 2009 [4290] info: logger: removing stderr method Mon Aug 3 06:27:58 2009 [4292] info: spamd: server started on port 783/tcp (running version 3.2.5) Mon Aug 3 06:27:58 2009 [4292] info: spamd: server pid: 4292 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4293 Mon Aug 3 06:27:58 2009 [4292] info: spamd: server successfully spawned child process, pid 4294 Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: SI Mon Aug 3 06:27:58 2009 [4292] info: prefork: child states: II Mon Aug 3 06:34:31 2009 [4292] info: spamd: server killed by SIGTERM, shutting down That looks like the system log file (/var/log/messages). Have you looked in the mail log file (typically /var/log/maillog) ? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Homeland Security: Specializing in Tactical Band-aids for Strategic Problems. -- Eric K. in Bruce Schneier's blog --- 9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: A silly logging question
On Tue, 8 Sep 2009, Clunk Werclick wrote: I have it now - the only disappointment for me is it does not log the 'to' or 'from' or client ip. You may be able to determine that if you correlate more than one log. SA logs the message-ID, and the MTA log should give you enough information to determine the client IP address given the message-ID. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Homeland Security: Specializing in Tactical Band-aids for Strategic Problems. -- Eric K. in Bruce Schneier's blog --- 9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: whitelist_from_dkim
On Tue, 2009-09-08 at 18:54 +0200, Benny Pedersen wrote: On Tue 08 Sep 2009 06:25:49 PM CEST, Mark Martinec wrote Sure, if you want it to be be whitelisted. tidy give me 95 warns on the html part :) That's normal. The HTML generated by word processors, etc is seldom clean but everything I've seen MS Office generate has been abominable: just a steaming heap of fetid dingo's kidneys. Martin
Re: how to speed up scans of really large text-only emails?
On Tue, 8 Sep 2009, Jason Haar wrote: We're having problems with a particular class of email. 400K in size, text-only. Do you have any stats on how spammy this class of mail is? Is it pure ham that you can detect using other methods, e.g. it's sent from a trusted source? If so, you may be able to develop some rules to detect these messages and use those rules at the MTA glue level to tell it to skip SA processing for those messages. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Homeland Security: Specializing in Tactical Band-aids for Strategic Problems. -- Eric K. in Bruce Schneier's blog --- 9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: whitelist_from_dkim
On Tue, 2009-09-08 at 18:24 +0100, Martin Gregorie wrote: On Tue, 2009-09-08 at 18:54 +0200, Benny Pedersen wrote: On Tue 08 Sep 2009 06:25:49 PM CEST, Mark Martinec wrote Sure, if you want it to be be whitelisted. tidy give me 95 warns on the html part :) That's normal. The HTML generated by word processors, etc is seldom clean but everything I've seen MS Office generate has been abominable: just a steaming heap of fetid dingo's kidneys. Now wait just a second. It would have to improve, greatly, to be a steaming heap of fetid dingo kidneys. Maybe rancid squirrel pancreases, but in no way would it reach the level of dingo kidneys -- Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX www.austinenergy.com signature.asc Description: This is a digitally signed message part
Re: A silly logging question
On Tue, 8 Sep 2009, Clunk Werclick wrote: On Tue, 2009-09-08 at 09:34 -0700, John Hardin wrote: On Tue, 8 Sep 2009, Clunk Werclick wrote: I have it now - the only disappointment for me is it does not log the 'to' or 'from' or client ip. You may be able to determine that if you correlate more than one log. SA logs the message-ID, and the MTA log should give you enough information to determine the client IP address given the message-ID. Sadly, no. As Fetchmail is polling a remote POP3 server, the only part of the system to see *all* of the information, is Spamassassin. The MTA only sees 'localhost' from Fetchmail. Postfix parses out some information, but the client IP is missing. If I could change the way Spamassassin logs and what it logs, I would be - how do you put it - 'cooking on gas'. And I don't suppose you can get the logs from the MTA fetchmail is retrieving from. Do you have your trust list set up to trust that MTA? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- It is criminal to teach a man not to defend himself when he is the constant victim of brutal attacks. -- Malcolm X (1964) --- 9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: how to speed up scans of really large text-only emails?
On 09/09/2009 04:07 AM, John Hardin wrote: Do you have any stats on how spammy this class of mail is? Is it pure ham that you can detect using other methods, e.g. it's sent from a trusted source? Actually, it's HAM - not spam. In the end it's really become clear it shows limitations in perl's parsing power - so either we get gruntier boxes - or increase the timeout. We've gone with the latter. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: A silly logging question
On Tue, 2009-09-08 at 12:08 -0700, John Hardin wrote: On Tue, 8 Sep 2009, Clunk Werclick wrote: Sadly, no. As Fetchmail is polling a remote POP3 server, the only part of the system to see *all* of the information, is Spamassassin. The MTA only sees 'localhost' from Fetchmail. Postfix parses out some information, but the client IP is missing. If I could change the way Spamassassin logs and what it logs, I would be - how do you put it - 'cooking on gas'. My system is fairly close to that: - fetchmail gets incoming mail from my ISP using pop3 - fetchmail passes it on by handing it to a script that is called by the 'mda' directive in .fetchmailrc - the script is a pipeline that is essentially spamc | spamkiller -c sendmail where spamkiller sends anything marked as spam to /dev/null and the rest are handed to postfix.sendmail to be passed to Postfix In my low throughput mailing system I can trace messages through this fairly easily: - fetchmail reports reading the message, giving recipient and size - spamd reports the msgid and, because fetchmail is single threaded, this MUST be the msgid of the message fetchmail last read because its reported as spamd receives it from spamc - spamkiller reports what it will do with the message, logging the envelope sender. Again, this MUST be the message that spamd last released because its output immediately after spamd released the message. This can be linked back to the msgid reported by spamd by the spamd thread's pid. - postfix/pickup reports the envelope sender, providing a link back to spamkiller. - fetchmail reports flushing the message or reading the next one. In summary, it is possible to trace the message's progress through the chain even though the same identifier isn't used by all the processes in the chain. This is helped by the fact that all the processes in the pipeline are single threaded and their processing is constrained to be a single synchronised execution path by the pipeline. The effect of this is that fetchmail never reads the next message before the current one has been binned by spamkiller or passed to postfix. In checking this I followed through two sessions in which fetchmail picked up 2 and 13 messages. Martin
Re: how to speed up scans of really large text-only emails?
On Wed, 9 Sep 2009, Jason Haar wrote: On 09/09/2009 04:07 AM, John Hardin wrote: Do you have any stats on how spammy this class of mail is? Is it pure ham that you can detect using other methods, e.g. it's sent from a trusted source? Actually, it's HAM - not spam. My point. If those messages are 100% ham, and they have indicators that are easy for the MTA to detect - e.g., from a known correspondent MTA IP address - then put some rules in at the MTA glue layer to say don't pass these messages to SA at all. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- It is criminal to teach a man not to defend himself when he is the constant victim of brutal attacks. -- Malcolm X (1964) --- 9 days until the 222nd anniversary of the signing of the U.S. Constitution
Re: A silly logging question
On Tue, 2009-09-08 at 11:50 +0300, Jari Fredriksson wrote:
This is probably a dumb question, but my looking through
the docs is just confusing me.
Can I get SpamAssassin to fully log what it is doing? The
best I can ever get is something like this;
Mon Aug 3 06:27:57 2009 [4290] info: logger: removing
stderr method Mon Aug 3 06:27:58 2009 [4292] info:
spamd: server started on port 783/tcp (running version
3.2.5)
Mon Aug 3 06:27:58 2009 [4292] info: spamd: server pid:
4292
Mon Aug 3 06:27:58 2009 [4292] info: spamd: server
successfully spawned child process, pid 4293
Mon Aug 3 06:27:58 2009 [4292] info: spamd: server
successfully spawned child process, pid 4294
Mon Aug 3 06:27:58 2009 [4292] info: prefork: child
states: SI
Mon Aug 3 06:27:58 2009 [4292] info: prefork: child
states: II
Mon Aug 3 06:34:31 2009 [4292] info: spamd: server
killed by SIGTERM, shutting down
But never any actual information on the mail scanning
process. The problem, from my perspective, is I reject
with a milter at the SMTP stage - so I never get to see
any blocked messages. I have a few in my logs that
Fetchmail has picked up from a pop account, before
dumping them into Postfix - and I have no Spamassassin
log I can view to see what they caught on and if I need
to take any action.
I had a look at:
Mail::SpamAssassin::Logger - SpamAssassin logging module
But I'm not entirely sure how to invoke that or if I can
get it to give me what I need. It is very terse in
information.
Perhaps I can start SA differently to produce a log of
the scanning it is doing? I'm sure this is a beginners
question and I feel very stupid having to ask - but I
cannot find the obvious answer.
SA logs to local3.info, not mail.info, in my configuration. I do not recall
changing that, it may be default. Does your logger catch local3.info?
I'm using syslog-ng, but despite listening to;
unix-stream(/dev/log);
It gets nothing - but I don't expect it to as the default spamassassin
conf has this line;
OPTIONS=--create-prefs --max-children 5 --username spamd
--helper-home-dir ${SAHOME} -s /var/log/spamassassin/spamd.log
Since August, it's produced empty logs here.
Re: A silly logging question
On Tue, 8 Sep 2009, Clunk Werclick wrote:
On Tue, 8 Sep 2009, Clunk Werclick wrote:
I have it now - the only disappointment for me is it does not log the
'to' or 'from' or client ip.
Blew away most of this thread already, before it started getting my
attention. Anyway, just checked archives, and there appears to be
something fishy that needs to be dealt with -- before we start barking
up the tree for custom logs.
The log snippet you did show us, and which you claimed is *all* you ever
got, is not what SA logs. Basically, it is starting, stopping and
the prefork log messages.
Granted, you killed spamd just a few minutes after starting. But since
you said you never saw anything else, I'll go by that. This is, what SA
actually logs per message -- both.
http://wiki.apache.org/spamassassin/SpamdSyslogFormat
Better? There you got all rules hit. You're missing out on that.
Sadly, no. As Fetchmail is polling a remote POP3 server, the only part
of the system to see *all* of the information, is Spamassassin. The MTA
only sees 'localhost' from Fetchmail. Postfix parses out some
information, but the client IP is missing. If I could change the way
Spamassassin logs and what it logs, I would be - how do you put it -
'cooking on gas'.
In some other post you said, you want to log the last external hop.
Well, frankly, since you are feeding spamd (at least partially) from
fetchmail, it is not exactly SA's fault that your MTA doesn't know about
the last-external, handing-over client.
Now, there would be quite a lot of possibilities. Cause SA does know.
Since the last external IP and rDNS are available as templates (see the
Conf docs), it is trivial to have SA add them as a custom Last-External
header. However, you also said you Reject spam [1], so you don't deliver
these anywhere. Yet, you want the info.
If there is (or would be) procmail somewhere in that chain, which is
easy to do for the fetchmail chunk only, logging that specific header is
trivial. Along with other information.
Also, there's the possibility to either patch spamd to log more (see
above), or preferably, to hack a custom logging plugin. This of course
should have access to the metadata, and thus the last external hop.
However, as a pre-requisite, you need to sort out your logging. Again,
see the link above and compare to your actual logs. As long as you don't
get those, any solution *within* SA is a lost battle.
guenther
[1] Something that's quite disturbing. You are not rejecting fetchmail
fetched spam, are you? These have been accepted by the MX SMTP
already. Hope you do not bounce that spam back to the *forged*
sender...
--
char *t=\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: A silly logging question
On Tuesday September 8 2009 12:10:41 Clunk Werclick wrote:
I'm using syslog-ng, but despite listening to;
unix-stream(/dev/log);
It gets nothing - but I don't expect it to as the default spamassassin
conf has this line;
OPTIONS=--create-prefs --max-children 5 --username spamd
--helper-home-dir ${SAHOME} -s /var/log/spamassassin/spamd.log
Since August, it's produced empty logs here.
See spamd man page. The -s option is supposed to take a
syslog facility name, not a filename.
Mark
Re: how to speed up scans of really large text-only emails?
On Tuesday September 8 2009 21:23:42 Jason Haar wrote: Actually, it's HAM - not spam. In the end it's really become clear it shows limitations in perl's parsing power - so either we get gruntier boxes - or increase the timeout. We've gone with the latter. Some regexps do perform terribly when given a large chunk of text with multiple matching opportunities. Some monolithic HTML with nested tables is one such example. If you still have the sample message, it would be interesting to try it on the current 3.3 code. The main difference could come from the fact that the current code splits mail text into smaller chunks and does not allow a rule regexp to work on an entire mail block. In some corner cases this brings a significant speedup, while on most of the rest it makes no difference. http://people.apache.org/~jm/mcsnapshot.tgz Mark
Re: how to speed up scans of really large text-only emails?
On 09/09/2009 12:53 PM, Karsten Bräckelmann wrote: Ah, good point, Mark -- that reminds me of the infamous issue of un-bound or nested quantifiers in RE rules. In some pathological cases, I've even debugged these to be the culprit of bringing SA down to its knees. Any custom rules? Do you still see the same timing when disabling them temporarily? Might be worth a shot. No really. Removing all .cf files and running it again knocked around 4 seconds off - so that isn't the cause by itself. I also got SA-3.3dev up, but it appears sa-update doesn't support it yet, and it complained about there being no 3.3 rules yet, and so I copied in the last 3.2.5 rules to give it something to chew on - and it complained about that, yadda, yadda. After all that it appeared to run ~10% faster - but given the sad state 3.3 said it was in, I don't trust the result. I'm afraid I'm not up to that level of testing yet - a bit early meethinks ;-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: user prefs from sql problem
Karel Beneš wrote: Hi, I am trying to load user preferences from SQL db (mysql). Setup was done according to doc/spamassassin/sql/README.gz, but user preferences are still loaded from files. No error message is raised into log file in debug mode. DB-based bayes and awl works fine. Debian GNU/Linux 5.0.3, spamassassin 3.2.5, mysql 5.0.51a. Spamassassin is invoked by spamc in /etc/procmailrc. spamd --max-children 2 --helper-home-dir --setuid-with-sql -d --pidfile=x What is going wrong? Did you set these options in your local.cf?: user_scores_dsn DBI:driver:connection user_scores_*sql*_username dbusername user_scores_*sql*_password dbpassword And what did you set user_scores_dsn to? See also: sql/README from the tarball (web copy for 3.2.x at: http://svn.apache.org/repos/asf/spamassassin/branches/3.2/sql/README) Thanks a lot, --kb
