Re: Live upgrade safe?
Am 11.09.2015 um 17:54 schrieb RW: On Fri, 11 Sep 2015 08:21:15 -0700 Ian Zimmerman wrote: On 2015-08-14 17:45 +0200, Reindl Harald wrote: Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local configuration files, and without regenerating the Bayes database? (I use the default bdb Bayes store.) yes, but you need to run "sa-update" before restart to fetch the latest rules and hopefully have a distribution which restarts automatically after update the package Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? You need a restart to run the new software or pickup new rules. You don't need to avoid a restart between a package update and a rule update. i saw spamassassin just crash after upgrade on Fedora before /var/lib/spamassassin/3.004001/ was filled by sa-update and i remember at least one post on this list observing the same problem on a different environment If you are running sa-update from cron you don't really need to run it manually unless you are updating from a very old version that didn't support sa-update or no longer receives updates you are aware that the previous version don't matter 3.4.1 uses /var/lib/spamassassin/3.004001/ 3.4.0 uses /var/lib/spamassassin/3.004000/ so it is *completly* irrelevant from which version you upgrade signature.asc Description: OpenPGP digital signature
Re: Live upgrade safe?
Am 11.09.2015 um 18:05 schrieb Ian Zimmerman: On 2015-09-11 17:35 +0200, Reindl Harald wrote: Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local configuration files, and without regenerating the Bayes database? (I use the default bdb Bayes store.) yes, but you need to run "sa-update" before restart to fetch the latest rules and hopefully have a distribution which restarts automatically after update the package Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? i hope you have a testing environment for production and so just make the "sa-update" there and rsync the rule-updates to the liveserver I appreciate you trying to help, but you don't really answer my question. Even if I could do what you suggest, the rsync would still take finite time - longer than the interval between the upgrade and the restart on the production system. no, you don't need to change anything else in most setups regenerate bayes would even be impossible because you don't have the autolearned messages to do so if you have your whole corpus (like we do) you should rebuild the bayes again from the samples - especially if you are using "normalize_charset 1" and possibly to make "bayes_token_sources all" if you chose to use it also benefit from the older samples but there is no need to do so, the old bayes don't become useles signature.asc Description: OpenPGP digital signature
Re: SA doesn't respect my user_prefs
On Fri, 11 Sep 2015 16:53:17 +0200 Benny Pedersen wrote: > but there was a dokument error on what -x do on spamd What I found confusing is that --virtual-config-dir doesn't work without -x. In other words you have to set the nouser-config option to make spamd read the user config. > and -u on spamd is not usefull if user_prefs is needed or enabled It is if you have virtual users.
Re: Live upgrade safe?
Am 11.09.2015 um 18:12 schrieb Benny Pedersen: Ian Zimmerman skrev den 2015-09-11 18:05: I appreciate you trying to help, but you don't really answer my question. Even if I could do what you suggest, the rsync would still take finite time - longer than the interval between the upgrade and the restart on the production system. if you recently upgraded: sa-update ... more sa-update if you use custom channels sa-compile restart spamd or other glues sa-compile is only need if you use the plugin put it all in a bash file and run whenever its needed in cron, but not more then daily there is no reason to fiddle around with cron, on distributions with as you call it "precompiled problems" it's taken care that: a) sa-update runs once per day b) on a random timeframe to not overload upstream servers compared all installations doing it at the same moment c) restart the service *only* if there where updates signature.asc Description: OpenPGP digital signature
Re: Live upgrade safe?
Ian Zimmerman skrev den 2015-09-11 17:21: Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? ask the precompiled problem maintainer, not here, your packege is not doing well if that part misses
Re: Live upgrade safe?
On 2015-09-11 17:35 +0200, Reindl Harald wrote: > >>>Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local > >>>configuration files, and without regenerating the Bayes database? (I > >>>use the default bdb Bayes store.) > >> > >>yes, but you need to run "sa-update" before restart to fetch the > >>latest rules and hopefully have a distribution which restarts > >>automatically after update the package > > > >Isn't this a contradiction? If my distribution automatically restarts > >(which it does), how can I sneak in a sa-update run after the upgrade > >but before the restart? > > i hope you have a testing environment for production and so just make > the "sa-update" there and rsync the rule-updates to the liveserver I appreciate you trying to help, but you don't really answer my question. Even if I could do what you suggest, the rsync would still take finite time - longer than the interval between the upgrade and the restart on the production system. -- Please *no* private copies of mailing list or newsgroup messages. Rule 420: All persons more than eight miles high to leave the court.
Re: SA doesn't respect my user_prefs
Guess this means that I have to run "spamassassin" instead of spamc, don't I? I do not understand the reason for spamc to exist then - but based upon the conversation result, it seems like the way to go ... hope my host can handle the load. Am 10.09.2015 um 12:50 schrieb Marc Richter: Hi @ all, maybe I'm doing it wrong here - I do not insist on being unfailable. But what's the correct way to do it then? Best regards, Marc Am 10.09.2015 um 01:48 schrieb RW: On Wed, 9 Sep 2015 14:48:14 -0700 jdow wrote: On 2015-09-09 13:51, RW wrote: On Wed, 9 Sep 2015 17:27:54 +0200 Marc Richter wrote: Hi RW, Do you mean that ww is a unix user? The normal way to do this is to run spamd as root and run spamc as the unix user. Passing -u to spamc is really intended for virtual users, I'm not sure whether it works for unix users. Are you sure it worked before? ww is a unix user, yes. And it worked before, yes. Supporting that sounds like a really bad idea. It would mean that any user could make a spamd child run as any unix user they choose - possibly even root. It's an unnecessary risk of privilege escalation. It also gives users too much access to each other's databases. A malicious user would be able to miss-train another user's Bayes or manipulate reputations in TxRep or AWL. It would also be possible to infer some of the contents of another users TxRep database from suitable test emails. Why don't you try to run spamc -u root as a common user and see what happens then talk about the results if it is warranted? Given that it doesn't appear to be currently working with non-root accounts, what would that prove? And it's still wrong even if root is a special case.
Re: SA doesn't respect my user_prefs
Spamc exists to save startup compilation time. If you have real users and use procmail then spamc will be much faster and pass along the username. If you use a glue or have virtual users, you might need logic to call spamc or spamassassin with a desired username. But for me, I would anticipate switching will just make things slower and not solve the issue. Regards, KAM On September 11, 2015 5:35:12 AM AST, Marc Richterwrote: >Guess this means that I have to run "spamassassin" instead of spamc, >don't I? > >I do not understand the reason for spamc to exist then
Re: SA doesn't respect my user_prefs
I can't disagree as I was answering the why it exists. What are using user prefs to accomplish because I prefer using sql based prefs? Regards, KAM On September 11, 2015 5:50:43 AM AST, Marc Richterwrote: >Hi KAM, > >why not - spamassassin seems to respect the user_prefs file. Of course >I'd like to stick ti spamc, but if there is no solution for the >user_prefs - issue, it fits only half of my needs. > >Best regard, >Marc > >Am 11.09.2015 um 11:47 schrieb Kevin A. McGrail: >> Spamc exists to save startup compilation time. >> >> If you have real users and use procmail then spamc will be much >faster and pass along the username. >> >> If you use a glue or have virtual users, you might need logic to call >spamc or spamassassin with a desired username. But for me, I would >anticipate switching will just make things slower and not solve the >issue. >> >> Regards, >> KAM >> >> On September 11, 2015 5:35:12 AM AST, Marc Richter > wrote: >>> Guess this means that I have to run "spamassassin" instead of spamc, >>> don't I? >>> >>> I do not understand the reason for spamc to exist then >>
Re: SA doesn't respect my user_prefs
Hi KAM, why not - spamassassin seems to respect the user_prefs file. Of course I'd like to stick ti spamc, but if there is no solution for the user_prefs - issue, it fits only half of my needs. Best regard, Marc Am 11.09.2015 um 11:47 schrieb Kevin A. McGrail: Spamc exists to save startup compilation time. If you have real users and use procmail then spamc will be much faster and pass along the username. If you use a glue or have virtual users, you might need logic to call spamc or spamassassin with a desired username. But for me, I would anticipate switching will just make things slower and not solve the issue. Regards, KAM On September 11, 2015 5:35:12 AM AST, Marc Richterwrote: Guess this means that I have to run "spamassassin" instead of spamc, don't I? I do not understand the reason for spamc to exist then
Re: SA doesn't respect my user_prefs
Am 11.09.2015 um 11:35 schrieb Marc Richter: Guess this means that I have to run "spamassassin" instead of spamc, don't I? I do not understand the reason for spamc to exist then uhm because it does the real work? in the case below milter -> spamd -> spamc preforkers [root@mail-gw:~]$ systemctl status spamassassin.service ● spamassassin.service - Spamassassin Daemon Loaded: loaded (/etc/systemd/system/spamassassin.service; enabled) Active: active (running) since Fr 2015-09-11 00:59:27 CEST; 10h ago Process: 24463 ExecReload=/usr/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Process: 10162 ExecStartPre=/usr/bin/find /var/lib/spamassassin/ -type f -exec /bin/chmod 0644 {} ; (code=exited, status=0/SUCCESS) Process: 10153 ExecStartPre=/usr/bin/find /var/lib/spamassassin/ -type d -exec /bin/chmod 0755 {} ; (code=exited, status=0/SUCCESS) Main PID: 10235 (spamd) CGroup: /system.slice/spamassassin.service ├─10235 /usr/bin/perl -T -w /usr/bin/spamd --max-children=20 --min-children=5 --min-spare=5 --max-spare=10 --max-conn-per-child=200 --socketpath=/run/spamassassin/spamassassin.sock --socketmode=0666 ├─24469 spamd chil ├─24470 spamd chil ├─24471 spamd chil ├─24472 spamd chil ├─24473 spamd chil ├─24474 spamd chil ├─24475 spamd chil ├─24476 spamd chil ├─24477 spamd chil └─24478 spamd chil signature.asc Description: OpenPGP digital signature
Re: SA doesn't respect my user_prefs
Marc Richterwrites: > Hi KAM, > > why not - spamassassin seems to respect the user_prefs file. Of course > I'd like to stick ti spamc, but if there is no solution for the > user_prefs - issue, it fits only half of my needs. Sorry for jumping in the conversation, I have not read all the messages, but if I remember well, un order for spamc -u to work, you need to run spamd as high priviledged user. For security reasons, user's .spamassassin directory is readble only by that user. Spamc -u tells spamd to become that user, but spamd must be allowed by the system to change user, that means spamd must be running as root to begin with. So I would say: - start spamd as root - spamc -u user - or become user and spamassassin All this is from memory, because I use SA though amavisd nowdays. Best regards, Olivier > Best regard, > Marc > > Am 11.09.2015 um 11:47 schrieb Kevin A. McGrail: >> Spamc exists to save startup compilation time. >> >> If you have real users and use procmail then spamc will be much faster and >> pass along the username. >> >> If you use a glue or have virtual users, you might need logic to call spamc >> or spamassassin with a desired username. But for me, I would anticipate >> switching will just make things slower and not solve the issue. >> >> Regards, >> KAM >> >> On September 11, 2015 5:35:12 AM AST, Marc Richter >> wrote: >>> Guess this means that I have to run "spamassassin" instead of spamc, >>> don't I? >>> >>> I do not understand the reason for spamc to exist then >> > --
Re: Live upgrade safe?
>Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local >configuration files, and without regenerating the Bayes database? (I >use the default bdb Bayes store.) On 2015-08-14 17:45 +0200, Reindl Harald wrote: yes, but you need to run "sa-update" before restart to fetch the latest rules and hopefully have a distribution which restarts automatically after update the package On 11.09.15 08:21, Ian Zimmerman wrote: Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? if your distribution restarts spamassassin, it will most probably download the rules before. Not everyone uses distributions... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. It's now safe to throw off your computer.
Re: Large volume of 0.0 scores suddenly
On 11 Sep 2015, at 6:12, Peter Kelly wrote: Hi, Starting on 3rd Sept, I have seen a huge number of 0.0 scores being returned from spamassassin - see attached screenshot from my logs that show I never once received a 0.0 score before 3rd Sept. The default scores for the rules shown do not add up to 0.00 in the 2 examples I did the arithmetic manually for. This indicates that whatever is generating those records (I'm guessing it is a version of this: https://github.com/peterkellyonline/mailchecker ?) is doing something wrong, I'd guess in parsing the spamc output.
Re: Large volume of 0.0 scores suddenly
Sorry, yeah - all BAYES rules were gone after --clear. When I trained it with another fresh 1000 spam and ham it started again with BAYES_00. I will need to go through the spam and ham again On 11 September 2015 at 23:26, RWwrote: > On Fri, 11 Sep 2015 22:25:46 +0100 > Peter Kelly wrote: > > > > I can actually see the 0.0 scores directly in the logs > > > I tested one and it was out, but only by 0.04. I thought it was > probably due to your cron job running a bit too early for this morning's > update. > > > > It must be the bayes is completely messed up. I actually did a > > sa-learn --clear and ran a fresh 1000 spam and ham (verified) through > > again, still seeing BAYES_00 an unusually high number of times. > > Are you sure you ran that on the right database? With an empty > database you should see log entries without any BAYES_* rules. > > > I will look into my own DNS nameserver for the RELAY and TRUSTED > > problem. > > ALL_TRUSTED doesn't have anything to do with DNS, it's determined from > the received headers. spamassassin -D is useful for debugging it. >
Re: Large volume of 0.0 scores suddenly
On Fri, 11 Sep 2015 22:25:46 +0100 Peter Kelly wrote: > I can actually see the 0.0 scores directly in the logs I tested one and it was out, but only by 0.04. I thought it was probably due to your cron job running a bit too early for this morning's update. > It must be the bayes is completely messed up. I actually did a > sa-learn --clear and ran a fresh 1000 spam and ham (verified) through > again, still seeing BAYES_00 an unusually high number of times. Are you sure you ran that on the right database? With an empty database you should see log entries without any BAYES_* rules. > I will look into my own DNS nameserver for the RELAY and TRUSTED > problem. ALL_TRUSTED doesn't have anything to do with DNS, it's determined from the received headers. spamassassin -D is useful for debugging it.
Re: Large volume of 0.0 scores suddenly
Bill, I checked there first, I always assume it is something I am doing wrong first. Yes mailchecker (not that obsolete version) is the http service we use and it in turn uses this Golang lib for spamc - https://github.com/saintienn/go-spamc I can actually see the 0.0 scores directly in the logs at /var/log/mail.log on the spamassassin servers. E.g. Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: clean message (0.0/5.0) for (unknown):65534 in 0.4 seconds, 8128 bytes. Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: result: . 0 - BAYES_00,FREEMAIL_FROM,HTML_IMAGE_ONLY_32,HTML_MESSAGE,HTML_TAG_BALANCE_BODY,NO_RELAYS,SPF_NEUTRAL,T_DKIM_INVALID,URIBL_BLOCKED It must be the bayes is completely messed up. I actually did a sa-learn --clear and ran a fresh 1000 spam and ham (verified) through again, still seeing BAYES_00 an unusually high number of times. I will look into my own DNS nameserver for the RELAY and TRUSTED problem. On 11 September 2015 at 21:58, Bill Cole < sausers-20150...@billmail.scconsult.com> wrote: > On 11 Sep 2015, at 6:12, Peter Kelly wrote: > > Hi, >> >> Starting on 3rd Sept, I have seen a huge number of 0.0 scores being >> returned from spamassassin - see attached screenshot from my logs that >> show >> I never once received a 0.0 score before 3rd Sept. >> > > The default scores for the rules shown do not add up to 0.00 in the 2 > examples I did the arithmetic manually for. > > This indicates that whatever is generating those records (I'm guessing it > is a version of this: https://github.com/peterkellyonline/mailchecker ?) > is doing something wrong, I'd guess in parsing the spamc output. >
Re: Fwd: Large volume of 0.0 scores suddenly
On 09/11/2015 03:13 PM, Peter Kelly wrote: Axb, We have a SaaS app hosted in AWS that takes in 500k emails a month. We parse these emails and convert them into tickets for the customer - they see a Helpdesk system like Zendesk. Every incoming email gets run through spamassassin via the daemon. Here is a link to the output of --lint -Dhttp://pastebin.com/8eM88hX2 btw: Sep 11 13:14:04.305 [2812] dbg: diag: [...] module not installed: Encode::Detect ('require' failed) Sep 11 13:14:04.305 [2812] dbg: diag: [...] module not installed: Digest::SHA1 ('require' failed) etc... suggest you install the "required" modules and see if spamassassin --lint -D detects them. missing modules may "disable" features...
Re: SA doesn't respect my user_prefs
Hi Olivier, thanks for your ideas, they look reasonable. But I think it might be not the solution, since 1. my spamd runs as spamd:spamd and my home-dirs/-files have rw permissions for at least group spamd: ww@tango012 ~ $ ls -ald .spamassassin .spamassassin/* drwxrwx--- 2 wwspamd 4096 11. Sep 11:40 .spamassassin -rw-rw 1 wwspamd 10387456 27. Aug 14:19 .spamassassin/auto-whitelist -rw--- 1 spamd spamd6 27. Aug 14:19 .spamassassin/auto-whitelist.mutex -rw-rw 1 wwspamd 8667 11. Sep 11:40 .spamassassin/user_prefs ww@tango012 ~ $ 2. I nevertheless tried to run spamd as root and this is what it results in: spamd: cannot run as nonexistent user or root with -u option Best regards, Marc Am 11.09.2015 um 12:05 schrieb Olivier Nicole: Marc Richterwrites: Hi KAM, why not - spamassassin seems to respect the user_prefs file. Of course I'd like to stick ti spamc, but if there is no solution for the user_prefs - issue, it fits only half of my needs. Sorry for jumping in the conversation, I have not read all the messages, but if I remember well, un order for spamc -u to work, you need to run spamd as high priviledged user. For security reasons, user's .spamassassin directory is readble only by that user. Spamc -u tells spamd to become that user, but spamd must be allowed by the system to change user, that means spamd must be running as root to begin with. So I would say: - start spamd as root - spamc -u user - or become user and spamassassin All this is from memory, because I use SA though amavisd nowdays. Best regards, Olivier Best regard, Marc Am 11.09.2015 um 11:47 schrieb Kevin A. McGrail: Spamc exists to save startup compilation time. If you have real users and use procmail then spamc will be much faster and pass along the username. If you use a glue or have virtual users, you might need logic to call spamc or spamassassin with a desired username. But for me, I would anticipate switching will just make things slower and not solve the issue. Regards, KAM On September 11, 2015 5:35:12 AM AST, Marc Richter wrote: Guess this means that I have to run "spamassassin" instead of spamc, don't I? I do not understand the reason for spamc to exist then
Re: Fwd: Large volume of 0.0 scores suddenly
On 09/11/2015 03:13 PM, Peter Kelly wrote: Axb, We have a SaaS app hosted in AWS that takes in 500k emails a month. We parse these emails and convert them into tickets for the customer - they see a Helpdesk system like Zendesk. Every incoming email gets run through spamassassin via the daemon. Here is a link to the output of --lint -D http://pastebin.com/8eM88hX2 is the app feeding spamd directly or are you using spamc ? or using the API interface? can you get hold of one of those pristine messages and test them manually against spamassasssin ? if the results look massively different, chances is that your app is not doing the right thing and like Matus suspects, SA is not getting the right thing. Rules & scores do change via sa-update so depending on lots of stuff the results may vary, possibly quite a lot. As we don't have a sample msg of yours (pastebin) we can't compare with any other setups... ball over... On 11 September 2015 at 13:08, Axbwrote: On 09/11/2015 01:17 PM, Peter Kelly wrote: - How are you using SA? (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail, Fuglu, etc, etc) Just spamassassin on its own, calling the daemon from an app an "app"? Pls be more explicit. can you pastebin the output of spamassassin --lint -D - Are you using a local, non forwarding, DNS resolver/caching server ? No you should, to avoid URIBL_BLOCKED (http://uribl.com/refused.shtml)
Re: SA doesn't respect my user_prefs
Am 11.09.2015 um 16:05 schrieb Marc Richter: thanks for your ideas, they look reasonable. But I think it might be not the solution, since 1. my spamd runs as spamd:spamd and my home-dirs/-files have rw permissions for at least group spamd: ww@tango012 ~ $ ls -ald .spamassassin .spamassassin/* drwxrwx--- 2 wwspamd 4096 11. Sep 11:40 .spamassassin -rw-rw 1 wwspamd 10387456 27. Aug 14:19 .spamassassin/auto-whitelist -rw--- 1 spamd spamd6 27. Aug 14:19 .spamassassin/auto-whitelist.mutex -rw-rw 1 wwspamd 8667 11. Sep 11:40 .spamassassin/user_prefs ww@tango012 ~ $ 2. I nevertheless tried to run spamd as root and this is what it results in: spamd: cannot run as nonexistent user or root with -u option spamd must not be startet with the -u option as root, the whole purpose is to have the daemon process running as root and then "spamc" is invoked with the -u param of the user which is target of the incoming message Am 11.09.2015 um 12:05 schrieb Olivier Nicole: Marc Richterwrites: Hi KAM, why not - spamassassin seems to respect the user_prefs file. Of course I'd like to stick ti spamc, but if there is no solution for the user_prefs - issue, it fits only half of my needs. Sorry for jumping in the conversation, I have not read all the messages, but if I remember well, un order for spamc -u to work, you need to run spamd as high priviledged user. For security reasons, user's .spamassassin directory is readble only by that user. Spamc -u tells spamd to become that user, but spamd must be allowed by the system to change user, that means spamd must be running as root to begin with. So I would say: - start spamd as root - spamc -u user - or become user and spamassassin All this is from memory, because I use SA though amavisd nowdays. Best regards, Olivier Best regard, Marc Am 11.09.2015 um 11:47 schrieb Kevin A. McGrail: Spamc exists to save startup compilation time. If you have real users and use procmail then spamc will be much faster and pass along the username. If you use a glue or have virtual users, you might need logic to call spamc or spamassassin with a desired username. But for me, I would anticipate switching will just make things slower and not solve the issue. Regards, KAM On September 11, 2015 5:35:12 AM AST, Marc Richter wrote: Guess this means that I have to run "spamassassin" instead of spamc, don't I? I do not understand the reason for spamc to exist then -- Reindl Harald the lounge interactive design GmbH A-1060 Vienna, Hofmühlgasse 17 CTO / CISO / Software-Development m: +43 (676) 40 221 40, p: +43 (1) 595 3999 33 icq: 154546673, http://www.thelounge.net/ http://www.thelounge.net/signature.asc.what.htm signature.asc Description: OpenPGP digital signature
Re: Fwd: Large volume of 0.0 scores suddenly
No, no changes. Run a manual check with -D and look for issues. Maybe your sql password changed or something that your install uses? I would also look at the uribl blocked issue. Maybe that started on the 3rd for you? Perhaps your dns server is not working right and causing timeouts. See https://wiki.apache.org/spamassassin/DnsBlocklists under the first faq. Regards, KAM On September 11, 2015 6:12:14 AM AST, Peter Kellywrote: >Hi, > >Starting on 3rd Sept, I have seen a huge number of 0.0 scores being >returned from spamassassin - see attached screenshot from my logs that >show >I never once received a 0.0 score before 3rd Sept. > >I use version 3.4.0 and process about 20k emails a day through it. I >used >bayes and this has been regularly updated with 1000 ham and spam emails >(every months or so). Autolearning is on, at the default scores (0.1 >and >12.0). > >I have the cronjob enabled to update the rules nightly. Did anything >change >on 3rd Sept that would explain this? Nothing has changed in my >configuration of spamassassin in months. I am now seeing a huge amount >of >0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never >have. > >Any help greatly appreciated, > >Peter
Re: Fwd: Large volume of 0.0 scores suddenly
Peter Kellywrites: > [1:multipart/alternative Hide] > > > [1/1:text/plain Hide] > > Hi, > > Starting on 3rd Sept, I have seen a huge number of 0.0 scores being > returned from spamassassin - see attached screenshot from my logs that show > I never once received a 0.0 score before 3rd Sept. Like others said, on 7 days backlog, the score closer to zero was 0.051 I am useing SA 3.4.1 with sa-update daily. Olivier > I use version 3.4.0 and process about 20k emails a day through it. I used > bayes and this has been regularly updated with 1000 ham and spam emails > (every months or so). Autolearning is on, at the default scores (0.1 and > 12.0). > > I have the cronjob enabled to update the rules nightly. Did anything change > on 3rd Sept that would explain this? Nothing has changed in my > configuration of spamassassin in months. I am now seeing a huge amount of > 0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never > have. > > Any help greatly appreciated, > > Peter > > [1/2:text/html Show] > > > [2:image/png Show Save:Screen Shot 2015-09-11 at 10.42.16 AM.png (498kB)] > --
Re: Fwd: Large volume of 0.0 scores suddenly
On 09/11/2015 12:12 PM, Peter Kelly wrote: Hi, Starting on 3rd Sept, I have seen a huge number of 0.0 scores being returned from spamassassin - see attached screenshot from my logs that show I never once received a 0.0 score before 3rd Sept. I use version 3.4.0 and process about 20k emails a day through it. I used bayes and this has been regularly updated with 1000 ham and spam emails (every months or so). Autolearning is on, at the default scores (0.1 and 12.0). I have the cronjob enabled to update the rules nightly. Did anything change on 3rd Sept that would explain this? Nothing has changed in my configuration of spamassassin in months. I am now seeing a huge amount of 0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never have. You're not giving us much information to help you... pls see: https://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/emailed/sa-list-template.txt and try to provide us with as much info as possible. iow, pls help *us* help *you*
Re: Fwd: Large volume of 0.0 scores suddenly
please keep list mail on list... On 09/11/2015 01:17 PM, Peter Kelly wrote: - Please post missed spam samples in pastebin.com - do not post samples to mailing lists I'll post example shortly - What SA version are you using? and on what operating system? 3.4.0 on Ubuntu 14.04 - How are you using SA? (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail, Fuglu, etc, etc) Just spamassassin on its own, calling the daemon from an app - Are you using SA in a PC/notebook? or on a server? Server - What plugins are you using? (pls specify: Razor, Pyzor, DCC, etc) Razor, Pyzor - Are you using RBLs? (specify: at SMTP level, only SA's lookups, etc) SA lookups - Are you using any additional rulesets? No - Are you using a local, non forwarding, DNS resolver/caching server ? No - Are you using per/user or site wide Bayes? site-wide - What Bayes backend are you using? (specify: default, SDBM, SQL, Redis, other) default, file - Are you handling mail for a company, personal email, ISP, one domain, many domains, etc? Handling mail for thousands of different companies - we run a SaaS Helpdesk system like Zendesk. We see a huge range of emails and domains. On 11 September 2015 at 11:22, Axbwrote: On 09/11/2015 12:12 PM, Peter Kelly wrote: Hi, Starting on 3rd Sept, I have seen a huge number of 0.0 scores being returned from spamassassin - see attached screenshot from my logs that show I never once received a 0.0 score before 3rd Sept. I use version 3.4.0 and process about 20k emails a day through it. I used bayes and this has been regularly updated with 1000 ham and spam emails (every months or so). Autolearning is on, at the default scores (0.1 and 12.0). I have the cronjob enabled to update the rules nightly. Did anything change on 3rd Sept that would explain this? Nothing has changed in my configuration of spamassassin in months. I am now seeing a huge amount of 0.0 scores and TRUSTED_ALL rules. I have no trusted_networks set, never have. You're not giving us much information to help you... pls see: https://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/emailed/sa-list-template.txt and try to provide us with as much info as possible. iow, pls help *us* help *you*
Re: Fwd: Large volume of 0.0 scores suddenly
On 09/11/2015 01:17 PM, Peter Kelly wrote: - How are you using SA? (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail, Fuglu, etc, etc) Just spamassassin on its own, calling the daemon from an app an "app"? Pls be more explicit. can you pastebin the output of spamassassin --lint -D - Are you using a local, non forwarding, DNS resolver/caching server ? No you should, to avoid URIBL_BLOCKED (http://uribl.com/refused.shtml)
Re: Large volume of 0.0 scores suddenly
On 11 Sep 2015, at 17:25, Peter Kelly wrote: Bill, I checked there first, I always assume it is something I am doing wrong first. Yes mailchecker (not that obsolete version) is the http service we use and it in turn uses this Golang lib for spamc - https://github.com/saintienn/go-spamc I can actually see the 0.0 scores directly in the logs at /var/log/mail.log on the spamassassin servers. E.g. Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: clean message (0.0/5.0) for (unknown):65534 in 0.4 seconds, 8128 bytes. Sep 11 21:07:19 ip-10-181-62-231 spamd[13929]: spamd: result: . 0 - BAYES_00,FREEMAIL_FROM,HTML_IMAGE_ONLY_32,HTML_MESSAGE,HTML_TAG_BALANCE_BODY,NO_RELAYS,SPF_NEUTRAL,T_DKIM_INVALID,URIBL_BLOCKED Hmmm That reminds me: spamd only returns a score with a single decimal place, so the fact that this one would add up to 0.039 doesn't make the log line wrong in the way I thought your screenshot was wrong. However, it DOES make any tool reporting greater precision (i.e. the "0.00" values in your screenshot) misleading by design, so the problem isn't (probably) misparsing but incorrect conversion of 0.0 to 0.00. It must be the bayes is completely messed up. I actually did a sa-learn --clear and ran a fresh 1000 spam and ham (verified) through again, still seeing BAYES_00 an unusually high number of times. I will look into my own DNS nameserver for the RELAY and TRUSTED problem. That's not a DNS issue, it's a configuration issue and/or a data preparation issue. Messages that match NO_RELAYS are either not properly formatted, have had their headers mangled, or originated on the same machine where they were delivered. ALL_TRUSTED can result from lesser header damage and/or a mis-specified trusted_networks setting in SpamAssassin.
Re: Live upgrade safe?
Am 11.09.2015 um 21:08 schrieb Matus UHLAR - fantomas: >Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local >configuration files, and without regenerating the Bayes database? (I >use the default bdb Bayes store.) On 2015-08-14 17:45 +0200, Reindl Harald wrote: yes, but you need to run "sa-update" before restart to fetch the latest rules and hopefully have a distribution which restarts automatically after update the package On 11.09.15 08:21, Ian Zimmerman wrote: Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? if your distribution restarts spamassassin, it will most probably download the rules before. Not everyone uses distributions... no, the service restarts are usually rpm-macros in the %post section and not invoke sa-update signature.asc Description: OpenPGP digital signature
Re: Fwd: Large volume of 0.0 scores suddenly
> On 09/11/2015 01:17 PM, Peter Kelly wrote: > > - Are you using a local, non forwarding, DNS resolver/caching server ? > > > > No > > - Are you handling mail for a company, personal email, ISP, one domain, > > many domains, etc? > > > > Handling mail for thousands of different companies - we run a SaaS > > Helpdesk system like Zendesk. We see a huge range of emails and > > domains. Without a local DNS server I'm amazed you haven't had problems before now. Antony. -- I want to build a machine that will be proud of me. - Danny Hillis, creator of The Connection Machine Please reply to the list; please *don't* CC me.
Re: Fwd: Large volume of 0.0 scores suddenly
Why Antony? What would that do for me other than save hits against URIBL? I am signing up for their paid service so I will not have the URIBL_BLOCKED issue anymore. It does not explain the 0.0 issue I am having anyway. On 11 September 2015 at 13:42, Antony Stone < antony.st...@spamassassin.open.source.it> wrote: > > On 09/11/2015 01:17 PM, Peter Kelly wrote: > > > > - Are you using a local, non forwarding, DNS resolver/caching server ? > > > > > > No > > > > - Are you handling mail for a company, personal email, ISP, one domain, > > > many domains, etc? > > > > > > Handling mail for thousands of different companies - we run a SaaS > > > Helpdesk system like Zendesk. We see a huge range of emails and > > > domains. > > Without a local DNS server I'm amazed you haven't had problems before now. > > > Antony. > > -- > I want to build a machine that will be proud of me. > > - Danny Hillis, creator of The Connection Machine > >Please reply to the > list; > please *don't* CC > me. >
Re: Large volume of 0.0 scores suddenly
Am 11.09.2015 um 15:03 schrieb Peter Kelly: Why Antony? What would that do for me other than save hits against URIBL? I am signing up for their paid service so I will not have the URIBL_BLOCKED issue anymore. It does not explain the 0.0 issue I am having anyway. what is so hard to understand in the fact that when you use a DNS forwarder shared with other people any DNSBL/URIBL sees always the same IP (not yours) and so the summary *of all users* using that DNS server in other words: you don't need to pay for them until you not have your own recursion resolver because URIBL_BLOCKED won't go away in that case signature.asc Description: OpenPGP digital signature
Re: Fwd: Large volume of 0.0 scores suddenly
Axb, We have a SaaS app hosted in AWS that takes in 500k emails a month. We parse these emails and convert them into tickets for the customer - they see a Helpdesk system like Zendesk. Every incoming email gets run through spamassassin via the daemon. Here is a link to the output of --lint -D http://pastebin.com/8eM88hX2 On 11 September 2015 at 13:08, Axbwrote: > On 09/11/2015 01:17 PM, Peter Kelly wrote: > >> - How are you using SA? >> (pls specify: amavis, MIMEDefang, a milter, Mailscanner, procmail, >> Fuglu, etc, etc) >> >> Just spamassassin on its own, calling the daemon from an app >> > > an "app"? Pls be more explicit. > > can you pastebin the output of > > spamassassin --lint -D > > > - Are you using a local, non forwarding, DNS resolver/caching server ? >> >> No >> > > you should, to avoid URIBL_BLOCKED > (http://uribl.com/refused.shtml) > > > >
Re: Fwd: Large volume of 0.0 scores suddenly
Peter Kelly skrev den 2015-09-11 15:01: This has nothing to do with URIBL. It has always been blocked for me. I am in the process of paying for their service. It has always been like that, yet the 0.0 scores only started last week. Been running for months before that. so you already have a local dns server, and is now recieving so much spam that you need to pay uribl for dataservice ? wish it was me :=) but my point is that a missing local dns could olso show other problems
Re: Fwd: Large volume of 0.0 scores suddenly
On 11.09.15 14:13, Peter Kelly wrote: We have a SaaS app hosted in AWS that takes in 500k emails a month. We parse these emails and convert them into tickets for the customer - they see a Helpdesk system like Zendesk. Every incoming email gets run through spamassassin via the daemon. does spamassassin see the whole e-mails, including all headers? the NO_RELAYS test looks like you are not pushing the whole mails to SA, which significantly decreases score. Also, do you have way to train your spamassassin with spam? Since all spams have BAYES_00, you should train them... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Your mouse has moved. Windows NT will now restart for changes to take to take effect. [OK]
Re: Fwd: Large volume of 0.0 scores suddenly
Peter Kelly skrev den 2015-09-11 12:12: Any help greatly appreciated, google URIBL_BLOCKED https://www.google.dk/search?q=uribl_blcoked http://uribl.com/refused.shtml plenty of other links to see how and why do you miss a local dns resolver ? if yes you use shared problems and things like your questions come up randomly when more people dont read about it, its free to do nothing :=)
Re: Fwd: Large volume of 0.0 scores suddenly
Hi Benny, This has nothing to do with URIBL. It has always been blocked for me. I am in the process of paying for their service. It has always been like that, yet the 0.0 scores only started last week. Been running for months before that. Peter On 11 September 2015 at 13:38, Benny Pedersenwrote: > Peter Kelly skrev den 2015-09-11 12:12: > > Any help greatly appreciated, >> > > google URIBL_BLOCKED > > https://www.google.dk/search?q=uribl_blcoked > > http://uribl.com/refused.shtml > > plenty of other links to see how and why > > do you miss a local dns resolver ? > > if yes you use shared problems and things like your questions come up > randomly when more people dont read about it, its free to do nothing :=) >
Re: SA doesn't respect my user_prefs
Am 09.09.2015 um 15:01 schrieb Matus UHLAR - fantomas: how do you plug spamassassin into your mail flow? How do you call spamassassin? mta, mail client ... ? On 09.09.15 16:11, Marc Richter wrote: I'm running postfix as my MTA. In it's master.cf there is configured to pipe my mail through a script: smtp inet n - n - - smtpd -o content_filter=spamassassin spamassassin unix - n n - - pipe flags=Rq user=spamd argv=/var/lib/spamassassin/filter.sh -oi -f ${sender} ${recipient} have you tried running spamass-milter? I haven't tried it with postfix, but runs fine with sendmail, supporting different users... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Due to unexpected conditions Windows 2000 will be released in first quarter of year 1901
Re: Large volume of 0.0 scores suddenly
Reindl Harald skrev den 2015-09-11 15:08: in other words: you don't need to pay for them until you not have your own recursion resolver because URIBL_BLOCKED won't go away in that case its just badly marketing :=)
Re: SA doesn't respect my user_prefs
Hi @ everyone, GOTCHA ! Finally, I found the solution myself: The issue is in the systemd spamassassin.service unit file of Arch Linux! This is how /usr/lib/systemd/system/spamassassin.service looks like: [Unit] Description=Spamassassin daemon After=syslog.target network.target [Service] ExecStart=/usr/bin/vendor_perl/spamd -x -u spamd -g spamd StandardOutput=null StandardError=null Restart=always [Install] WantedBy=multi-user.target Looking at https://spamassassin.apache.org/full/3.0.x/dist/doc/spamd.html , it isn't clear what exactly "-x" is doing, since it is listed within one single line of two opposite clear-text options: """ -x, --nouser-config, --user-config Turn off(on) reading of per-user configuration files (user_prefs) from the user's home directory. The default behaviour is to read per-user configuration from the user's home directory. """ So, -x could have meant both, to turn this on or off in my reading. More clearly this is written in the manpage of spamd: -x, --nouser-config Disable user config files Seems as if when -x is set, "allow_user_rules 1" neither has any effect, nor is a warning printed anywhere that there are opposite options in place or ignored, nor has this apeared in Debuging output. I have solved this by 1. cp /usr/lib/systemd/system/spamassassin.service /etc/systemd/system/ 2. Changed "ExecStart=" from "/usr/bin/vendor_perl/spamd -x -u spamd -g spamd" to "/usr/bin/vendor_perl/spamd -u spamd -g spamd" 3. systemctl daemon-reload 4. systemctl restart spamassassin Now it works again like a charm, running spamd as spamd:spamd, and using spamc. Thanks @ all for trying to help in this case! :) Best regards, Marc Am 09.09.2015 um 08:46 schrieb Marc Richter: Hi everyone, I'm running SA 3.4.1 with Perl 5.22.0 . It works quite well, but since a few weeks, it looks like my user_prefs isn't taken into account by SA anymore. Let's show this by example: There are *lots* of blacklist_from entries in there; one of them is: blacklist_from *@neuronation.* Today, I got another mail with the following (relevant) headers: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tango012.marc-richter.info X-Spam-Level: *** X-Spam-Status: No, score=3.6 required=4.0 tests=BAYES_99,BAYES_999,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,RP_MATCHES_RCVD,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.1 From: NeuroNationDate: Wed, 09 Sep 2015 06:05:02 + (UTC) Thus, this mail should get +100 for matching my blacklist_from entry. But, as you can see, it isn't. When I'm running "spamassassin --test-mode < my_maildir_file", I get expected results: spamassassin --test-mode < .maildir/cur/msg.SbGC\:2\,S [...] Inhaltsanalyse im Detail: (99.9 Punkte, 3.0 ben�tigt) Pkte Regelname Beschreibung -- -- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: neuronation.de] -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [192.254.116.16 listed in wl.mailspike.net] 100 USER_IN_BLACKLIST From: address is in the user's black-list 0.0 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail domains are different -0.0 SPF_PASS SPF: Senderechner entspricht SPF-Datensatz 0.0 RP_MATCHES_RCVDEnvelope sender domain matches handover relay domain 0.0 HTML_MESSAGE BODY: Nachricht enth�lt HTML -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNEDMessage has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders SA is started by postfix; in the master.cf of postfix there are these lines: smtp inet n-n--smtpd -o content_filter=spamassassin spamassassin unix -nn--pipe flags=Rq user=spamfilter argv=/home/spamfilter/filter.sh -oi -f ${sender} ${recipient} /home/spamfilter/filter.sh contains: #!/bin/sh # filter.sh # # This script redirects mail flagged as spam to a separate account # You must first create a user account named "spamvac" to hold the flagged mail SENDMAIL="/usr/sbin/sendmail -i" SPAMASSASSIN=/usr/bin/vendor_perl/spamc COMMAND="$SENDMAIL $@" USER=`echo $COMMAND | awk '{ print $NF }' | sed 's/@.*$//'` NEW_COMMAND=`echo $COMMAND | awk '{ $6 = "spamfilter"; NF = 6; print }'` # Exit codes from EX_TEMPFAIL=75 EX_UNAVAILABLE=69 umask 077
Re: SA doesn't respect my user_prefs
Reindl Harald skrev den 2015-09-11 16:12: spamd: cannot run as nonexistent user or root with -u option spamd must not be startet with the -u option as root, the whole purpose is to have the daemon process running as root and then "spamc" is invoked with the -u param of the user which is target of the incoming message any daemons binding to port below 1024 must be started as root, knowing this could solve alot of problems on maillists :=) but there was a dokument error on what -x do on spamd note apache or postfix or dovecot droppriveleges to not serve daemonsd as root later, this is very easy to see in top, for apache that there is ONE apache running as root, but multiple apache not running as root spamd does the same seen from spamc and -u on spamd is not usefull if user_prefs is needed or enabled
Re: Live upgrade safe?
On 2015-08-14 17:45 +0200, Reindl Harald wrote: > >Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local > >configuration files, and without regenerating the Bayes database? (I > >use the default bdb Bayes store.) > > yes, but you need to run "sa-update" before restart to fetch the > latest rules and hopefully have a distribution which restarts > automatically after update the package Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? -- Please *no* private copies of mailing list or newsgroup messages. Rule 420: All persons more than eight miles high to leave the court.
Re: SA doesn't respect my user_prefs
Am 11.09.2015 um 16:53 schrieb Benny Pedersen: Reindl Harald skrev den 2015-09-11 16:12: spamd: cannot run as nonexistent user or root with -u option spamd must not be startet with the -u option as root, the whole purpose is to have the daemon process running as root and then "spamc" is invoked with the -u param of the user which is target of the incoming message any daemons binding to port below 1024 must be started as root, knowing this could solve alot of problems on maillists :=) but there was a dokument error on what -x do on spamd note apache or postfix or dovecot droppriveleges to not serve daemonsd as root later, this is very easy to see in top, for apache that there is ONE apache running as root, but multiple apache not running as root spamd does the same seen from spamc and -u on spamd is not usefull if user_prefs is needed or enabled exactly what i said signature.asc Description: OpenPGP digital signature
Re: Live upgrade safe?
Am 11.09.2015 um 17:21 schrieb Ian Zimmerman: On 2015-08-14 17:45 +0200, Reindl Harald wrote: Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any local configuration files, and without regenerating the Bayes database? (I use the default bdb Bayes store.) yes, but you need to run "sa-update" before restart to fetch the latest rules and hopefully have a distribution which restarts automatically after update the package Isn't this a contradiction? If my distribution automatically restarts (which it does), how can I sneak in a sa-update run after the upgrade but before the restart? i hope you have a testing environment for production and so just make the "sa-update" there and rsync the rule-updates to the liveserver signature.asc Description: OpenPGP digital signature
Re: Live upgrade safe?
On Fri, 11 Sep 2015 08:21:15 -0700 Ian Zimmerman wrote: > On 2015-08-14 17:45 +0200, Reindl Harald wrote: > > > >Can I safely upgrade SA from 3.4.0 to 3.4.1 without changing any > > >local configuration files, and without regenerating the Bayes > > >database? (I use the default bdb Bayes store.) > > > > yes, but you need to run "sa-update" before restart to fetch the > > latest rules and hopefully have a distribution which restarts > > automatically after update the package > > Isn't this a contradiction? If my distribution automatically restarts > (which it does), how can I sneak in a sa-update run after the upgrade > but before the restart? You need a restart to run the new software or pickup new rules. You don't need to avoid a restart between a package update and a rule update. If you are running sa-update from cron you don't really need to run it manually unless you are updating from a very old version that didn't support sa-update or no longer receives updates.
Re: Live upgrade safe?
Ian Zimmerman skrev den 2015-09-11 18:05: I appreciate you trying to help, but you don't really answer my question. Even if I could do what you suggest, the rsync would still take finite time - longer than the interval between the upgrade and the restart on the production system. if you recently upgraded: sa-update ... more sa-update if you use custom channels sa-compile restart spamd or other glues sa-compile is only need if you use the plugin put it all in a bash file and run whenever its needed in cron, but not more then daily