filter-spamassassin gone?

2016-07-26 Thread Simon McFarlane 

Hi,

I used to run filter-spamassassin on my MX, but after the filter API 
bump, the filter no longer worked and I had to disable spam filtering 
(aside from spamd). Now, it looks like opensmtpd-extras-spamassassin is 
missing from ports.


Is filter-spamassassin gone for good?

Thanks,
Simon

--
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin crashes smtpd on boot

2016-06-30 Thread Joerg Jung 
Yes, me introduced this buggy behaviour in the recent snapshots in the 
resolving function and 
due to the refactoring to non-blocking IO. The resolving should and will be 
changed to happen async 
and non-blocking as well. As I said, known bug, and we are working on this, 
sorry for the inconvenience. 

> On 30 Jun 2016, at 07:24, Jason Tubnor <ja...@tubnor.net> wrote:
> 
> Same applies when smtpd is started at boot on OpenBSD as the smtpd instance 
> in the system starts before pkg_scripts (spamassassin), thus breaking and not 
> processing email until smtpd has been restarted.
> 
> On 30 June 2016 at 14:42, Joerg Jung <m...@umaxx.net <mailto:m...@umaxx.net>> 
> wrote:
> 
> 
> > Am 30.06.2016 um 00:41 schrieb Simon McFarlane <s...@desu.ne.jp 
> > <mailto:s...@desu.ne.jp>>:
> >
> > Hi,
> >
> > I have a rather uninteresting setup consisting of Dovecot and OpenSMTPD 
> > with few configuration tweaks on an OpenBSD system. The only filter I have 
> > is filter-spamassassin. I'm running the latest -current snapshot and filter 
> > and extras from pkg.
> >
> > I can rcctl start smtpd just fine, however, smtpd fails to start 
> > automatically on boot. Every time I reboot the system, I have to shell back 
> > in and start smtpd manually. I get this in /var/log/maillog:
> >
> > Jun 29 22:24:01 mx smtpd[76577]: info: OpenSMTPD 5.9.1 starting
> > Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> control[77736] fd=4
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> klondike[64252] fd=5
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> control[77736] fd=8
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> lookup[45356] fd=6
> > Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> control[77736] fd=6
> > Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> pony 
> > express[82409] fd=5
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> pony express[82409] 
> > fd=7
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> pony express[82409] 
> > fd=9
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> queue[65295] fd=8
> > Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> pony express[82409] 
> > fd=7
> > Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> control[77736] 
> > fd=9
> > Jun 29 22:24:01 mx smtpd[64252]: setup_proc: klondike done
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: ca[64252] done
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> lookup[45356] fd=10
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> scheduler[50073] 
> > fd=9
> > Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> queue[65295] fd=8
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> control[77736] 
> > fd=7
> > Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> queue[65295] fd=10
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> scheduler[50073] fd=11
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: control[77736] done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> 
> > klondike[64252] fd=8
> > Jun 29 22:24:01 mx smtpd[77736]: setup_proc: control done
> > Jun 29 22:24:01 mx smtpd[45356]: setup_proc: lookup done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> lookup[45356] 
> > fd=9
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: lka[45356] done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> queue[65295] 
> > fd=10
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: pony[82409] done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_proc: pony express done
> > Jun 29 22:24:01 mx smtpd[65295]: setup_proc: queue done
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: queue[65295] done
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: scheduler[50073] done
> > Jun 29 22:24:01 mx smtpd[50073]: setup_proc: scheduler done
> > Jun 29 22:24:01 mx smtpd[79591]: smtpd: setup done
> > Jun 29 22:24:01 mx filter-spamassassin[81659]: fatal: resolve: failed
> > Jun 29 22:24:01 mx smtpd[82409]: warn: pony -> filter-spamassassin: pipe 
> > closed
> > Jun 29 22:24:01 mx smtpd[82409]: warn: filter "filter-spamassassin" closed 
> > unexpectedly
> > Jun 29 22:24:01 mx smtpd[82409]: fatal: exiting
> > Jun 29 22:24:01 mx smtpd[79591]: warn: parent -> pony: pipe closed
> > Jun 29 22:24:01 mx smtpd[64252]: warn: ca -> pony express: pipe closed
> > Jun 29 22:24:01 mx smtpd[77736]: warn: control -> pony express: pipe closed
> > Jun 29 22:24:01 mx smt

Re: filter-spamassassin crashes smtpd on boot

2016-06-29 Thread Jason Tubnor 
Same applies when smtpd is started at boot on OpenBSD as the smtpd instance
in the system starts before pkg_scripts (spamassassin), thus breaking and
not processing email until smtpd has been restarted.

On 30 June 2016 at 14:42, Joerg Jung <m...@umaxx.net> wrote:

>
>
> > Am 30.06.2016 um 00:41 schrieb Simon McFarlane <s...@desu.ne.jp>:
> >
> > Hi,
> >
> > I have a rather uninteresting setup consisting of Dovecot and OpenSMTPD
> with few configuration tweaks on an OpenBSD system. The only filter I have
> is filter-spamassassin. I'm running the latest -current snapshot and filter
> and extras from pkg.
> >
> > I can rcctl start smtpd just fine, however, smtpd fails to start
> automatically on boot. Every time I reboot the system, I have to shell back
> in and start smtpd manually. I get this in /var/log/maillog:
> >
> > Jun 29 22:24:01 mx smtpd[76577]: info: OpenSMTPD 5.9.1 starting
> > Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> control[77736]
> fd=4
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> klondike[64252]
> fd=5
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> control[77736] fd=8
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> lookup[45356]
> fd=6
> > Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> control[77736]
> fd=6
> > Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> pony
> express[82409] fd=5
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> pony
> express[82409] fd=7
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> pony
> express[82409] fd=9
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> queue[65295] fd=8
> > Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> pony
> express[82409] fd=7
> > Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> control[77736]
> fd=9
> > Jun 29 22:24:01 mx smtpd[64252]: setup_proc: klondike done
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: ca[64252] done
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> lookup[45356] fd=10
> > Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> scheduler[50073]
> fd=9
> > Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> queue[65295] fd=8
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express ->
> control[77736] fd=7
> > Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> queue[65295]
> fd=10
> > Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> scheduler[50073]
> fd=11
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: control[77736] done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express ->
> klondike[64252] fd=8
> > Jun 29 22:24:01 mx smtpd[77736]: setup_proc: control done
> > Jun 29 22:24:01 mx smtpd[45356]: setup_proc: lookup done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express ->
> lookup[45356] fd=9
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: lka[45356] done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express ->
> queue[65295] fd=10
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: pony[82409] done
> > Jun 29 22:24:01 mx smtpd[82409]: setup_proc: pony express done
> > Jun 29 22:24:01 mx smtpd[65295]: setup_proc: queue done
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: queue[65295] done
> > Jun 29 22:24:01 mx smtpd[79591]: setup_done: scheduler[50073] done
> > Jun 29 22:24:01 mx smtpd[50073]: setup_proc: scheduler done
> > Jun 29 22:24:01 mx smtpd[79591]: smtpd: setup done
> > Jun 29 22:24:01 mx filter-spamassassin[81659]: fatal: resolve: failed
> > Jun 29 22:24:01 mx smtpd[82409]: warn: pony -> filter-spamassassin: pipe
> closed
> > Jun 29 22:24:01 mx smtpd[82409]: warn: filter "filter-spamassassin"
> closed unexpectedly
> > Jun 29 22:24:01 mx smtpd[82409]: fatal: exiting
> > Jun 29 22:24:01 mx smtpd[79591]: warn: parent -> pony: pipe closed
> > Jun 29 22:24:01 mx smtpd[64252]: warn: ca -> pony express: pipe closed
> > Jun 29 22:24:01 mx smtpd[77736]: warn: control -> pony express: pipe
> closed
> > Jun 29 22:24:01 mx smtpd[50073]: warn: scheduler -> control: pipe closed
> > Jun 29 22:24:01 mx smtpd[65295]: warn: queue -> pony express: pipe closed
> > Jun 29 22:24:01 mx smtpd[45356]: warn: lka -> control: pipe closed
> >
> > It tries to start, but spamassassin crashes with "fatal: resolve:
> failed", bringing smtpd with it.
> >
> > Has anyone seen this before? Is this a bug in filter-spamassassin or a
> fault in my setup?
>
> This usually happens when SpamAssassin spamd is not running at the time
> filter-spam

Re: filter-spamassassin crashes smtpd on boot

2016-06-29 Thread Joerg Jung 


> Am 30.06.2016 um 00:41 schrieb Simon McFarlane <s...@desu.ne.jp>:
> 
> Hi,
> 
> I have a rather uninteresting setup consisting of Dovecot and OpenSMTPD with 
> few configuration tweaks on an OpenBSD system. The only filter I have is 
> filter-spamassassin. I'm running the latest -current snapshot and filter and 
> extras from pkg.
> 
> I can rcctl start smtpd just fine, however, smtpd fails to start 
> automatically on boot. Every time I reboot the system, I have to shell back 
> in and start smtpd manually. I get this in /var/log/maillog:
> 
> Jun 29 22:24:01 mx smtpd[76577]: info: OpenSMTPD 5.9.1 starting
> Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> control[77736] fd=4
> Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> klondike[64252] fd=5
> Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> control[77736] fd=8
> Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> lookup[45356] fd=6
> Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> control[77736] fd=6
> Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> pony express[82409] 
> fd=5
> Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> pony express[82409] 
> fd=7
> Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> pony express[82409] fd=9
> Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> queue[65295] fd=8
> Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> pony express[82409] 
> fd=7
> Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> control[77736] fd=9
> Jun 29 22:24:01 mx smtpd[64252]: setup_proc: klondike done
> Jun 29 22:24:01 mx smtpd[79591]: setup_done: ca[64252] done
> Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> lookup[45356] fd=10
> Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> scheduler[50073] fd=9
> Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> queue[65295] fd=8
> Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> control[77736] 
> fd=7
> Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> queue[65295] fd=10
> Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> scheduler[50073] fd=11
> Jun 29 22:24:01 mx smtpd[79591]: setup_done: control[77736] done
> Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> klondike[64252] 
> fd=8
> Jun 29 22:24:01 mx smtpd[77736]: setup_proc: control done
> Jun 29 22:24:01 mx smtpd[45356]: setup_proc: lookup done
> Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> lookup[45356] 
> fd=9
> Jun 29 22:24:01 mx smtpd[79591]: setup_done: lka[45356] done
> Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> queue[65295] 
> fd=10
> Jun 29 22:24:01 mx smtpd[79591]: setup_done: pony[82409] done
> Jun 29 22:24:01 mx smtpd[82409]: setup_proc: pony express done
> Jun 29 22:24:01 mx smtpd[65295]: setup_proc: queue done
> Jun 29 22:24:01 mx smtpd[79591]: setup_done: queue[65295] done
> Jun 29 22:24:01 mx smtpd[79591]: setup_done: scheduler[50073] done
> Jun 29 22:24:01 mx smtpd[50073]: setup_proc: scheduler done
> Jun 29 22:24:01 mx smtpd[79591]: smtpd: setup done
> Jun 29 22:24:01 mx filter-spamassassin[81659]: fatal: resolve: failed
> Jun 29 22:24:01 mx smtpd[82409]: warn: pony -> filter-spamassassin: pipe 
> closed
> Jun 29 22:24:01 mx smtpd[82409]: warn: filter "filter-spamassassin" closed 
> unexpectedly
> Jun 29 22:24:01 mx smtpd[82409]: fatal: exiting
> Jun 29 22:24:01 mx smtpd[79591]: warn: parent -> pony: pipe closed
> Jun 29 22:24:01 mx smtpd[64252]: warn: ca -> pony express: pipe closed
> Jun 29 22:24:01 mx smtpd[77736]: warn: control -> pony express: pipe closed
> Jun 29 22:24:01 mx smtpd[50073]: warn: scheduler -> control: pipe closed
> Jun 29 22:24:01 mx smtpd[65295]: warn: queue -> pony express: pipe closed
> Jun 29 22:24:01 mx smtpd[45356]: warn: lka -> control: pipe closed
> 
> It tries to start, but spamassassin crashes with "fatal: resolve: failed", 
> bringing smtpd with it.
> 
> Has anyone seen this before? Is this a bug in filter-spamassassin or a fault 
> in my setup?

This usually happens when SpamAssassin spamd is not running at the time
filter-spamassassin is started. This is considered a bug and a know problem, 
beeing worked on.

> Thanks,
> Simon
> 
> -- 
> You received this mail because you are subscribed to misc@opensmtpd.org
> To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
> 


--
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

filter-spamassassin crashes smtpd on boot

2016-06-29 Thread Simon McFarlane 

Hi,

I have a rather uninteresting setup consisting of Dovecot and OpenSMTPD 
with few configuration tweaks on an OpenBSD system. The only filter I 
have is filter-spamassassin. I'm running the latest -current snapshot 
and filter and extras from pkg.


I can rcctl start smtpd just fine, however, smtpd fails to start 
automatically on boot. Every time I reboot the system, I have to shell 
back in and start smtpd manually. I get this in /var/log/maillog:


Jun 29 22:24:01 mx smtpd[76577]: info: OpenSMTPD 5.9.1 starting
Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> control[77736] fd=4
Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> klondike[64252] fd=5
Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> control[77736] fd=8
Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> lookup[45356] fd=6
Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> control[77736] fd=6
Jun 29 22:24:01 mx smtpd[64252]: setup_peer: klondike -> pony 
express[82409] fd=5
Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> pony 
express[82409] fd=7
Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> pony 
express[82409] fd=9

Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> queue[65295] fd=8
Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> pony 
express[82409] fd=7
Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> control[77736] 
fd=9

Jun 29 22:24:01 mx smtpd[64252]: setup_proc: klondike done
Jun 29 22:24:01 mx smtpd[79591]: setup_done: ca[64252] done
Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> lookup[45356] fd=10
Jun 29 22:24:01 mx smtpd[77736]: setup_peer: control -> scheduler[50073] 
fd=9

Jun 29 22:24:01 mx smtpd[45356]: setup_peer: lookup -> queue[65295] fd=8
Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> 
control[77736] fd=7

Jun 29 22:24:01 mx smtpd[50073]: setup_peer: scheduler -> queue[65295] fd=10
Jun 29 22:24:01 mx smtpd[65295]: setup_peer: queue -> scheduler[50073] fd=11
Jun 29 22:24:01 mx smtpd[79591]: setup_done: control[77736] done
Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> 
klondike[64252] fd=8

Jun 29 22:24:01 mx smtpd[77736]: setup_proc: control done
Jun 29 22:24:01 mx smtpd[45356]: setup_proc: lookup done
Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> 
lookup[45356] fd=9

Jun 29 22:24:01 mx smtpd[79591]: setup_done: lka[45356] done
Jun 29 22:24:01 mx smtpd[82409]: setup_peer: pony express -> 
queue[65295] fd=10

Jun 29 22:24:01 mx smtpd[79591]: setup_done: pony[82409] done
Jun 29 22:24:01 mx smtpd[82409]: setup_proc: pony express done
Jun 29 22:24:01 mx smtpd[65295]: setup_proc: queue done
Jun 29 22:24:01 mx smtpd[79591]: setup_done: queue[65295] done
Jun 29 22:24:01 mx smtpd[79591]: setup_done: scheduler[50073] done
Jun 29 22:24:01 mx smtpd[50073]: setup_proc: scheduler done
Jun 29 22:24:01 mx smtpd[79591]: smtpd: setup done
Jun 29 22:24:01 mx filter-spamassassin[81659]: fatal: resolve: failed
Jun 29 22:24:01 mx smtpd[82409]: warn: pony -> filter-spamassassin: pipe 
closed
Jun 29 22:24:01 mx smtpd[82409]: warn: filter "filter-spamassassin" 
closed unexpectedly

Jun 29 22:24:01 mx smtpd[82409]: fatal: exiting
Jun 29 22:24:01 mx smtpd[79591]: warn: parent -> pony: pipe closed
Jun 29 22:24:01 mx smtpd[64252]: warn: ca -> pony express: pipe closed
Jun 29 22:24:01 mx smtpd[77736]: warn: control -> pony express: pipe closed
Jun 29 22:24:01 mx smtpd[50073]: warn: scheduler -> control: pipe closed
Jun 29 22:24:01 mx smtpd[65295]: warn: queue -> pony express: pipe closed
Jun 29 22:24:01 mx smtpd[45356]: warn: lka -> control: pipe closed

It tries to start, but spamassassin crashes with "fatal: resolve: 
failed", bringing smtpd with it.


Has anyone seen this before? Is this a bug in filter-spamassassin or a 
fault in my setup?


Thanks,
Simon

--
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: more detailed logging from filter-spamassassin

2016-06-16 Thread Joerg Jung 

> Am 15.06.2016 um 19:55 schrieb Andrew Ruscica <andrew.li...@ruscica.com>:
> 
> This behaviour has been the same since I started using filter-spamassassin 
> which was before the May 23 snapshot.  Currently on the June 6 snapshot of 
> opensmtpd & opensmtpd-extras.
> 
> Using filter-spamassassin like this
> 
> filter filter-spamassassin spamassassin "-s reject"
> When an email is rejected, the log entry looks like this:
> 
> Jun 15 13:28:04 mxgw3 smtpd[23583]: 94725446b2523387 smtp event=connected 
> address=60.167.113.80 host=60.167.113.80
> Jun 15 13:28:11 mxgw3 filter-spamassassin[10058]: warn: session 
> 94725446b2523387: on_eom: REJECT spam
> Jun 15 13:28:11 mxgw3 smtpd[23583]: 94725446b2523387 smtp 
> event=failed-command command=DATA result=554 5.7.1 Message considered spam
> Jun 15 13:28:11 mxgw3 smtpd[23583]: 94725446b2523387 smtp event=closed 
> reason=quit
> 
> 
> I also use an IP whitelist - actually the nospamd table used for spamd. PF 
> redirects whitelisted connections to a listener at opensmtpd which doesn't do 
> filter-spamassassin.
> 
> The problem is troubleshooting false positives so I can improve my whitelist. 
>  Unless I obtain a sample NDR from the sender, it's difficult to match up the 
> source MTA with a sending domain. 
> 
> The ideal solution is to display the from and to address in the rejection log 
> line just like clamsmtp does it:
> 
Makes sense to me and should be easy to implement.
> Jun 15 13:14:48 mxgw3 clamsmtpd: 102B1B: from=bou...@bizmailtoday.com, 
> to=u...@mydomain.com, status=VIRUS:Heuristics.Phishing.Email.SSL-Spoof
> 
> Incidentally, I've tried adding the -v option to the filter-spamassassin 
> directive at smtpd.conf.  This in fact breaks smtpd such that it accepts no 
> connections at all.  I believe I'll need to file a separate bug report for 
> that.
> 
Yes, please do.

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-15 Thread Edgar Pettijohn 


Sent from my iPhone

> On Jun 15, 2016, at 12:43 PM, Andrew Ruscica  wrote:
> 
> > On Jun 6, 2016, at 5:37 PM, Gilles Chehade  wrote:
> >...
> >
> > Please do test new -extras snapshot that were just published, rebuild
> > your filters and reinstall them as the fix lies in the api/ layer
> >
> > Your problem should disappear ;-)
> 
> June 6 snapshots for both opensmtpd and extras installed at my high volume
> inbound gateway.
> 
> So far so good.  For the last several weeks I've been collecting 5-minute 
> stats
> and restarting opensmtpd when processes > 400. That has been happening
> consistently every 2-3 hours (even after I moved antivirus from filter-clamav 
> to
> clamsmtp).  
> 
> So I should be able to determine pretty quickly that this issue was resolved.
> 
> Thanks again-
> 

Just curious if you are running filter-regex also. If not try it in front of 
spamassassin. It cuts through a lot of spam so I'm not wasting resources on a 
hungry spamassin.
>

more detailed logging from filter-spamassassin

2016-06-15 Thread Andrew Ruscica 
This behaviour has been the same since I started using filter-spamassassin
which was before the May 23 snapshot.  Currently on the June 6 snapshot of
opensmtpd & opensmtpd-extras.

Using filter-spamassassin like this

filter filter-spamassassin spamassassin "-s reject"

When an email is rejected, the log entry looks like this:

Jun 15 13:28:04 mxgw3 smtpd[23583]: 94725446b2523387 smtp event=connected
address=60.167.113.80 host=60.167.113.80
Jun 15 13:28:11 mxgw3 filter-spamassassin[10058]: warn: session
94725446b2523387: on_eom: REJECT spam
Jun 15 13:28:11 mxgw3 smtpd[23583]: 94725446b2523387 smtp
event=failed-command command=DATA result=554 5.7.1 Message considered spam
Jun 15 13:28:11 mxgw3 smtpd[23583]: 94725446b2523387 smtp event=closed
reason=quit


I also use an IP whitelist - actually the nospamd table used for spamd. PF
redirects whitelisted connections to a listener at opensmtpd which doesn't
do filter-spamassassin.

The problem is troubleshooting false positives so I can improve my
whitelist.  Unless I obtain a sample NDR from the sender, it's difficult to
match up the source MTA with a sending domain.

The ideal solution is to display the from and to address in the rejection
log line just like clamsmtp does it:

Jun 15 13:14:48 mxgw3 clamsmtpd: 102B1B: from=bou...@bizmailtoday.com, to=
u...@mydomain.com, status=VIRUS:Heuristics.Phishing.Email.SSL-Spoof

Incidentally, I've tried adding the -v option to the filter-spamassassin
directive at smtpd.conf.  This in fact breaks smtpd such that it accepts no
connections at all.  I believe I'll need to file a separate bug report for
that.

Thanks,
Andrew

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-15 Thread Andrew Ruscica 
> On Jun 6, 2016, at 5:37 PM, Gilles Chehade  wrote:
>...
>
> Please do test new -extras snapshot that were just published, rebuild
> your filters and reinstall them as the fix lies in the api/ layer
>
> Your problem should disappear ;-)

June 6 snapshots for both opensmtpd and extras installed at my high volume
inbound gateway.

So far so good.  For the last several weeks I've been collecting 5-minute
stats
and restarting opensmtpd when processes > 400. That has been happening
consistently every 2-3 hours (even after I moved antivirus from
filter-clamav to
clamsmtp).

So I should be able to determine pretty quickly that this issue was
resolved.

Thanks again-

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-07 Thread Andrew Ruscica 
Thank you so much!
It will be at least a few days before I can update. I will report on
the results as soon as I do.

> On Jun 6, 2016, at 5:37 PM, Gilles Chehade <gil...@poolp.org> wrote:
>
>> On Wed, May 25, 2016 at 04:28:59PM -0400, Andrew Ruscica wrote:
>> Not sure if the issue lies with the spamassassin daemon or the
>> opensmtp-filter.
>>
>> During a surge of incoming email, the following errors were logged.
>> OpenSMTP stopped relaying mail during this time:
>>
>> May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
>> shutdown: Bad file descriptor
>>
>> May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
>> 627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed
>>
>>
>> The immediate resolution was to temporarily disable the filter and restart
>> smtpd, although I think I probably could have resolved by restarting
>> spamassassin as well..
>>
>> I'm pre-forking 20 spamassassin children, with max of 40, and limiting
>> incoming connections to opensmtpd using pf max states rules.
>>
>> While this was occurring the spamd-child processes surged to about 30 and
>> were starting to be released and returning back to normal as I was forcing
>> down connections at pf.
>>
>> using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
>> from openbsd packages.
>>
>>
>> Please let me know what other system details may be relevant here...
>
>
> Please do test new -extras snapshot that were just published, rebuild
> your filters and reinstall them as the fix lies in the api/ layer
>
> Your problem should disappear ;-)
>
>
> --
> Gilles Chehade
>
> https://www.poolp.org  @poolpOrg

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-06 Thread Gilles Chehade 
On Wed, May 25, 2016 at 04:28:59PM -0400, Andrew Ruscica wrote:
> Not sure if the issue lies with the spamassassin daemon or the
> opensmtp-filter.
> 
> During a surge of incoming email, the following errors were logged.
> OpenSMTP stopped relaying mail during this time:
> 
> May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
> shutdown: Bad file descriptor
> 
> May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
> 627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed
> 
> 
> The immediate resolution was to temporarily disable the filter and restart
> smtpd, although I think I probably could have resolved by restarting
> spamassassin as well..
> 
> I'm pre-forking 20 spamassassin children, with max of 40, and limiting
> incoming connections to opensmtpd using pf max states rules.
> 
> While this was occurring the spamd-child processes surged to about 30 and
> were starting to be released and returning back to normal as I was forcing
> down connections at pf.
> 
> using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
> from openbsd packages.
> 
> 
> Please let me know what other system details may be relevant here...


Please do test new -extras snapshot that were just published, rebuild
your filters and reinstall them as the fix lies in the api/ layer

Your problem should disappear ;-)


-- 
Gilles Chehade

https://www.poolp.org  @poolpOrg

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-27 Thread Joerg Jung 
On Thu, May 26, 2016 at 08:14:12PM +0200, Joerg Jung wrote:
> On Thu, May 26, 2016 at 01:51:20PM -0400, Andrew Ruscica wrote:
> > On Thu, May 26, 2016 at 10:50 AM, Joerg Jung <m...@umaxx.net> wrote:
> > > Am 26.05.2016 um 14:11 schrieb Andrew Ruscica <andrew.li...@ruscica.com>:
> > > On Wed, May 25, 2016 at 4:39 PM, Joerg Jung <m...@umaxx.net> wrote:
> > >
> > >> Can you provide smtpctl show stats and fstat -u _smtpd output?
> > >>
> > >
> > > If the event happens again, I will provide output before restarting the
> > > daemon. Here it is now.. had to restart it a short while ago;
> > >
> > > # smtpctl show stats
> > > control.session=1
> > > mta.connector=1
> > > mta.domain=1
> > > mta.envelope=0
> > > mta.host=1
> > > mta.relay=1
> > > mta.route=1
> > > mta.session=1
> > > mta.source=1
> > > mta.task=0
> > > mta.task.running=0
> > > queue.evpcache.load.hit=680
> > > queue.evpcache.size=20
> > > scheduler.delivery.ok=340
> > > scheduler.envelope=0
> > > scheduler.envelope.incoming=20
> > > scheduler.envelope.inflight=0
> > > scheduler.ramqueue.envelope=20
> > > scheduler.ramqueue.message=9
> > > scheduler.ramqueue.update=9
> > > smtp.session=21
> > > smtp.session.inet4=278
> > >
> > >
> > > Out of 278 total sessions in 11min
> > > you have 21 current active ones, guess most
> > > of them are hanging...
> > >
> > > This looks like #698 to me.
> > >
> > > Have you applied the smtpd errata?
> > >
> > Hmm,
> > 
> > # smtpd -h
> > version: OpenSMTPD 5.9.2
> > also present before installing 5.9.2 from the tarball:
> > 
> > # pkg_info | grep smtpd
> > binpatch59-amd64-smtpd-1.0 Binary Patch for 006_smtpd
> > 
> > If I don't limit states at pf (currently at 30), my maximum connections
> > always reach the 500 (493) limit, and all relaying stalls.
> 
> I guess, you also have increased limits in login.conf?
> 
> > I know filter-spamassassin is expensive, and I've pre-forked (now 30)
> > children (currently the box has 8 cores and 8GB RAM, both apparently
> > underutilized), but the only way I can ensure the system doesn't get
> > bottlenecked is to throttle the connections at pf.
> 
> What do you see from spamassassin spamd in the logs (maybe enable debug),
> are the sessions finished by spamd? Have you tried the limit option of
> filter-spamassassin?
> 
> Your fstat output also shows connections to clamav, so maybe not
> spamassassin the culprit here?
>  
> > Also, limits are present:
> > 
> > limit session max-mails 40
> > limit scheduler max-inflight 30
> > but it doesn't appear to change the behaviour - the queue is never full,
> > just a ton of connections..
> 
> I assume hanging and dead, but hard to debug without logs, so:
> 
> Please add your exact versions as mentioned above, full config
> (including spamassassin spamd and clamd tweaks), logs from
> smtpd -dv -Tall and fstat as well as smtpctl show stats output to #698,
> thanks!

Maybe you have also some time on Monday to join the debugging session
Gilles mentioned in the other thread to Peter Fraser helping us to hunt
the bug down?

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Joerg Jung 
On Thu, May 26, 2016 at 01:51:20PM -0400, Andrew Ruscica wrote:
> On Thu, May 26, 2016 at 10:50 AM, Joerg Jung <m...@umaxx.net> wrote:
> 
> >
> >
> > Am 26.05.2016 um 14:11 schrieb Andrew Ruscica <andrew.li...@ruscica.com>:
> >
> > On Wed, May 25, 2016 at 4:39 PM, Joerg Jung <m...@umaxx.net> wrote:
> >
> >> Can you provide smtpctl show stats and fstat -u _smtpd output?
> >>
> >
> > If the event happens again, I will provide output before restarting the
> > daemon. Here it is now.. had to restart it a short while ago;
> >
> >
> > # smtpctl show stats
> > control.session=1
> > mta.connector=1
> > mta.domain=1
> > mta.envelope=0
> > mta.host=1
> > mta.relay=1
> > mta.route=1
> > mta.session=1
> > mta.source=1
> > mta.task=0
> > mta.task.running=0
> > queue.evpcache.load.hit=680
> > queue.evpcache.size=20
> > scheduler.delivery.ok=340
> > scheduler.envelope=0
> > scheduler.envelope.incoming=20
> > scheduler.envelope.inflight=0
> > scheduler.ramqueue.envelope=20
> > scheduler.ramqueue.message=9
> > scheduler.ramqueue.update=9
> > smtp.session=21
> > smtp.session.inet4=278
> >
> >
> > Out of 278 total sessions in 11min
> > you have 21 current active ones, guess most
> > of them are hanging...
> >
> > This looks like #698 to me.
> >
> > Have you applied the smtpd errata?
> >
> >
> Hmm,
> 
> # smtpd -h
> version: OpenSMTPD 5.9.2
> also present before installing 5.9.2 from the tarball:
> 
> # pkg_info | grep smtpd
> binpatch59-amd64-smtpd-1.0 Binary Patch for 006_smtpd
> 
> 
> If I don't limit states at pf (currently at 30), my maximum connections
> always reach the 500 (493) limit, and all relaying stalls.

I guess, you also have increased limits in login.conf?

> I know filter-spamassassin is expensive, and I've pre-forked (now 30)
> children (currently the box has 8 cores and 8GB RAM, both apparently
> underutilized), but the only way I can ensure the system doesn't get
> bottlenecked is to throttle the connections at pf.

What do you see from spamassassin spamd in the logs (maybe enable debug),
are the sessions finished by spamd? Have you tried the limit option of
filter-spamassassin?

Your fstat output also shows connections to clamav, so maybe not
spamassassin the culprit here?
 
> Also, limits are present:
> 
> limit session max-mails 40
> limit scheduler max-inflight 30
> but it doesn't appear to change the behaviour - the queue is never full,
> just a ton of connections..

I assume hanging and dead, but hard to debug without logs, so:

Please add your exact versions as mentioned above, full config
(including spamassassin spamd and clamd tweaks), logs from
smtpd -dv -Tall and fstat as well as smtpctl show stats output to #698,
thanks!

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Andrew Ruscica 
On Thu, May 26, 2016 at 10:50 AM, Joerg Jung <m...@umaxx.net> wrote:

>
>
> Am 26.05.2016 um 14:11 schrieb Andrew Ruscica <andrew.li...@ruscica.com>:
>
> On Wed, May 25, 2016 at 4:39 PM, Joerg Jung <m...@umaxx.net> wrote:
>
>> Can you provide smtpctl show stats and fstat -u _smtpd output?
>>
>
> If the event happens again, I will provide output before restarting the
> daemon. Here it is now.. had to restart it a short while ago;
>
>
> # smtpctl show stats
> control.session=1
> mta.connector=1
> mta.domain=1
> mta.envelope=0
> mta.host=1
> mta.relay=1
> mta.route=1
> mta.session=1
> mta.source=1
> mta.task=0
> mta.task.running=0
> queue.evpcache.load.hit=680
> queue.evpcache.size=20
> scheduler.delivery.ok=340
> scheduler.envelope=0
> scheduler.envelope.incoming=20
> scheduler.envelope.inflight=0
> scheduler.ramqueue.envelope=20
> scheduler.ramqueue.message=9
> scheduler.ramqueue.update=9
> smtp.session=21
> smtp.session.inet4=278
>
>
> Out of 278 total sessions in 11min
> you have 21 current active ones, guess most
> of them are hanging...
>
> This looks like #698 to me.
>
> Have you applied the smtpd errata?
>
>
Hmm,

# smtpd -h
version: OpenSMTPD 5.9.2
also present before installing 5.9.2 from the tarball:

# pkg_info | grep smtpd
binpatch59-amd64-smtpd-1.0 Binary Patch for 006_smtpd


If I don't limit states at pf (currently at 30), my maximum connections
always reach the 500 (493) limit, and all relaying stalls.
I know filter-spamassassin is expensive, and I've pre-forked (now 30)
children (currently the box has 8 cores and 8GB RAM, both apparently
underutilized), but the only way I can ensure the system doesn't get
bottlenecked is to throttle the connections at pf.


Also, limits are present:

limit session max-mails 40
limit scheduler max-inflight 30
but it doesn't appear to change the behaviour - the queue is never full,
just a ton of connections..

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Joerg Jung 


> Am 26.05.2016 um 14:11 schrieb Andrew Ruscica :
> 
>> On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:
>> Can you provide smtpctl show stats and fstat -u _smtpd output?
> 
> If the event happens again, I will provide output before restarting the 
> daemon. Here it is now.. had to restart it a short while ago;
> 
> 
> # smtpctl show stats
> control.session=1
> mta.connector=1
> mta.domain=1
> mta.envelope=0
> mta.host=1
> mta.relay=1
> mta.route=1
> mta.session=1
> mta.source=1
> mta.task=0
> mta.task.running=0
> queue.evpcache.load.hit=680
> queue.evpcache.size=20
> scheduler.delivery.ok=340
> scheduler.envelope=0
> scheduler.envelope.incoming=20
> scheduler.envelope.inflight=0
> scheduler.ramqueue.envelope=20
> scheduler.ramqueue.message=9
> scheduler.ramqueue.update=9
> smtp.session=21
> smtp.session.inet4=278

Out of 278 total sessions in 11min
you have 21 current active ones, guess most
of them are hanging...

This looks like #698 to me.

Have you applied the smtpd errata?

> smtp.tls=14
> uptime=684
> uptime.human=11m24s
> 
> 
> fstat output at the bottom of this email.
> 
>  
>> 
>> 
>> You can try the latest -extras snapshot and especially the limit option.
> 
> 
> I'm on the may 23 snapshot and will look at the limit option now.
> 
> Thanks,
> Andrew
> 
> 
> 
> 
> # fstat -u _smtpd
> 
> USER CMD  PID   FD MOUNTINUM MODE   R/WSZ|DV
> _smtpd   filter-spamassas 31978   wd /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-spamassas 31978 root /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-spamassas 319780* unix stream 0x81575200 <-> 
> 0x80a7ac80
> _smtpd   filter-spamassas 319781 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-spamassas 319782 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-spamassas 319783 kqueue 0xff0200e1c3c0 0 state:
> _smtpd   filter-spamassas 319784* internet stream tcp 0x80ceab40 
> 127.0.0.1:33104 --> 127.0.0.1:783
> _smtpd   filter-spamassas 319785* unix stream 0x814b7b00 <-> 
> 0x8149bc00
> _smtpd   filter-spamassas 319787* unix stream 0x80af6980 <-> 
> 0x80f6f800
> _smtpd   filter-spamassas 319788* unix stream 0x80bae000 <-> 
> 0x814fa000
> _smtpd   filter-spamassas 319789* unix stream 0x80ec7480 <-> 
> 0x814ae000
> _smtpd   filter-spamassas 31978   10* unix stream 0x80f92b80 <-> 
> 0x80ecd400
> _smtpd   filter-spamassas 31978   11* unix stream 0x80be8100 <-> 
> 0x80bfdf80
> _smtpd   filter-spamassas 31978   12* unix stream 0x8155ff80 <-> 
> 0x81461100
> _smtpd   filter-spamassas 31978   13* internet stream tcp 0x80fdedf0 
> 127.0.0.1:27127 --> 127.0.0.1:783
> _smtpd   filter-spamassas 31978   14* internet stream tcp 0x81605188 
> 127.0.0.1:25541 --> 127.0.0.1:783
> _smtpd   filter-spamassas 31978   15* unix stream 0x8140c180 <-> 
> 0x81489f80
> _smtpd   filter-spamassas 31978   16* internet stream tcp 0x80fde990 
> 127.0.0.1:39347 --> 127.0.0.1:783
> _smtpd   filter-spamassas 31978   18* unix stream 0x80aec500 <-> 
> 0x80a9e300
> _smtpd   filter-regex  3752   wd /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-regex  3752 root /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-regex  37520* unix stream 0x80ce2380 <-> 
> 0x80be9580
> _smtpd   filter-regex  37521 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-regex  37522 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-regex  37523 kqueue 0xff0200e1cb40 4 state:
> _smtpd   filter-regex  37526* unix stream 0x80bfdf80 <-> 
> 0x80be8100
> _smtpd   filter-regex  37527* unix stream 0x80c75d80 <-> 
> 0x80c1a380
> _smtpd   filter-regex  37528* unix stream 0x81461100 <-> 
> 0x8155ff80
> _smtpd   filter-regex  37529* unix stream 0x814c7e00 <-> 
> 0x8160fe00
> _smtpd   filter-regex  3752   10* unix stream 0x81489f80 <-> 
> 0x8140c180
> _smtpd   filter-regex  3752   11* unix stream 0x814cc500 <-> 
> 0x814ad200
> _smtpd   filter-regex  3752   12* unix stream 0x813c8700 <-> 
> 0x80f48c00
> _smtpd   filter-regex  3752   13* unix stream 0x814fa000 <-> 
> 0x80bae000
> _smtpd   filter-pause  3600   wd /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-pause  3600 root /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-pause  36000* unix stream 0x813c0d00 <-> 
> 0x80c75900
> _smtpd   filter-pause  36001 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-pause  36002 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-pause  36003 kqueue 0xff0200e1cc80 0 state: W
> _smtpd   filter-dnsbl 23238   wd /var

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Andrew Ruscica 
On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:

> Can you provide smtpctl show stats and fstat -u _smtpd output?
>

If the event happens again, I will provide output before restarting the
daemon. Here it is now.. had to restart it a short while ago;


# smtpctl show stats
control.session=1
mta.connector=1
mta.domain=1
mta.envelope=0
mta.host=1
mta.relay=1
mta.route=1
mta.session=1
mta.source=1
mta.task=0
mta.task.running=0
queue.evpcache.load.hit=680
queue.evpcache.size=20
scheduler.delivery.ok=340
scheduler.envelope=0
scheduler.envelope.incoming=20
scheduler.envelope.inflight=0
scheduler.ramqueue.envelope=20
scheduler.ramqueue.message=9
scheduler.ramqueue.update=9
smtp.session=21
smtp.session.inet4=278
smtp.tls=14
uptime=684
uptime.human=11m24s


fstat output at the bottom of this email.



>
>
> You can try the latest -extras snapshot and especially the limit option.
>
>

I'm on the may 23 snapshot and will look at the limit option now.

Thanks,
Andrew




# fstat -u _smtpd
USER CMD  PID   FD MOUNTINUM MODE   R/WSZ|DV
_smtpd   filter-spamassas 31978   wd /var  1429120 drwxr-xr-x   r
512
_smtpd   filter-spamassas 31978 root /var  1429120 drwxr-xr-x   r
512
_smtpd   filter-spamassas 319780* unix stream 0x81575200 <->
0x80a7ac80
_smtpd   filter-spamassas 319781 /   27142 crw-rw-rw-  rw
 null
_smtpd   filter-spamassas 319782 /   27142 crw-rw-rw-  rw
 null
_smtpd   filter-spamassas 319783 kqueue 0xff0200e1c3c0 0 state:
_smtpd   filter-spamassas 319784* internet stream tcp
0x80ceab40 127.0.0.1:33104 --> 127.0.0.1:783
_smtpd   filter-spamassas 319785* unix stream 0x814b7b00 <->
0x8149bc00
_smtpd   filter-spamassas 319787* unix stream 0x80af6980 <->
0x80f6f800
_smtpd   filter-spamassas 319788* unix stream 0x80bae000 <->
0x814fa000
_smtpd   filter-spamassas 319789* unix stream 0x80ec7480 <->
0x814ae000
_smtpd   filter-spamassas 31978   10* unix stream 0x80f92b80 <->
0x80ecd400
_smtpd   filter-spamassas 31978   11* unix stream 0x80be8100 <->
0x80bfdf80
_smtpd   filter-spamassas 31978   12* unix stream 0x8155ff80 <->
0x81461100
_smtpd   filter-spamassas 31978   13* internet stream tcp
0x80fdedf0 127.0.0.1:27127 --> 127.0.0.1:783
_smtpd   filter-spamassas 31978   14* internet stream tcp
0x81605188 127.0.0.1:25541 --> 127.0.0.1:783
_smtpd   filter-spamassas 31978   15* unix stream 0x8140c180 <->
0x81489f80
_smtpd   filter-spamassas 31978   16* internet stream tcp
0x80fde990 127.0.0.1:39347 --> 127.0.0.1:783
_smtpd   filter-spamassas 31978   18* unix stream 0x80aec500 <->
0x80a9e300
_smtpd   filter-regex  3752   wd /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-regex  3752 root /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-regex  37520* unix stream 0x80ce2380 <->
0x80be9580
_smtpd   filter-regex  37521 /   27142 crw-rw-rw-  rw null
_smtpd   filter-regex  37522 /   27142 crw-rw-rw-  rw null
_smtpd   filter-regex  37523 kqueue 0xff0200e1cb40 4 state:
_smtpd   filter-regex  37526* unix stream 0x80bfdf80 <->
0x80be8100
_smtpd   filter-regex  37527* unix stream 0x80c75d80 <->
0x80c1a380
_smtpd   filter-regex  37528* unix stream 0x81461100 <->
0x8155ff80
_smtpd   filter-regex  37529* unix stream 0x814c7e00 <->
0x8160fe00
_smtpd   filter-regex  3752   10* unix stream 0x81489f80 <->
0x8140c180
_smtpd   filter-regex  3752   11* unix stream 0x814cc500 <->
0x814ad200
_smtpd   filter-regex  3752   12* unix stream 0x813c8700 <->
0x80f48c00
_smtpd   filter-regex  3752   13* unix stream 0x814fa000 <->
0x80bae000
_smtpd   filter-pause  3600   wd /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-pause  3600 root /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-pause  36000* unix stream 0x813c0d00 <->
0x80c75900
_smtpd   filter-pause  36001 /   27142 crw-rw-rw-  rw null
_smtpd   filter-pause  36002 /   27142 crw-rw-rw-  rw null
_smtpd   filter-pause  36003 kqueue 0xff0200e1cc80 0 state: W
_smtpd   filter-dnsbl 23238   wd /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-dnsbl 23238 root /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-dnsbl 232380* unix stream 0x813c4000 <->
0x80f68e00
_smtpd   filter-dnsbl 232381 /   27142 crw-rw-rw-  rw null
_smtpd   filter-dnsbl 232382 /   27142 crw-rw-rw-  rw null
_smtpd   filter-dnsbl 232383 kqueue 0xff0200e1ca00 0 state: W
_smtpd   filter-dnsbl 232384* unix stream 0x80f31580 <->
0x8052e500
_smtpd

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-25 Thread Joerg Jung 
On Wed, May 25, 2016 at 04:28:59PM -0400, Andrew Ruscica wrote:
> Not sure if the issue lies with the spamassassin daemon or the
> opensmtp-filter.
> 
> During a surge of incoming email, the following errors were logged.
> OpenSMTP stopped relaying mail during this time:
> 
> May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
> shutdown: Bad file descriptor
> 
> May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
> 627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed
> 
> 
> The immediate resolution was to temporarily disable the filter and restart
> smtpd, although I think I probably could have resolved by restarting
> spamassassin as well..
> 
> I'm pre-forking 20 spamassassin children, with max of 40, and limiting
> incoming connections to opensmtpd using pf max states rules.
> 
> While this was occurring the spamd-child processes surged to about 30 and
> were starting to be released and returning back to normal as I was forcing
> down connections at pf.
> 
> using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
> from openbsd packages.
> 
> 
> Please let me know what other system details may be relevant here...

Can you provide smtpctl show stats and fstat -u _smtpd output?

You can try the latest -extras snapshot and especially the limit option.


-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-25 Thread Andrew Ruscica 
Not sure if the issue lies with the spamassassin daemon or the
opensmtp-filter.

During a surge of incoming email, the following errors were logged.
OpenSMTP stopped relaying mail during this time:

May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
shutdown: Bad file descriptor

May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed


The immediate resolution was to temporarily disable the filter and restart
smtpd, although I think I probably could have resolved by restarting
spamassassin as well..

I'm pre-forking 20 spamassassin children, with max of 40, and limiting
incoming connections to opensmtpd using pf max states rules.

While this was occurring the spamd-child processes surged to about 30 and
were starting to be released and returning back to normal as I was forcing
down connections at pf.

using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
from openbsd packages.


Please let me know what other system details may be relevant here...

Re: filter-spamassassin fails to deliver with large messages

2016-04-05 Thread LÉVAI Dániel 
Joerg Jung @ 2016-04-05T12:58:23 +0200:
[...]
> > Indeed, this was because of the --timeout-child option to spamd(1).
> > However, raising this limit just because it might cover every message
> > size doesn't really make sense. Even with 600 secs, there will be a
> > message large enough for smtpd to spit back (given, depending on eg. the
> > connection speed -- I could easily scan a ~10MB mail submitting from
> > localhost, but not from a remote site eg. gmail). 
> 
> A 10 MB mail should be no problem within the default 300s. Even with a slow 
> link.
> I have remote sites sending me 20-30MB mails taking about 1-3s to scan.
> 
> As I wrote earlier, you may want to double-check your spamassassin config to 
> figure out why it takes so long to check.

I don't think I should have to bend over spamassassin to an smtpd filter
that fails to implement a basic check for the message size. With eg.
filter-clamav, it is a bit easier, because you define the size limit for
clamd. But with the the spamd/spamc duo, you'd do that with spamc (and
in this case, filter-spamassassin is the 'client').

> 
> > So the only option I have is to set the global message size in
> > smtpd.conf, and hope for the best?
> 
> What’s wrong with the message size setting in smtpd.conf?

It's not the solution to the problem. I *want to* receive "large"
messages. I just don't want them to be *scanned* by spamassassin. See
the difference?

> > Am I correct in that a client doesn't
> > necesarrily have to specify the message size at the beginning of a
> > transaction?
> 
> Yes. But it depends, ESMTP supports the SIZE command. 
> There are also reasonable timeout values specified in RFC for SMTP 
> transactions.
> 
> > What if smtpd doesn't know the message size beforehand —
> 
> No “what if” here. It may not know.

Okay, understood, so the (my) use-case still stands.

> > then it'll just 4xx messages seemingly random times based on how fast
> > the connection can push lines into it?
> 
> Yes, which is fine. 4xx means: try again later (maybe your line is fast 
> enough then). 

But it really is not (in my case) :) See, my connection to a remote site
doesn't get upgraded after a 4xx respone... But I see your point. If
*my* link was saturated, there'll be more chance for delivery next time.

> > Doesn't seem this spamassassin filter was thought through properly :-/
> > (seeing this option present in spamc (--max-size))
> 
> As the author of the filter, I feel a bit offended by this statement now.
> The filter works fine for me in larger production environments.  It does more 
> or less exactly the same as the similar sendmail milters, which also work
> fine for many people in production since years.
> 
> I’m trying to help you with some vague issue here in this thread:
> For some reason you want a max-size option for the filter, but fail to 
> explain why. 

Okay, okay. Sorry about that. Naturally I saw your name in the source;
didn't want to offend you of course (but I tend to have a big mouth).

This is simply my use-case. I can understand why no one else sees this
as a problem; I'm not saying that the filter is useless.

> > Anyway, I just injected a hacky patch in spamassassin_on_dataline() with
> > a basic test of a `static size_t' that records the lines' length…
> 
> So now you drop the message earlier, once the size limit is reached instead
> of running into the timeout later. I fail to see how this helps you? You get 
> less
> mail and even more 4xx replies this way.

Nope. Why would I drop the message? Of course that wouldn't make sense.
I just push the first n bytes to spamd(1), then when it reaches the
configured limit, it just bypasses spamd(1) for the rest of the lines.

So basically, I get only the first n bytes scanned by spamd(1) and not
the rest.
See, this is arguably not a good solution for a lot of people, but for
my use-case, this is perfect. I'm willing to sacrifice getting large
spam mails because of this. I won't try to bore you with statistics from
google about how big a spam message is, in average, but they rarely hit
1+ megabytes.

> > Equally hacky as the filter itself... :-P
> 
> No need to become annoying.  I have not seen any code or diff from you yet.

All right... I see we got off on the wrong foot here, and it is my
fault. So, I'm saying sorry, and how about just start off with a clean
slate.

I won't send my diff to the list, because as I've said, it's a
hackity-hack, and don't want someone to find it in the archives, and god
forbid use it.

If you're interested, or want to get an idea of what I was explaining,
I'll gladly send it to you in private.


Daniel

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails to deliver with large messages

2016-04-05 Thread Joerg Jung 

> On 04 Apr 2016, at 23:27, LÉVAI Dániel <l...@ecentrum.hu> wrote:
> 
>>> filter-spamassassin doesn't have a parameter to specify a maximum
>>> message size like spamc(1), and when it encounters a message weighing in at 
>>> a
>>> few MBs, it fails, and drops back the message with a 4xx.
>>> 
>>> Is this like a caveat, a known limitation or something?
>> 
>> Not really, the filter just pipes everything through towards spamd(1).
>> The message size can be adjusted in smtpd.conf.
>> 
>>> smtpd:
>>> Apr  4 10:40:48 malcolm smtpd[12759]: smtp-in: New session f49a69df17bc3046 
>>> from host hostname [ip_address]
>>> Apr  4 10:40:49 malcolm smtpd[12759]: smtp-in: Started TLS on session 
>>> f49a69df17bc3046: version=TLSv1.2, cipher=ECDHE-RSA-CHACHA20-POLY1305, 
>>> bits=256
>>> Apr 04 10:40:49 malcolm spamd[540]: spamd: connection from localhost 
>>> [127.0.0.1]:46081 to port 783, fd 6
>>> Apr 04 10:40:59 malcolm spamd[540]: spamd: timeout: (10 second timeout 
>>> while trying to PROCESS)
>> 
>> Your message does not really fail because of size, spamd run into
>> timeout while processing.  This can have various reasons depending on
>> your spamassassin config, e.g. additional dnsbl/pyzor/razor/whatever
>> checks might take a while.  
>> 
>> But yes, this might also be related to size, longer mails take longer to
>> transfer and to check and will run more likely into a timeout.
>> 
>> You seemed to have tweaked the timeouts: the default is 300s and makes
>> sense --  not the 10s mentioned in your log above. 
>> So You get what you requested ;)
> 
> Indeed, this was because of the --timeout-child option to spamd(1).
> However, raising this limit just because it might cover every message
> size doesn't really make sense. Even with 600 secs, there will be a
> message large enough for smtpd to spit back (given, depending on eg. the
> connection speed -- I could easily scan a ~10MB mail submitting from
> localhost, but not from a remote site eg. gmail). 

A 10 MB mail should be no problem within the default 300s. Even with a slow 
link.
I have remote sites sending me 20-30MB mails taking about 1-3s to scan.

As I wrote earlier, you may want to double-check your spamassassin config to 
figure out why it takes so long to check.

> So the only option I have is to set the global message size in
> smtpd.conf, and hope for the best?

What’s wrong with the message size setting in smtpd.conf?

> Am I correct in that a client doesn't
> necesarrily have to specify the message size at the beginning of a
> transaction?

Yes. But it depends, ESMTP supports the SIZE command. 
There are also reasonable timeout values specified in RFC for SMTP transactions.

> What if smtpd doesn't know the message size beforehand —

No “what if” here. It may not know.

> then it'll just 4xx messages seemingly random times based on how fast
> the connection can push lines into it?

Yes, which is fine. 4xx means: try again later (maybe your line is fast enough 
then). 

> Doesn't seem this spamassassin filter was thought through properly :-/
> (seeing this option present in spamc (--max-size))

As the author of the filter, I feel a bit offended by this statement now.
The filter works fine for me in larger production environments.  It does more 
or less exactly the same as the similar sendmail milters, which also work
fine for many people in production since years.

I’m trying to help you with some vague issue here in this thread:
For some reason you want a max-size option for the filter, but fail to explain 
why. 

> Anyway, I just injected a hacky patch in spamassassin_on_dataline() with
> a basic test of a `static size_t' that records the lines' length…

So now you drop the message earlier, once the size limit is reached instead
of running into the timeout later. I fail to see how this helps you? You get 
less
mail and even more 4xx replies this way.

> Equally hacky as the filter itself... :-P

No need to become annoying.  I have not seen any code or diff from you yet.


--
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

filter-spamassassin

2016-01-03 Thread Edgar Pettijohn 
I finally got around to trying out filter-spamassassin.  It appears to 
work correctly.  However, I was trying to change the default action from 
accept to reject. I've tried everything I can think of but continue to 
get syntax errors.  I assumed something like this would work:


filter spam spamassassin -s reject

listen on egress filter spam

But it didn't.

Any help is appreciated.

Edgar Pettijohn

--
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin

2016-01-03 Thread Joerg Jung 
On Sun, Jan 03, 2016 at 08:56:00AM -0600, Edgar Pettijohn wrote:
> I finally got around to trying out filter-spamassassin.  It appears to work
> correctly.  However, I was trying to change the default action from accept
> to reject. I've tried everything I can think of but continue to get syntax
> errors.  I assumed something like this would work:
> 
> filter spam spamassassin -s reject

You need to quote the args: filter spam spamassassin "-s reject"
 
> listen on egress filter spam
> 
> But it didn't.
> 
> Any help is appreciated.
> 
> Edgar Pettijohn
> 
> -- 
> You received this mail because you are subscribed to misc@opensmtpd.org
> To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org
> 

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org