Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-15 Thread Edgar Pettijohn 


Sent from my iPhone

> On Jun 15, 2016, at 12:43 PM, Andrew Ruscica  wrote:
> 
> > On Jun 6, 2016, at 5:37 PM, Gilles Chehade  wrote:
> >...
> >
> > Please do test new -extras snapshot that were just published, rebuild
> > your filters and reinstall them as the fix lies in the api/ layer
> >
> > Your problem should disappear ;-)
> 
> June 6 snapshots for both opensmtpd and extras installed at my high volume
> inbound gateway.
> 
> So far so good.  For the last several weeks I've been collecting 5-minute 
> stats
> and restarting opensmtpd when processes > 400. That has been happening
> consistently every 2-3 hours (even after I moved antivirus from filter-clamav 
> to
> clamsmtp).  
> 
> So I should be able to determine pretty quickly that this issue was resolved.
> 
> Thanks again-
> 

Just curious if you are running filter-regex also. If not try it in front of 
spamassassin. It cuts through a lot of spam so I'm not wasting resources on a 
hungry spamassin.
>

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-15 Thread Andrew Ruscica 
> On Jun 6, 2016, at 5:37 PM, Gilles Chehade  wrote:
>...
>
> Please do test new -extras snapshot that were just published, rebuild
> your filters and reinstall them as the fix lies in the api/ layer
>
> Your problem should disappear ;-)

June 6 snapshots for both opensmtpd and extras installed at my high volume
inbound gateway.

So far so good.  For the last several weeks I've been collecting 5-minute
stats
and restarting opensmtpd when processes > 400. That has been happening
consistently every 2-3 hours (even after I moved antivirus from
filter-clamav to
clamsmtp).

So I should be able to determine pretty quickly that this issue was
resolved.

Thanks again-

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-07 Thread Andrew Ruscica 
Thank you so much!
It will be at least a few days before I can update. I will report on
the results as soon as I do.

> On Jun 6, 2016, at 5:37 PM, Gilles Chehade  wrote:
>
>> On Wed, May 25, 2016 at 04:28:59PM -0400, Andrew Ruscica wrote:
>> Not sure if the issue lies with the spamassassin daemon or the
>> opensmtp-filter.
>>
>> During a surge of incoming email, the following errors were logged.
>> OpenSMTP stopped relaying mail during this time:
>>
>> May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
>> shutdown: Bad file descriptor
>>
>> May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
>> 627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed
>>
>>
>> The immediate resolution was to temporarily disable the filter and restart
>> smtpd, although I think I probably could have resolved by restarting
>> spamassassin as well..
>>
>> I'm pre-forking 20 spamassassin children, with max of 40, and limiting
>> incoming connections to opensmtpd using pf max states rules.
>>
>> While this was occurring the spamd-child processes surged to about 30 and
>> were starting to be released and returning back to normal as I was forcing
>> down connections at pf.
>>
>> using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
>> from openbsd packages.
>>
>>
>> Please let me know what other system details may be relevant here...
>
>
> Please do test new -extras snapshot that were just published, rebuild
> your filters and reinstall them as the fix lies in the api/ layer
>
> Your problem should disappear ;-)
>
>
> --
> Gilles Chehade
>
> https://www.poolp.org  @poolpOrg

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-06-06 Thread Gilles Chehade 
On Wed, May 25, 2016 at 04:28:59PM -0400, Andrew Ruscica wrote:
> Not sure if the issue lies with the spamassassin daemon or the
> opensmtp-filter.
> 
> During a surge of incoming email, the following errors were logged.
> OpenSMTP stopped relaying mail during this time:
> 
> May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
> shutdown: Bad file descriptor
> 
> May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
> 627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed
> 
> 
> The immediate resolution was to temporarily disable the filter and restart
> smtpd, although I think I probably could have resolved by restarting
> spamassassin as well..
> 
> I'm pre-forking 20 spamassassin children, with max of 40, and limiting
> incoming connections to opensmtpd using pf max states rules.
> 
> While this was occurring the spamd-child processes surged to about 30 and
> were starting to be released and returning back to normal as I was forcing
> down connections at pf.
> 
> using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
> from openbsd packages.
> 
> 
> Please let me know what other system details may be relevant here...


Please do test new -extras snapshot that were just published, rebuild
your filters and reinstall them as the fix lies in the api/ layer

Your problem should disappear ;-)


-- 
Gilles Chehade

https://www.poolp.org  @poolpOrg

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-27 Thread Joerg Jung 
On Thu, May 26, 2016 at 08:14:12PM +0200, Joerg Jung wrote:
> On Thu, May 26, 2016 at 01:51:20PM -0400, Andrew Ruscica wrote:
> > On Thu, May 26, 2016 at 10:50 AM, Joerg Jung  wrote:
> > > Am 26.05.2016 um 14:11 schrieb Andrew Ruscica :
> > > On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:
> > >
> > >> Can you provide smtpctl show stats and fstat -u _smtpd output?
> > >>
> > >
> > > If the event happens again, I will provide output before restarting the
> > > daemon. Here it is now.. had to restart it a short while ago;
> > >
> > > # smtpctl show stats
> > > control.session=1
> > > mta.connector=1
> > > mta.domain=1
> > > mta.envelope=0
> > > mta.host=1
> > > mta.relay=1
> > > mta.route=1
> > > mta.session=1
> > > mta.source=1
> > > mta.task=0
> > > mta.task.running=0
> > > queue.evpcache.load.hit=680
> > > queue.evpcache.size=20
> > > scheduler.delivery.ok=340
> > > scheduler.envelope=0
> > > scheduler.envelope.incoming=20
> > > scheduler.envelope.inflight=0
> > > scheduler.ramqueue.envelope=20
> > > scheduler.ramqueue.message=9
> > > scheduler.ramqueue.update=9
> > > smtp.session=21
> > > smtp.session.inet4=278
> > >
> > >
> > > Out of 278 total sessions in 11min
> > > you have 21 current active ones, guess most
> > > of them are hanging...
> > >
> > > This looks like #698 to me.
> > >
> > > Have you applied the smtpd errata?
> > >
> > Hmm,
> > 
> > # smtpd -h
> > version: OpenSMTPD 5.9.2
> > also present before installing 5.9.2 from the tarball:
> > 
> > # pkg_info | grep smtpd
> > binpatch59-amd64-smtpd-1.0 Binary Patch for 006_smtpd
> > 
> > If I don't limit states at pf (currently at 30), my maximum connections
> > always reach the 500 (493) limit, and all relaying stalls.
> 
> I guess, you also have increased limits in login.conf?
> 
> > I know filter-spamassassin is expensive, and I've pre-forked (now 30)
> > children (currently the box has 8 cores and 8GB RAM, both apparently
> > underutilized), but the only way I can ensure the system doesn't get
> > bottlenecked is to throttle the connections at pf.
> 
> What do you see from spamassassin spamd in the logs (maybe enable debug),
> are the sessions finished by spamd? Have you tried the limit option of
> filter-spamassassin?
> 
> Your fstat output also shows connections to clamav, so maybe not
> spamassassin the culprit here?
>  
> > Also, limits are present:
> > 
> > limit session max-mails 40
> > limit scheduler max-inflight 30
> > but it doesn't appear to change the behaviour - the queue is never full,
> > just a ton of connections..
> 
> I assume hanging and dead, but hard to debug without logs, so:
> 
> Please add your exact versions as mentioned above, full config
> (including spamassassin spamd and clamd tweaks), logs from
> smtpd -dv -Tall and fstat as well as smtpctl show stats output to #698,
> thanks!

Maybe you have also some time on Monday to join the debugging session
Gilles mentioned in the other thread to Peter Fraser helping us to hunt
the bug down?

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Joerg Jung 
On Thu, May 26, 2016 at 01:51:20PM -0400, Andrew Ruscica wrote:
> On Thu, May 26, 2016 at 10:50 AM, Joerg Jung  wrote:
> 
> >
> >
> > Am 26.05.2016 um 14:11 schrieb Andrew Ruscica :
> >
> > On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:
> >
> >> Can you provide smtpctl show stats and fstat -u _smtpd output?
> >>
> >
> > If the event happens again, I will provide output before restarting the
> > daemon. Here it is now.. had to restart it a short while ago;
> >
> >
> > # smtpctl show stats
> > control.session=1
> > mta.connector=1
> > mta.domain=1
> > mta.envelope=0
> > mta.host=1
> > mta.relay=1
> > mta.route=1
> > mta.session=1
> > mta.source=1
> > mta.task=0
> > mta.task.running=0
> > queue.evpcache.load.hit=680
> > queue.evpcache.size=20
> > scheduler.delivery.ok=340
> > scheduler.envelope=0
> > scheduler.envelope.incoming=20
> > scheduler.envelope.inflight=0
> > scheduler.ramqueue.envelope=20
> > scheduler.ramqueue.message=9
> > scheduler.ramqueue.update=9
> > smtp.session=21
> > smtp.session.inet4=278
> >
> >
> > Out of 278 total sessions in 11min
> > you have 21 current active ones, guess most
> > of them are hanging...
> >
> > This looks like #698 to me.
> >
> > Have you applied the smtpd errata?
> >
> >
> Hmm,
> 
> # smtpd -h
> version: OpenSMTPD 5.9.2
> also present before installing 5.9.2 from the tarball:
> 
> # pkg_info | grep smtpd
> binpatch59-amd64-smtpd-1.0 Binary Patch for 006_smtpd
> 
> 
> If I don't limit states at pf (currently at 30), my maximum connections
> always reach the 500 (493) limit, and all relaying stalls.

I guess, you also have increased limits in login.conf?

> I know filter-spamassassin is expensive, and I've pre-forked (now 30)
> children (currently the box has 8 cores and 8GB RAM, both apparently
> underutilized), but the only way I can ensure the system doesn't get
> bottlenecked is to throttle the connections at pf.

What do you see from spamassassin spamd in the logs (maybe enable debug),
are the sessions finished by spamd? Have you tried the limit option of
filter-spamassassin?

Your fstat output also shows connections to clamav, so maybe not
spamassassin the culprit here?
 
> Also, limits are present:
> 
> limit session max-mails 40
> limit scheduler max-inflight 30
> but it doesn't appear to change the behaviour - the queue is never full,
> just a ton of connections..

I assume hanging and dead, but hard to debug without logs, so:

Please add your exact versions as mentioned above, full config
(including spamassassin spamd and clamd tweaks), logs from
smtpd -dv -Tall and fstat as well as smtpctl show stats output to #698,
thanks!

-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Andrew Ruscica 
On Thu, May 26, 2016 at 10:50 AM, Joerg Jung  wrote:

>
>
> Am 26.05.2016 um 14:11 schrieb Andrew Ruscica :
>
> On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:
>
>> Can you provide smtpctl show stats and fstat -u _smtpd output?
>>
>
> If the event happens again, I will provide output before restarting the
> daemon. Here it is now.. had to restart it a short while ago;
>
>
> # smtpctl show stats
> control.session=1
> mta.connector=1
> mta.domain=1
> mta.envelope=0
> mta.host=1
> mta.relay=1
> mta.route=1
> mta.session=1
> mta.source=1
> mta.task=0
> mta.task.running=0
> queue.evpcache.load.hit=680
> queue.evpcache.size=20
> scheduler.delivery.ok=340
> scheduler.envelope=0
> scheduler.envelope.incoming=20
> scheduler.envelope.inflight=0
> scheduler.ramqueue.envelope=20
> scheduler.ramqueue.message=9
> scheduler.ramqueue.update=9
> smtp.session=21
> smtp.session.inet4=278
>
>
> Out of 278 total sessions in 11min
> you have 21 current active ones, guess most
> of them are hanging...
>
> This looks like #698 to me.
>
> Have you applied the smtpd errata?
>
>
Hmm,

# smtpd -h
version: OpenSMTPD 5.9.2
also present before installing 5.9.2 from the tarball:

# pkg_info | grep smtpd
binpatch59-amd64-smtpd-1.0 Binary Patch for 006_smtpd


If I don't limit states at pf (currently at 30), my maximum connections
always reach the 500 (493) limit, and all relaying stalls.
I know filter-spamassassin is expensive, and I've pre-forked (now 30)
children (currently the box has 8 cores and 8GB RAM, both apparently
underutilized), but the only way I can ensure the system doesn't get
bottlenecked is to throttle the connections at pf.


Also, limits are present:

limit session max-mails 40
limit scheduler max-inflight 30
but it doesn't appear to change the behaviour - the queue is never full,
just a ton of connections..

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Joerg Jung 


> Am 26.05.2016 um 14:11 schrieb Andrew Ruscica :
> 
>> On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:
>> Can you provide smtpctl show stats and fstat -u _smtpd output?
> 
> If the event happens again, I will provide output before restarting the 
> daemon. Here it is now.. had to restart it a short while ago;
> 
> 
> # smtpctl show stats
> control.session=1
> mta.connector=1
> mta.domain=1
> mta.envelope=0
> mta.host=1
> mta.relay=1
> mta.route=1
> mta.session=1
> mta.source=1
> mta.task=0
> mta.task.running=0
> queue.evpcache.load.hit=680
> queue.evpcache.size=20
> scheduler.delivery.ok=340
> scheduler.envelope=0
> scheduler.envelope.incoming=20
> scheduler.envelope.inflight=0
> scheduler.ramqueue.envelope=20
> scheduler.ramqueue.message=9
> scheduler.ramqueue.update=9
> smtp.session=21
> smtp.session.inet4=278

Out of 278 total sessions in 11min
you have 21 current active ones, guess most
of them are hanging...

This looks like #698 to me.

Have you applied the smtpd errata?

> smtp.tls=14
> uptime=684
> uptime.human=11m24s
> 
> 
> fstat output at the bottom of this email.
> 
>  
>> 
>> 
>> You can try the latest -extras snapshot and especially the limit option.
> 
> 
> I'm on the may 23 snapshot and will look at the limit option now.
> 
> Thanks,
> Andrew
> 
> 
> 
> 
> # fstat -u _smtpd
> 
> USER CMD  PID   FD MOUNTINUM MODE   R/WSZ|DV
> _smtpd   filter-spamassas 31978   wd /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-spamassas 31978 root /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-spamassas 319780* unix stream 0x81575200 <-> 
> 0x80a7ac80
> _smtpd   filter-spamassas 319781 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-spamassas 319782 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-spamassas 319783 kqueue 0xff0200e1c3c0 0 state:
> _smtpd   filter-spamassas 319784* internet stream tcp 0x80ceab40 
> 127.0.0.1:33104 --> 127.0.0.1:783
> _smtpd   filter-spamassas 319785* unix stream 0x814b7b00 <-> 
> 0x8149bc00
> _smtpd   filter-spamassas 319787* unix stream 0x80af6980 <-> 
> 0x80f6f800
> _smtpd   filter-spamassas 319788* unix stream 0x80bae000 <-> 
> 0x814fa000
> _smtpd   filter-spamassas 319789* unix stream 0x80ec7480 <-> 
> 0x814ae000
> _smtpd   filter-spamassas 31978   10* unix stream 0x80f92b80 <-> 
> 0x80ecd400
> _smtpd   filter-spamassas 31978   11* unix stream 0x80be8100 <-> 
> 0x80bfdf80
> _smtpd   filter-spamassas 31978   12* unix stream 0x8155ff80 <-> 
> 0x81461100
> _smtpd   filter-spamassas 31978   13* internet stream tcp 0x80fdedf0 
> 127.0.0.1:27127 --> 127.0.0.1:783
> _smtpd   filter-spamassas 31978   14* internet stream tcp 0x81605188 
> 127.0.0.1:25541 --> 127.0.0.1:783
> _smtpd   filter-spamassas 31978   15* unix stream 0x8140c180 <-> 
> 0x81489f80
> _smtpd   filter-spamassas 31978   16* internet stream tcp 0x80fde990 
> 127.0.0.1:39347 --> 127.0.0.1:783
> _smtpd   filter-spamassas 31978   18* unix stream 0x80aec500 <-> 
> 0x80a9e300
> _smtpd   filter-regex  3752   wd /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-regex  3752 root /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-regex  37520* unix stream 0x80ce2380 <-> 
> 0x80be9580
> _smtpd   filter-regex  37521 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-regex  37522 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-regex  37523 kqueue 0xff0200e1cb40 4 state:
> _smtpd   filter-regex  37526* unix stream 0x80bfdf80 <-> 
> 0x80be8100
> _smtpd   filter-regex  37527* unix stream 0x80c75d80 <-> 
> 0x80c1a380
> _smtpd   filter-regex  37528* unix stream 0x81461100 <-> 
> 0x8155ff80
> _smtpd   filter-regex  37529* unix stream 0x814c7e00 <-> 
> 0x8160fe00
> _smtpd   filter-regex  3752   10* unix stream 0x81489f80 <-> 
> 0x8140c180
> _smtpd   filter-regex  3752   11* unix stream 0x814cc500 <-> 
> 0x814ad200
> _smtpd   filter-regex  3752   12* unix stream 0x813c8700 <-> 
> 0x80f48c00
> _smtpd   filter-regex  3752   13* unix stream 0x814fa000 <-> 
> 0x80bae000
> _smtpd   filter-pause  3600   wd /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-pause  3600 root /var  1429120 drwxr-xr-x   r  512
> _smtpd   filter-pause  36000* unix stream 0x813c0d00 <-> 
> 0x80c75900
> _smtpd   filter-pause  36001 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-pause  36002 /   27142 crw-rw-rw-  rw null
> _smtpd   filter-pause  36003 kqueue 0xff0200e1cc80 0 state: W
> _smtpd   filter-dnsbl 23238   wd /var

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-26 Thread Andrew Ruscica 
On Wed, May 25, 2016 at 4:39 PM, Joerg Jung  wrote:

> Can you provide smtpctl show stats and fstat -u _smtpd output?
>

If the event happens again, I will provide output before restarting the
daemon. Here it is now.. had to restart it a short while ago;


# smtpctl show stats
control.session=1
mta.connector=1
mta.domain=1
mta.envelope=0
mta.host=1
mta.relay=1
mta.route=1
mta.session=1
mta.source=1
mta.task=0
mta.task.running=0
queue.evpcache.load.hit=680
queue.evpcache.size=20
scheduler.delivery.ok=340
scheduler.envelope=0
scheduler.envelope.incoming=20
scheduler.envelope.inflight=0
scheduler.ramqueue.envelope=20
scheduler.ramqueue.message=9
scheduler.ramqueue.update=9
smtp.session=21
smtp.session.inet4=278
smtp.tls=14
uptime=684
uptime.human=11m24s


fstat output at the bottom of this email.



>
>
> You can try the latest -extras snapshot and especially the limit option.
>
>

I'm on the may 23 snapshot and will look at the limit option now.

Thanks,
Andrew




# fstat -u _smtpd
USER CMD  PID   FD MOUNTINUM MODE   R/WSZ|DV
_smtpd   filter-spamassas 31978   wd /var  1429120 drwxr-xr-x   r
512
_smtpd   filter-spamassas 31978 root /var  1429120 drwxr-xr-x   r
512
_smtpd   filter-spamassas 319780* unix stream 0x81575200 <->
0x80a7ac80
_smtpd   filter-spamassas 319781 /   27142 crw-rw-rw-  rw
 null
_smtpd   filter-spamassas 319782 /   27142 crw-rw-rw-  rw
 null
_smtpd   filter-spamassas 319783 kqueue 0xff0200e1c3c0 0 state:
_smtpd   filter-spamassas 319784* internet stream tcp
0x80ceab40 127.0.0.1:33104 --> 127.0.0.1:783
_smtpd   filter-spamassas 319785* unix stream 0x814b7b00 <->
0x8149bc00
_smtpd   filter-spamassas 319787* unix stream 0x80af6980 <->
0x80f6f800
_smtpd   filter-spamassas 319788* unix stream 0x80bae000 <->
0x814fa000
_smtpd   filter-spamassas 319789* unix stream 0x80ec7480 <->
0x814ae000
_smtpd   filter-spamassas 31978   10* unix stream 0x80f92b80 <->
0x80ecd400
_smtpd   filter-spamassas 31978   11* unix stream 0x80be8100 <->
0x80bfdf80
_smtpd   filter-spamassas 31978   12* unix stream 0x8155ff80 <->
0x81461100
_smtpd   filter-spamassas 31978   13* internet stream tcp
0x80fdedf0 127.0.0.1:27127 --> 127.0.0.1:783
_smtpd   filter-spamassas 31978   14* internet stream tcp
0x81605188 127.0.0.1:25541 --> 127.0.0.1:783
_smtpd   filter-spamassas 31978   15* unix stream 0x8140c180 <->
0x81489f80
_smtpd   filter-spamassas 31978   16* internet stream tcp
0x80fde990 127.0.0.1:39347 --> 127.0.0.1:783
_smtpd   filter-spamassas 31978   18* unix stream 0x80aec500 <->
0x80a9e300
_smtpd   filter-regex  3752   wd /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-regex  3752 root /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-regex  37520* unix stream 0x80ce2380 <->
0x80be9580
_smtpd   filter-regex  37521 /   27142 crw-rw-rw-  rw null
_smtpd   filter-regex  37522 /   27142 crw-rw-rw-  rw null
_smtpd   filter-regex  37523 kqueue 0xff0200e1cb40 4 state:
_smtpd   filter-regex  37526* unix stream 0x80bfdf80 <->
0x80be8100
_smtpd   filter-regex  37527* unix stream 0x80c75d80 <->
0x80c1a380
_smtpd   filter-regex  37528* unix stream 0x81461100 <->
0x8155ff80
_smtpd   filter-regex  37529* unix stream 0x814c7e00 <->
0x8160fe00
_smtpd   filter-regex  3752   10* unix stream 0x81489f80 <->
0x8140c180
_smtpd   filter-regex  3752   11* unix stream 0x814cc500 <->
0x814ad200
_smtpd   filter-regex  3752   12* unix stream 0x813c8700 <->
0x80f48c00
_smtpd   filter-regex  3752   13* unix stream 0x814fa000 <->
0x80bae000
_smtpd   filter-pause  3600   wd /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-pause  3600 root /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-pause  36000* unix stream 0x813c0d00 <->
0x80c75900
_smtpd   filter-pause  36001 /   27142 crw-rw-rw-  rw null
_smtpd   filter-pause  36002 /   27142 crw-rw-rw-  rw null
_smtpd   filter-pause  36003 kqueue 0xff0200e1cc80 0 state: W
_smtpd   filter-dnsbl 23238   wd /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-dnsbl 23238 root /var  1429120 drwxr-xr-x   r  512
_smtpd   filter-dnsbl 232380* unix stream 0x813c4000 <->
0x80f68e00
_smtpd   filter-dnsbl 232381 /   27142 crw-rw-rw-  rw null
_smtpd   filter-dnsbl 232382 /   27142 crw-rw-rw-  rw null
_smtpd   filter-dnsbl 232383 kqueue 0xff0200e1ca00 0 state: W
_smtpd   filter-dnsbl 232384* unix stream 0x80f31580 <->
0x8052e500
_smtpd

Re: filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-25 Thread Joerg Jung 
On Wed, May 25, 2016 at 04:28:59PM -0400, Andrew Ruscica wrote:
> Not sure if the issue lies with the spamassassin daemon or the
> opensmtp-filter.
> 
> During a surge of incoming email, the following errors were logged.
> OpenSMTP stopped relaying mail during this time:
> 
> May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
> shutdown: Bad file descriptor
> 
> May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
> 627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed
> 
> 
> The immediate resolution was to temporarily disable the filter and restart
> smtpd, although I think I probably could have resolved by restarting
> spamassassin as well..
> 
> I'm pre-forking 20 spamassassin children, with max of 40, and limiting
> incoming connections to opensmtpd using pf max states rules.
> 
> While this was occurring the spamd-child processes surged to about 30 and
> were starting to be released and returning back to normal as I was forcing
> down connections at pf.
> 
> using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
> from openbsd packages.
> 
> 
> Please let me know what other system details may be relevant here...

Can you provide smtpctl show stats and fstat -u _smtpd output?

You can try the latest -extras snapshot and especially the limit option.


-- 
You received this mail because you are subscribed to misc@opensmtpd.org
To unsubscribe, send a mail to: misc+unsubscr...@opensmtpd.org

filter-spamassassin fails under heavy load (may 23 extras snapshot)

2016-05-25 Thread Andrew Ruscica 
Not sure if the issue lies with the spamassassin daemon or the
opensmtp-filter.

During a surge of incoming email, the following errors were logged.
OpenSMTP stopped relaying mail during this time:

May 25 15:57:50 mxgw3 filter-spamassassin-reject[32220]: warn: response:
shutdown: Bad file descriptor

May 25 15:57:50 mxgw3 smtpd[25777]: smtp-in: Failed command on session
627872db6e9c0dc0: "DATA" => 451 4.7.1 Spam filter failed


The immediate resolution was to temporarily disable the filter and restart
smtpd, although I think I probably could have resolved by restarting
spamassassin as well..

I'm pre-forking 20 spamassassin children, with max of 40, and limiting
incoming connections to opensmtpd using pf max states rules.

While this was occurring the spamd-child processes surged to about 30 and
were starting to be released and returning back to normal as I was forcing
down connections at pf.

using opensmtpd-extras from May 23 on OpenBSD 5.9/amd64 with spamassassin
from openbsd packages.


Please let me know what other system details may be relevant here...