Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-31 Thread Julian H. Stacey 
Mark Sapiro wrote:
> On 08/30/2017 05:13 PM, Julian H. Stacey wrote:
> > 
> > 2.1.24 has defaults 0 300.  My most troublesome list had max 5/week =
> > member_verbosity_threshold: 5
> > member_verbosity_interval:  604800  
> > which may have occasionaly triggered correctly, but at least one
> > of the shunted people was someone who very rarely posts so at least that
> > shunt was unintended by traffic criteria.
> 
> 
> With the bug, the shunting will occur whenever a member's moderate flag
> is set for verbosity. The exception occurs after the flag is set in the
> attempt to log the fact.
> 
> Note that with the above settings on the list flagging doesn't require
> >5 posts to this list within a week, but >5 posts combined to all lists
> with member_verbosity_threshold > 0.

I hadn't realised that.  Explains why I saw so many shunts. 
I've reverted my lists to defaults.  Thanks.

Cheers,
Julian
-- 
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich
 Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable.
 http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-30 Thread Mark Sapiro 
On 08/30/2017 05:13 PM, Julian H. Stacey wrote:
> 
> 2.1.24 has defaults 0 300.  My most troublesome list had max 5/week =
>   member_verbosity_threshold: 5
>   member_verbosity_interval:  604800  
> which may have occasionaly triggered correctly, but at least one
> of the shunted people was someone who very rarely posts so at least that
> shunt was unintended by traffic criteria.


With the bug, the shunting will occur whenever a member's moderate flag
is set for verbosity. The exception occurs after the flag is set in the
attempt to log the fact.

Note that with the above settings on the list flagging doesn't require
>5 posts to this list within a week, but >5 posts combined to all lists
with member_verbosity_threshold > 0.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-30 Thread Julian H. Stacey 
Mark Sapiro wrote:
> On 08/30/2017 08:36 AM, Julian H. Stacey wrote:
> > Mark Sapiro wrote:
> >> It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.
> > 
> > Yes, it's missing.  Well analysed without seeing it, Thanks !  
> > So I patched just that line in.  
> > Mailman 2.1.22 source misses that line. 2.1.23 2.1.24 have it.
> 
> 
> This is a bug introduced in 2.1.21, never noticed and inadvertently
> fixed in 2.1.23. Presumably, it was never noticed before now because it
> was introduced with the member verbosity feature and only triggered if
> Privacy options... -> Sender filters -> member_verbosity_threshold is
> set greater than the default 0 and a "verbose" member is actually set to
> moderated.

2.1.24 has defaults 0 300.  My most troublesome list had max 5/week =
member_verbosity_threshold: 5
member_verbosity_interval:  604800  
which may have occasionaly triggered correctly, but at least one
of the shunted people was someone who very rarely posts so at least that
shunt was unintended by traffic criteria.


> > 2.1.23/NEWS has "Bug Fixes and other patches" ... SpamDetect.py
> 
> But that is in the 2.1.16 section of NEWS.

Oh yes, I hadn't noticed it was in an old section.
So this is a till now undocumented old fix :-)

Cheers,
Julian
-- 
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich
 Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable.
 http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-30 Thread Mark Sapiro 
On 08/30/2017 08:36 AM, Julian H. Stacey wrote:
> Mark Sapiro wrote:
>>
>> It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.
> 
> Yes, it's missing.  Well analysed without seeing it, Thanks !  
> So I patched just that line in.  
> Mailman 2.1.22 source misses that line. 2.1.23 2.1.24 have it.


This is a bug introduced in 2.1.21, never noticed and inadvertently
fixed in 2.1.23. Presumably, it was never noticed before now because it
was introduced with the member verbosity feature and only triggered if
Privacy options... -> Sender filters -> member_verbosity_threshold is
set greater than the default 0 and a "verbose" member is actually set to
moderated.


> 2.1.23/NEWS has "Bug Fixes and other patches" ... SpamDetect.py


But that is in the 2.1.16 section of NEWS.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-30 Thread Julian H. Stacey 
Mark Sapiro wrote:
> On 08/29/2017 03:59 PM, Julian H. Stacey wrote:
> 
> > Now I have:
> > # Note that leading whitespace is trimmed from the regexp.  This can be
> > # circumvented in a number of ways, e.g. by escaping or bracketing it.
> > bounce_matching_headers = """
> > # Lines that *start* with a '#' are comments.
> > to: fri...@public.com
> > message-id: relay.comanche.denmark.eu
> > from: l...@listme.com
> > from: .*@uplinkpro.com"""
> > 
> > ## Bounce options
> > before it was
> > # Note that leading whitespace is trimmed from the regexp.  This can be
> > # circumvented in a number of ways, e.g. by escaping or bracketing it.
> > bounce_matching_headers = """# Lines that *start* with a '#' are 
> > comments.
> > to: fri...@public.com
> > message-id: relay.comanche.denmark.eu
> > from: l...@listme.com
> > from: .*@uplinkpro.com"""
> 
> Actually, the only difference between the two is "Now" has a leading
> blank line which won't make any difference as blank lines are ignored.

OK,


> >> What is the traceback from the 'shunt' message in Mailman's error log?
> > 
> > /usr/local/mailman/logs/error
> >   (Aside: I also spotted a few other unassociated errors such as Hostile
> >   listname: & fixed pages that pointed to non existant local Mailman list 
> > names)
> > 
> > All my remaining errors now seem to be the same sort:
> > 
> > NameError: global name 'syslog' is not defined
> > 
> > Aug 23 13:37:16 2017 (82399) SHUNTING: 
> > 1503488012.441064+20509f89ec545c49966a4f1d747c804ea1df5852
> > Aug 24 20:56:44 2017 (18093) Uncaught runner exception: global name 
> > 'syslog' is not defined
> > Aug 24 20:56:44 2017 (18093) Traceback (most recent call last):
> >   File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in 
> > _oneloop
> > self._onefile(msg, msgdata)
> >   File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in 
> > _onefile
> > keepqueued = self._dispose(mlist, msg, msgdata)
> >   File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, 
> > in _dispose
> > more = self._dopipeline(mlist, msg, msgdata, pipeline)
> >   File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, 
> > in _dopipeline
> > sys.modules[modname].process(mlist, msg, msgdata)
> >   File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 136, 
> > in process
> > syslog('vette',
> > NameError: global name 'syslog' is not defined
> > 
> > Aug 24 20:56:44 2017 (18093) SHUNTING: 
> > 1503601003.748515+0146d084002ddd93cacadc41656c7632efc83301
> > 
> > /usr/local/mailman/Mailman/Handlers/SpamDetect.py line 136
> > syslog('vette',
> > '%s: Automatically Moderated %s for verbose postings.',
> >  mlist.real_name, sender)
> > 
> > Line 34:from Mailman import mm_cfg
> 
> 
> That and the next few lines should be
> 
> from Mailman import mm_cfg
> from Mailman import Errors
> from Mailman import i18n
> from Mailman import Utils
> from Mailman.Handlers.Hold import hold_for_approval
> from Mailman.Logging.Syslog import syslog
> 
> 
> It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.

Yes, it's missing.  Well analysed without seeing it, Thanks !  
So I patched just that line in.  
Mailman 2.1.22 source misses that line. 2.1.23 2.1.24 have it.

2.1.23/NEWS has "Bug Fixes and other patches" ... SpamDetect.py
I'd best upgrade from 2.1.22 to 2.1.24 when I can monitor traffic after.  

I checked in case anything else missing, but all seems OK :-)
  I compared 2.1.22 & 2.1.24, all Handlers/*.py , & SpamDetect.py is
  the only one varying between versions with Mailman.Logging.Syslog .
  Various other Handlers/*.py do not have Mailman.Logging.Syslog
  in 2.1.22 & 2.1.24, but this quick check passes:
cd /usr/local/mailman/Mailman/Handlers
grep -l syslog *.py > ~/tmp/1
grep -l Mailman.Logging.Syslog *.py > ~/tmp/2
cmp ~/tmp/1 ~/tmp/2
  (But I don't know Python, & not considered other directories, includes, libs).

Thanks Mark!

Cheers,
Julian
-- 
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich
 Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable.
 http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-29 Thread Mark Sapiro 
On 08/29/2017 03:59 PM, Julian H. Stacey wrote:

> Now I have:
>   # Note that leading whitespace is trimmed from the regexp.  This can be
>   # circumvented in a number of ways, e.g. by escaping or bracketing it.
>   bounce_matching_headers = """
>   # Lines that *start* with a '#' are comments.
>   to: fri...@public.com
>   message-id: relay.comanche.denmark.eu
>   from: l...@listme.com
>   from: .*@uplinkpro.com"""
>   
>   ## Bounce options
> before it was
>   # Note that leading whitespace is trimmed from the regexp.  This can be
>   # circumvented in a number of ways, e.g. by escaping or bracketing it.
>   bounce_matching_headers = """# Lines that *start* with a '#' are 
> comments.
>   to: fri...@public.com
>   message-id: relay.comanche.denmark.eu
>   from: l...@listme.com
>   from: .*@uplinkpro.com"""

Actually, the only difference between the two is "Now" has a leading
blank line which won't make any difference as blank lines are ignored.


>> What is the traceback from the 'shunt' message in Mailman's error log?
> 
> /usr/local/mailman/logs/error
>   (Aside: I also spotted a few other unassociated errors such as Hostile
>   listname: & fixed pages that pointed to non existant local Mailman list 
> names)
> 
> All my remaining errors now seem to be the same sort:
> 
>   NameError: global name 'syslog' is not defined
>   
>   Aug 23 13:37:16 2017 (82399) SHUNTING: 
> 1503488012.441064+20509f89ec545c49966a4f1d747c804ea1df5852
>   Aug 24 20:56:44 2017 (18093) Uncaught runner exception: global name 
> 'syslog' is not defined
>   Aug 24 20:56:44 2017 (18093) Traceback (most recent call last):
> File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in 
> _oneloop
>   self._onefile(msg, msgdata)
> File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in 
> _onefile
>   keepqueued = self._dispose(mlist, msg, msgdata)
> File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, 
> in _dispose
>   more = self._dopipeline(mlist, msg, msgdata, pipeline)
> File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, 
> in _dopipeline
>   sys.modules[modname].process(mlist, msg, msgdata)
> File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 136, 
> in process
>   syslog('vette',
>   NameError: global name 'syslog' is not defined
>   
>   Aug 24 20:56:44 2017 (18093) SHUNTING: 
> 1503601003.748515+0146d084002ddd93cacadc41656c7632efc83301
> 
> /usr/local/mailman/Mailman/Handlers/SpamDetect.py line 136
>   syslog('vette',
> '%s: Automatically Moderated %s for verbose postings.',
>  mlist.real_name, sender)
> 
>   Line 34:from Mailman import mm_cfg


That and the next few lines should be

from Mailman import mm_cfg
from Mailman import Errors
from Mailman import i18n
from Mailman import Utils
from Mailman.Handlers.Hold import hold_for_approval
from Mailman.Logging.Syslog import syslog


It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-29 Thread Julian H. Stacey 
Hi Mark, thanks for reply.
PS you'r doing a great job answering so many of us :-)

Mark Sapiro wrote:
> On 08/25/2017 06:29 AM, Julian H. Stacey wrote:
> > Hi mailman-users@python.org
> > I hope I may have solved some postings being shunted on a few of my lists,
> > necessitating occasional
> > cd /usr/local/mailman; unshunt qfiles/shunt
> > 
> > I discovered 3 lists had missing '\n', shown with
> > cd /usr/local/mailman/lists; config_list -o - ListName
> > bounce_matching_headers = """# Lines that *start* with a '#' are comments.
> 
> 
> What is the rest of this up to the terminating """?

Ah yes, I see what you mean, Its not a '' line, but a """ block:
Now I have:
# Note that leading whitespace is trimmed from the regexp.  This can be
# circumvented in a number of ways, e.g. by escaping or bracketing it.
bounce_matching_headers = """
# Lines that *start* with a '#' are comments.
to: fri...@public.com
message-id: relay.comanche.denmark.eu
from: l...@listme.com
from: .*@uplinkpro.com"""

## Bounce options
before it was
# Note that leading whitespace is trimmed from the regexp.  This can be
# circumvented in a number of ways, e.g. by escaping or bracketing it.
bounce_matching_headers = """# Lines that *start* with a '#' are 
comments.
to: fri...@public.com
message-id: relay.comanche.denmark.eu
from: l...@listme.com
from: .*@uplinkpro.com"""

## Bounce options

So fixed, but I guess that wasnt the cause of the shunts. 
Especially as I rcall a shunt on one of the other lists that didnt have that 
damage.


> > Fixed with 
> > cd /usr/local/mailman/lists ; config_list -o ~/tmp/ListName ListName
> > vi  ~/tmp/ListName ; config_list -i ~/tmp/ListName ListName
> > & I hope the shunting will stop.
> > 
> > I did autopsy snapshots of /usr/local/mailman before & after last unshunt,
> > if anyone feels like telling me what to look for.
> 
> 
> What is the traceback from the 'shunt' message in Mailman's error log?

/usr/local/mailman/logs/error
  (Aside: I also spotted a few other unassociated errors such as Hostile
  listname: & fixed pages that pointed to non existant local Mailman list names)

All my remaining errors now seem to be the same sort:

NameError: global name 'syslog' is not defined

Aug 23 13:37:16 2017 (82399) SHUNTING: 
1503488012.441064+20509f89ec545c49966a4f1d747c804ea1df5852
Aug 24 20:56:44 2017 (18093) Uncaught runner exception: global name 
'syslog' is not defined
Aug 24 20:56:44 2017 (18093) Traceback (most recent call last):
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in 
_oneloop
self._onefile(msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in 
_onefile
keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, 
in _dispose
more = self._dopipeline(mlist, msg, msgdata, pipeline)
  File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, 
in _dopipeline
sys.modules[modname].process(mlist, msg, msgdata)
  File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 136, 
in process
syslog('vette',
NameError: global name 'syslog' is not defined

Aug 24 20:56:44 2017 (18093) SHUNTING: 
1503601003.748515+0146d084002ddd93cacadc41656c7632efc83301

/usr/local/mailman/Mailman/Handlers/SpamDetect.py line 136
syslog('vette',
'%s: Automatically Moderated %s for verbose postings.',
 mlist.real_name, sender)

Line 34:from Mailman import mm_cfg
Mailman/mm_cfg.py has no syslog
So I'm searching for syslog in my FreeBSD /usr/ports/mail/mailman 
Got a bit lost maybe this might be FreeBSD specific,
Added maintainer of the FreeBSD ports wrapper in case of a hint ?


> > Unfortunately the corrupt bounce_matching_headers lines in configuration
> > were undetected by these automatic checks 
> > cd /usr/local/etc/rc.d; ./mailman stop ; ./mailman start 
> > cd /usr/local/mailman/lists ; ../bin/check_db -a
> > cd /usr/local/mailman/lists ; ../bin/check_perms
> 
> 
> If bounce_matching_headers contains an invalid line, there should be
> entries in Mailman's 'config' log like
> 
> 'bad bounce_matching_header line: LISTNAME
> LINE
> 
> or
> 
> bad regexp in bounce_matching_header line: LISTNAME
> REGEXP (cause: ERROR)

cd /usr/local/mailman/logs
grep bad *
grep header *
No other errors, apart from 
admin: No such list "admin":

Thanks,
Julian
-- 
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich
 Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable.

Re: [Mailman-Users] periodic shunts might have been undetected bad bounce_matching_headers

2017-08-28 Thread Mark Sapiro 
On 08/25/2017 06:29 AM, Julian H. Stacey wrote:
> Hi mailman-users@python.org
> I hope I may have solved some postings being shunted on a few of my lists,
> necessitating occasional
>   cd /usr/local/mailman; unshunt qfiles/shunt
> 
> I discovered 3 lists had missing '\n', shown with
>   cd /usr/local/mailman/lists; config_list -o - ListName
> bounce_matching_headers = """# Lines that *start* with a '#' are comments.


What is the rest of this up to the terminating """?

> Fixed with 
>   cd /usr/local/mailman/lists ; config_list -o ~/tmp/ListName ListName
>   vi  ~/tmp/ListName ; config_list -i ~/tmp/ListName ListName
> & I hope the shunting will stop.
> 
> I did autopsy snapshots of /usr/local/mailman before & after last unshunt,
> if anyone feels like telling me what to look for.


What is the traceback from the 'shunt' message in Mailman's error log?


> Unfortunately the corrupt bounce_matching_headers lines in configuration
> were undetected by these automatic checks 
>   cd /usr/local/etc/rc.d; ./mailman stop ; ./mailman start 
>   cd /usr/local/mailman/lists ; ../bin/check_db -a
>   cd /usr/local/mailman/lists ; ../bin/check_perms


If bounce_matching_headers contains an invalid line, there should be
entries in Mailman's 'config' log like

'bad bounce_matching_header line: LISTNAME
LINE

or

bad regexp in bounce_matching_header line: LISTNAME
REGEXP (cause: ERROR)


-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
--
Mailman-Users mailing list Mailman-Users@python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-users/archive%40jab.org