Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 23:46, Kirill A. Korinsky  wrote:
> 
>> On 28. Dec 2023, at 23:34, Martijn van Duren  
>> wrote:
>> This would make filter-dnsbl grow way beyond of what it was ever
>> intended for into something where I'm afraid is not properly
>> maintainable for both programmer and admin.
>> 
>> Maybe you can write your own filter-dnswl with filter-dnsbl and
>> filter-admdscrub as inspiration.
> 
> Or I may make a series of patch over your code. It seems not that complicated.

I mean something like that: https://github.com/catap/filter-dnsbl 


I've split it into two commits to make logic clear.


--
wbr, Kirill




signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 23:34, Martijn van Duren  
> wrote:
> 
> I've never used sieve, but this already is a custom rule and not a
> X-Spam specific header check from sieve itself. However, a quick
> online search shows me the i;ascii-casemap comparator. Maybe you
> can give that a try.
> 
> Also, smtpd.conf(5), from which I've taken the string, uses a lower
> case "yes".

Well... OpenSMTPD checks that value via strcasecmp to avoid case issue, see:
 
https://github.com/OpenSMTPD/OpenSMTPD/blob/7.4.0p1/usr.sbin/smtpd/mail.maildir.c#L192-L196

but setup it as Yes, see: 
https://github.com/OpenSMTPD/OpenSMTPD/blob/7.4.0p1/usr.sbin/smtpd/smtp_session.c#L2772-L2773


>> The idea of both changes to use white lists to remove X-Spam: yes from both 
>> negative filters.
> 
> The discussion about having a single whitelist overrule a whole sluice
> of lists apart. If you want whitelisting in filter-dnsbl it would
> require special handling of a whole range of different options:
> - Which response means what for what list

I assume any found response, like it works now with black listing.

> - Which option takes precedence in what situation

I assume that white list override the black one.

> - Do we need to remove existing X-Spam headers?

If we may do it in one run, we simple don't need to setup it.

> - Are there other headers that need removing/modifying?

I don't see any of this.

> This would make filter-dnsbl grow way beyond of what it was ever
> intended for into something where I'm afraid is not properly
> maintainable for both programmer and admin.
> 
> Maybe you can write your own filter-dnswl with filter-dnsbl and
> filter-admdscrub as inspiration.

Or I may make a series of patch over your code. It seems not that complicated.

 But I haven't see any easy way to implement it for non -m case.
 
 During read the code of this filter I guess I've found third point which 
 I'd like to raise: filter fails in the case when one of provided DNSBL 
 returns error.
 
 Shall it continue?
>>> 
>>> If a filter (or the intermediate DNS layer) returns an error we are in
>>> limbo. If we accept the mail, but it's listed we're probably delivering
>>> spam; if we reject the mail we're very likely to drop legit mail. Both
>>> are undesirable. Failing the message asking to try again later seems the
>>> safest option to me.
>> 
>> I see your point.
>> 
>> My point: user may wait messages and to be very nervous if it delayed for a 
>> while.
>> 
>> Important message means something like a ticket for a train in 5-15 minutes 
>> or something like that.
>> 
>> And here DNS seems like a single point of failure.
> 
> Sure, but if I'm in a hurry and need a ticket I'm not going to rely on
> mail anyway. Either I'm going to buy it at the door, or I hope they have
> an option to download the ticket from the browser (which most of the
> ticket purchases I make have an option for). Only as a last resort I'm
> going to rely mail and just hope that everything works as it should.

Well, this is an example from the last week :)

If I haven't open DB application for a while, more than a month it had missed 
updated of so-called Deutschlandticket, and I wait the email with approval code 
to re-download it to the application.

I know that is edge case, but DNS failure is also edge case.

>> I think that it should be configurable by bypass DNS error by probability of 
>> delivering spam instead of delaying everything.
> 
> Even as an option I'm not particularly fond of the idea... But if enough
> people think it's a worthwhile addition (you're the first one in 4 years
> to have raised this request) and the diff is small enough I might
> consider it.


From another hand, locally installed unbound should solve that issue on the 
first place.

--
wbr, Kirill




signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Martijn van Duren]

On Thu, 2023-12-28 at 22:40 +0100, Kirill A. Korinsky wrote:
> > On 28. Dec 2023, at 22:17, Martijn van Duren  
> > wrote:
> > 
> > On Thu, 2023-12-28 at 20:05 +0100, Kirill A. Korinsky wrote:
> > > 
> > > Example of software is sieve: as far as I know it doesn't support 
> > > matching case insensitive strings.
> > 
> > Does this currently pose a problem for anyone? If not I don't see a
> > reason to make a new release for it.
> 
> 
> Well, right now it should be written in sieve like this:
> 
> > if anyof(header :is "X-Spam" "yes", header :is "X-Spam" "Yes") {
> >   fileinto :create "Junk";
> > }
> 
> 
> a bit ugly but I agree that it's not enough for a release.

I've never used sieve, but this already is a custom rule and not a
X-Spam specific header check from sieve itself. However, a quick
online search shows me the i;ascii-casemap comparator. Maybe you
can give that a try.

Also, smtpd.conf(5), from which I've taken the string, uses a lower
case "yes".
> 
> 
> > > 
> > > I've read the code and I agree that implementing whitelist can be quite 
> > > tricky.
> > > 
> > > Anyway, I do have one idea: let introduce flag -i (inverse). It should 
> > > remove X-Spam: yes if matches with -m.
> > 
> > Why? What does this bring? Why do you need this?
> 
> 
> Right now I'm making a kind of the experiment: living without any statistical 
> analyser for my mail.
> 
> The hypotesa: usual mail traffic for family mail server is too small to make 
> statistical analyser like spamassasian useful by impact of mails.
> 
> So, I'm testing another approach: only DNSBL which delivers everything that 
> is matched into Junk folder.
> 
> Long story short I do have the following configuration:
> 
>  - NiX Spam and blocklist.de at spamd
>  - smptd configuration:
> 
> > filter dnsbl proc-exec "filter-dnsbl -m \
> >        all.spamrats.com \
> >        bip.virusfree.cz \
> >        bl.mailspike.org \
> >        bl.spamcop.net \
> >        bl.spameatingmonkey.net \
> >        cbl.abuseat.org \
> >        dnsbl-1.uceprotect.net \
> >        dnsbl.dronebl.org \
> >        mail-abuse.blacklist.jippg.org \
> >        psbl.surriel.com \
> >        rbl.0spam.org \
> >        truncate.gbudb.net \
> >        zen.spamhaus.org \
> >        XXX.combined.mail.abusix.zone \
> >        "
> > 
> > filter dnswl proc-exec "filter-dnsbl -m \
> >        XXX.white.mail.abusix.zone \
> >        "
> > 
> > filter "senderscore" proc-exec "filter-senderscore -junkBelow 70"
> > 
> > listen on egress inet4 port smtp tls pki mx1.catap.net \
> >        filter { "rdns", "fcrdns", dnsbl, dnswl, "senderscore" }
> 
> - and sieve script to move mail:
> 
> > if allof(
> >   anyof(header :is "X-Spam" "yes", header :is "X-Spam" "Yes"),
> >   not header :is "X-Spam-DNSBL" "Listed at white.mail.abusix.zone") {
> >     fileinto :create "Junk";
> > }
> 
> this works like a charm, really.
> 
> The idea of both changes to use white lists to remove X-Spam: yes from both 
> negative filters.

The discussion about having a single whitelist overrule a whole sluice
of lists apart. If you want whitelisting in filter-dnsbl it would
require special handling of a whole range of different options:
- Which response means what for what list
- Which option takes precedence in what situation
- Do we need to remove existing X-Spam headers?
- Are there other headers that need removing/modifying?

This would make filter-dnsbl grow way beyond of what it was ever
intended for into something where I'm afraid is not properly
maintainable for both programmer and admin.

Maybe you can write your own filter-dnswl with filter-dnsbl and
filter-admdscrub as inspiration.
> 
> > > 
> > > But I haven't see any easy way to implement it for non -m case.
> > > 
> > > During read the code of this filter I guess I've found third point which 
> > > I'd like to raise: filter fails in the case when one of provided DNSBL 
> > > returns error.
> > > 
> > > Shall it continue?
> > 
> > If a filter (or the intermediate DNS layer) returns an error we are in
> > limbo. If we accept the mail, but it's listed we're probably delivering
> > spam; if we reject the mail we're very likely to drop legit mail. Both
> > are undesirable. Failing the message asking to try again later seems the
> > safest option to me.
> 
> I see your point.
> 
> My point: user may wait messages and to be very nervous if it delayed for a 
> while.
> 
> Important message means something like a ticket for a train in 5-15 minutes 
> or something like that.
> 
> And here DNS seems like a single point of failure.

Sure, but if I'm in a hurry and need a ticket I'm not going to rely on
mail anyway. Either I'm going to buy it at the door, or I hope they have
an option to download the ticket from the browser (which most of the
ticket purchases I make have an option for). Only as a last resort I'm
going to rely mail and just hope that everything works as it should.
> 
> I think that it should be configurable by bypass DNS error by probability of 
> 

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 22:17, Martijn van Duren  
> wrote:
> 
> On Thu, 2023-12-28 at 20:05 +0100, Kirill A. Korinsky wrote:
>> 
>> 
>> Example of software is sieve: as far as I know it doesn't support matching 
>> case insensitive strings.
> 
> Does this currently pose a problem for anyone? If not I don't see a
> reason to make a new release for it.

Well, right now it should be written in sieve like this:

> if anyof(header :is "X-Spam" "yes", header :is "X-Spam" "Yes") {
>   fileinto :create "Junk";
> }

a bit ugly but I agree that it's not enough for a release.


>> 
>> I've read the code and I agree that implementing whitelist can be quite 
>> tricky.
>> 
>> Anyway, I do have one idea: let introduce flag -i (inverse). It should 
>> remove X-Spam: yes if matches with -m.
> 
> Why? What does this bring? Why do you need this?

Right now I'm making a kind of the experiment: living without any statistical 
analyser for my mail.

The hypotesa: usual mail traffic for family mail server is too small to make 
statistical analyser like spamassasian useful by impact of mails.

So, I'm testing another approach: only DNSBL which delivers everything that is 
matched into Junk folder.

Long story short I do have the following configuration:

 - NiX Spam and blocklist.de  at spamd
 - smptd configuration:

> filter dnsbl proc-exec "filter-dnsbl -m \
>all.spamrats.com \
>bip.virusfree.cz \
>bl.mailspike.org \
>bl.spamcop.net \
>bl.spameatingmonkey.net \
>cbl.abuseat.org \
>dnsbl-1.uceprotect.net \
>dnsbl.dronebl.org \
>mail-abuse.blacklist.jippg.org \
>psbl.surriel.com \
>rbl.0spam.org \
>truncate.gbudb.net \
>zen.spamhaus.org \
>XXX.combined.mail.abusix.zone \
>"
> 
> filter dnswl proc-exec "filter-dnsbl -m \
>XXX.white.mail.abusix.zone \
>"
> 
> filter "senderscore" proc-exec "filter-senderscore -junkBelow 70"
> 
> listen on egress inet4 port smtp tls pki mx1.catap.net \
>filter { "rdns", "fcrdns", dnsbl, dnswl, "senderscore" }


- and sieve script to move mail:

> if allof(
>   anyof(header :is "X-Spam" "yes", header :is "X-Spam" "Yes"),
>   not header :is "X-Spam-DNSBL" "Listed at white.mail.abusix.zone") {
> fileinto :create "Junk";
> }


this works like a charm, really.

The idea of both changes to use white lists to remove X-Spam: yes from both 
negative filters.

>> 
>> But I haven't see any easy way to implement it for non -m case.
>> 
>> During read the code of this filter I guess I've found third point which I'd 
>> like to raise: filter fails in the case when one of provided DNSBL returns 
>> error.
>> 
>> Shall it continue?
> 
> If a filter (or the intermediate DNS layer) returns an error we are in
> limbo. If we accept the mail, but it's listed we're probably delivering
> spam; if we reject the mail we're very likely to drop legit mail. Both
> are undesirable. Failing the message asking to try again later seems the
> safest option to me.

I see your point.

My point: user may wait messages and to be very nervous if it delayed for a 
while.

Important message means something like a ticket for a train in 5-15 minutes or 
something like that.

And here DNS seems like a single point of failure.

I think that it should be configurable by bypass DNS error by probability of 
delivering spam instead of delaying everything.

--
wbr, Kirill




signature.asc
Description: Message signed with OpenPGP

Re: filter-dkimverify status

[Martijn van Duren]

On Thu, 2023-12-28 at 18:29 +0100, Kirill A. Korinsky wrote:
> Greetings,
> 
> This list seems to be the right place to ask about status of this filter 
> http://imperialat.at/dev/filter-dkimverify/
> 
> As I see via snv log the last commit was about a year ago.
> 
> Is it working? Or?
> 
> 
> 
> -- 
> wbr, Kirill
> 
It's fully functional. But due to circumstances I haven't made a release
yet. No promises on when I'll get around to it.
Feel free to use it as is and report any issues.

martijn@

Re: filter-dnsbl: feature request and bug report

[Martijn van Duren]

On Thu, 2023-12-28 at 20:05 +0100, Kirill A. Korinsky wrote:
> > On 28. Dec 2023, at 19:22, Martijn van Duren  
> > wrote:
> > 
> > On Thu, 2023-12-28 at 18:52 +0100, Kirill A. Korinsky wrote:
> > > 
> > > 
> > > 1. I'd like to report a small bug: OpenSMTPD injects header X-Spam: Yes 
> > > when filter decided that it is junk, and this filter injects yes (in the 
> > > lower case). Is it a bug?
> > 
> > I'm not aware about any specific standard when it comes to this header.
> > smtpd has `strcasecmp(line, "x-spam: yes")` in mail.maildir.c, so the
> > capitalisation isn't important there. However, if you use smtpd with
> > filter-dnsbl as a filter before forwarding it to another server which
> > does check this header in a case-sensitive manner it would matter.
> > So unless you can point to a piece of software which checks it in a
> > specific capitalisation I don't see a direct reason to fix it.
> 
> Neither do I.
> 
> Example of software is sieve: as far as I know it doesn't support matching 
> case insensitive strings.

Does this currently pose a problem for anyone? If not I don't see a
reason to make a new release for it.
> 
> > > 
> > > 2. Is it possible to add support of white list(s)? Let pass the message 
> > > if it contains on that list and optionally adds X-Spam-DNSWL: Listed at 
> > > ... header.
> > 
> > I've thought about adding whitelist support, but it's tough...
> > The RFC is explicitly vague on how to interpret responses, so there's
> > no way to determine how a response it to be interpreted without
> > extensive configuration either by the admin, or in the binary.
> > The latter would require in-depth knowledge of the different lists
> > on my end and actively maintain those, which I don't see myself doing.
> > 
> > Another reason why I don't see myself supporting whitelists is because
> > I don't see their value. Mail is whitelist based in principle, so what
> > blocking feature is it supposed to overwrite and how is filter-dnsbl
> > supposed to do this? And that's not even starting on the prioritisation
> > of which list takes precedence.
> > 
> > Do you have a specific use-case for whitelisting and if so, how would
> > you suggest to implement it in a generic way without making the filter
> > needlessly complicated?
> > 
> > In short: I'm not against whitelists per se, but without the specific
> > use-case and a good plan of how to implement it without turning it
> > into a coding/admin nightmare I don't see it happening.
> 
> 
> I've read the code and I agree that implementing whitelist can be quite 
> tricky.
> 
> Anyway, I do have one idea: let introduce flag -i (inverse). It should remove 
> X-Spam: yes if matches with -m.

Why? What does this bring? Why do you need this?
> 
> But I haven't see any easy way to implement it for non -m case.
> 
> During read the code of this filter I guess I've found third point which I'd 
> like to raise: filter fails in the case when one of provided DNSBL returns 
> error.
> 
> Shall it continue?

If a filter (or the intermediate DNS layer) returns an error we are in
limbo. If we accept the mail, but it's listed we're probably delivering
spam; if we reject the mail we're very likely to drop legit mail. Both
are undesirable. Failing the message asking to try again later seems the
safest option to me.
> 
> --
> wbr, Kirill
> 

Re: filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

> On 28. Dec 2023, at 19:22, Martijn van Duren  
> wrote:
> 
> On Thu, 2023-12-28 at 18:52 +0100, Kirill A. Korinsky wrote:
>> 
>> 
>> 1. I'd like to report a small bug: OpenSMTPD injects header X-Spam: Yes when 
>> filter decided that it is junk, and this filter injects yes (in the lower 
>> case). Is it a bug?
> 
> I'm not aware about any specific standard when it comes to this header.
> smtpd has `strcasecmp(line, "x-spam: yes")` in mail.maildir.c, so the
> capitalisation isn't important there. However, if you use smtpd with
> filter-dnsbl as a filter before forwarding it to another server which
> does check this header in a case-sensitive manner it would matter.
> So unless you can point to a piece of software which checks it in a
> specific capitalisation I don't see a direct reason to fix it.

Neither do I.

Example of software is sieve: as far as I know it doesn't support matching case 
insensitive strings.

>> 
>> 2. Is it possible to add support of white list(s)? Let pass the message if 
>> it contains on that list and optionally adds X-Spam-DNSWL: Listed at ... 
>> header.
> 
> I've thought about adding whitelist support, but it's tough...
> The RFC is explicitly vague on how to interpret responses, so there's
> no way to determine how a response it to be interpreted without
> extensive configuration either by the admin, or in the binary.
> The latter would require in-depth knowledge of the different lists
> on my end and actively maintain those, which I don't see myself doing.
> 
> Another reason why I don't see myself supporting whitelists is because
> I don't see their value. Mail is whitelist based in principle, so what
> blocking feature is it supposed to overwrite and how is filter-dnsbl
> supposed to do this? And that's not even starting on the prioritisation
> of which list takes precedence.
> 
> Do you have a specific use-case for whitelisting and if so, how would
> you suggest to implement it in a generic way without making the filter
> needlessly complicated?
> 
> In short: I'm not against whitelists per se, but without the specific
> use-case and a good plan of how to implement it without turning it
> into a coding/admin nightmare I don't see it happening.


I've read the code and I agree that implementing whitelist can be quite tricky.

Anyway, I do have one idea: let introduce flag -i (inverse). It should remove 
X-Spam: yes if matches with -m.

But I haven't see any easy way to implement it for non -m case.

During read the code of this filter I guess I've found third point which I'd 
like to raise: filter fails in the case when one of provided DNSBL returns 
error.

Shall it continue?

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

Re: filter-dnsbl: feature request and bug report

[Martijn van Duren]

On Thu, 2023-12-28 at 18:52 +0100, Kirill A. Korinsky wrote:
> Greetings,
> 
> This list seems as the right place to discuss about 
> http://imperialat.at/dev/filter-dnsbl

As the author, I agree. :-)
> 
> 1. I'd like to report a small bug: OpenSMTPD injects header X-Spam: Yes when 
> filter decided that it is junk, and this filter injects yes (in the lower 
> case). Is it a bug?

I'm not aware about any specific standard when it comes to this header.
smtpd has `strcasecmp(line, "x-spam: yes")` in mail.maildir.c, so the
capitalisation isn't important there. However, if you use smtpd with
filter-dnsbl as a filter before forwarding it to another server which
does check this header in a case-sensitive manner it would matter.
So unless you can point to a piece of software which checks it in a
specific capitalisation I don't see a direct reason to fix it.
> 
> 2. Is it possible to add support of white list(s)? Let pass the message if it 
> contains on that list and optionally adds X-Spam-DNSWL: Listed at ... header.

I've thought about adding whitelist support, but it's tough...
The RFC is explicitly vague on how to interpret responses, so there's
no way to determine how a response it to be interpreted without
extensive configuration either by the admin, or in the binary.
The latter would require in-depth knowledge of the different lists
on my end and actively maintain those, which I don't see myself doing.

Another reason why I don't see myself supporting whitelists is because
I don't see their value. Mail is whitelist based in principle, so what
blocking feature is it supposed to overwrite and how is filter-dnsbl
supposed to do this? And that's not even starting on the prioritisation
of which list takes precedence.

Do you have a specific use-case for whitelisting and if so, how would
you suggest to implement it in a generic way without making the filter
needlessly complicated?

In short: I'm not against whitelists per se, but without the specific
use-case and a good plan of how to implement it without turning it
into a coding/admin nightmare I don't see it happening.
> 
> The nice example of white list is dnswl.mail.abusix.zone
> 
> Thanks.
> 
> 
> 
> -- 
> wbr, Kirill
> 
martijn@

filter-dnsbl: feature request and bug report

[Kirill A. Korinsky]

Greetings,

This list seems as the right place to discuss about 
http://imperialat.at/dev/filter-dnsbl 

1. I'd like to report a small bug: OpenSMTPD injects header X-Spam: Yes when 
filter decided that it is junk, and this filter injects yes (in the lower 
case). Is it a bug?

2. Is it possible to add support of white list(s)? Let pass the message if it 
contains on that list and optionally adds X-Spam-DNSWL: Listed at ... header.

The nice example of white list is dnswl.mail.abusix.zone

Thanks.

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP

filter-dkimverify status

[Kirill A. Korinsky]

Greetings,

This list seems to be the right place to ask about status of this filter 
http://imperialat.at/dev/filter-dkimverify/ 


As I see via snv log the last commit was about a year ago.

Is it working? Or?

--
wbr, Kirill



signature.asc
Description: Message signed with OpenPGP