Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-12 Thread Phil Pennock 
On 2017-09-12 at 09:45 +0100, Jeremy Harris wrote:
> On 11/09/17 23:35, Phil Pennock wrote:
> > So there's some value in having an optional, EXPERIMENTAL_FOO feature,
> > for more advanced DNS handling.
> 
> As an alternate implementation, a standard (but custom-configured)
> resolver packaged with Exim in a container?
> 
> Would that suffice, or would the libresolv API still be insufficiently
> rich to do the Very Handy odd queries forseen fr apam-fighting?

Probably suffice for Rob, but the point around the "NS above the cut" is
that it's something no normal resolver ever asks for.  The libresolv
portable API provides no interface for choosing which IPs to send
messages to.

It's easy enough in Go, Python, Perl although in no case with the core
language or stdlib, only with "very common" ecosystem libraries.  For C,
we'd be looking at the same solution.  The people who created the getdns
API know what they're doing and so it's solidly worthy of consideration,
but I haven't done anything with it to be able to endorse it as "the
right way".  Only "the way I kept meaning to find time to look at."

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-12 Thread Vsevolod Stakhov 
On 11/09/17 23:35, Phil Pennock wrote:
> On 2017-09-11 at 20:58 +0200, Heiko Schlittermann via Exim-dev wrote:
>> I'm not sure if I got it. You want to re-invent a caching name service
>> inside Exim?
> 
> No, just a resolver which can be custom-configured.  There's a lot of
> value for spam-fighting in being able to do custom DNS queries, although
> whether that belongs in Exim vs rspamd or whatever is a different
> question.

That's all doable in Rspamd indeed: it has its own resolver library and
it uses persistent processes which also allows caching if needed (and it
uses quite powerful LFU algorithm: http://antirez.com/news/109).

However, the current integration between Rspamd and Exim is not very
good so far (legacy proto which I have to support in Rspamd just for
Exim, absence of interaction on before-data stages and so on). Perhaps,
it is a topic of some separate discussion at some point...


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-12 Thread Jeremy Harris 
On 11/09/17 23:35, Phil Pennock wrote:
> So there's some value in having an optional, EXPERIMENTAL_FOO feature,
> for more advanced DNS handling.

As an alternate implementation, a standard (but custom-configured)
resolver packaged with Exim in a container?

Would that suffice, or would the libresolv API still be insufficiently
rich to do the Very Handy odd queries forseen fr apam-fighting?
-- 
Cheers,
  Jeremy



-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Viktor Dukhovni 

> On Sep 11, 2017, at 6:35 PM, Phil Pennock  wrote:
> 
> For the DNSSEC folks, being able to affirm validation
> without trusting across-the-wire that AD bit is tamper-proof is nice.

DNSSEC in stub resolvers is rather nice in theory, however, in
practice DNSSEC needs to happen in the iterative resolver. See:

https://github.com/kazu-yamamoto/dns/issues/54#issuecomment-327932473

-- 
Viktor.


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Rob McEwen 

On 9/11/2017 6:35 PM, Phil Pennock wrote:

I think it's worth noting here that Rob_runs_  a commercial RBL (I'm
actually a paying customer of his) and so is in a good situation to be
able to make the call about load and caching.



Phil,

I confess, I didn't not know that until you just mentioned it. Forgive 
me for not recognizing your name! I just looked it up and, sure enough, 
you're a customer. Small world!


Currently, there are about 450 invaluement subscribers, with sometimes 
multiple contact people per account - so I have a hard time keeping up 
with everyone's name. I'll try to do better!


--
Rob McEwen
http://www.invaluement.com
+1 (478) 475-9032



--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Phil Pennock 
On 2017-09-11 at 20:58 +0200, Heiko Schlittermann via Exim-dev wrote:
> I'm not sure if I got it. You want to re-invent a caching name service
> inside Exim?

No, just a resolver which can be custom-configured.  There's a lot of
value for spam-fighting in being able to do custom DNS queries, although
whether that belongs in Exim vs rspamd or whatever is a different
question.

Example query: "nameservers for this domain as registered in the parent
zone".  NS records are authoritative below the cut and normal APIs will
always return the below-cut answer.  Fast-flux DNS abusing spammers,
pointing NS records at compromised hosts, use this and have done for
years.  Having the ability to look up the records registered in the
parent zone and use _those_ in an RBL-style lookup ... that would be
Very Handy.

So there's some value in having an optional, EXPERIMENTAL_FOO feature,
for more advanced DNS handling.  We have pseudo-RRtypes for dnsdb
already, but adding "NS at parent" couldn't be done with the current
libresolv API.  I've been repeatedly disgusted at the hoops we have to
go through with the `_res` symbol and handling it cleanly, including on
NetBSD which is Different From Everyone Else To Be More Correct.
Replacing all of the DNS lookups with getdns-based handling, if built
with EXPERIMENTAL_GETDNS, and then adding a few more features too, could
be sensible.  For the DNSSEC folks, being able to affirm validation
without trusting across-the-wire that AD bit is tamper-proof is nice.

I think it's worth noting here that Rob _runs_ a commercial RBL (I'm
actually a paying customer of his) and so is in a good situation to be
able to make the call about load and caching.

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Viktor Dukhovni 

> On Sep 11, 2017, at 4:39 PM, Rob McEwen  wrote:
> 
> Given such a tiny amount of time of the caching, and the fact that this would 
> be limited in scope to only DNSBLs for which the exim admin goes out of their 
> way to implement this feature - I highly doubt that there would be that much 
> added memory or CPU overhead resulting from such a feature - and it would be 
> totally innocuous to everyone else who didn't choose to use this feature.
> 
> If you don't particularly care for this feature (should it ever be 
> implemented) simply don't go out of your way to use it!

Adding a cache to the Exim server would be rather non-trivial,
because the Exim SMTP server is stateless.  IIRC Each connection
is handled by a separate process.  It still seems to me that the
problem is best solved by making it trivial to deploy the desired
nameserver configuration, without changing Exim.

As for me, I don't use Exim, I mostly just lurk on this and the
users list, and respond just to questions about TLS and DANE, but
DNS is also rather close to my area of expertise, so I decided to
delurk in this thread.

The real Exim developers will decide what they want to do with or
without my opinion...

-- 
Viktor.


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Rob McEwen 

On 9/11/2017 4:19 PM, Viktor Dukhovni wrote:
> On Sep 11, 2017, at 3:52 PM, Rob McEwen  wrote:
>>
>> Again, if this feature were implemented - as described - it would be 
completely innocuous to those didn't go out of their way to implement 
this. Of course, I would want the implementation by the end user to be 
very very simple too - but it wouldn't be something that someone could 
easily mistakenly do, either.

>
> The problem is that with resolver bypass you lose caching, even
> when receiving multiple concurrent or closely spaced in time
> messages from a legitimate high-volume sender.
>
> The performance impact of this is considerable and undesirable.
>
> As you may know, I am not an Exim developer, so take my opinion for
> what it is worth, but IMHO implementing a DNS cache in Exim would not
> be a good architectural choice, and without caching the feature is too
> costly.
>
> If the incentives on your end are compelling, provide your customers with
> a software package that installs a properly configured local resolver for
> their use.  This will also work with other MTAs and unpatched versions of
> Exim.
>
> Your problem seems real enough, and yet your proposed solution may not be
> the way forward.

Adding an internal caching features for internal-only DNSBL lookups in 
Exim... isn't trivial. However, generally speaking, *all* DNSBL caching 
is ONLY valuable for a few minutes, at most. Caching for more time 
causes unnecessary False Positives for DNSBL lookups. Also, as I had 
mentioned, this would be limited in scope to ONLY for the particular 
DNSBL(s) for which the Exim admin had explicitly implemented this 
feature, and only caching results for up to a few minutes (or the 
DNSBL's ttl)


NOTE: For perspective, Spamhaus's Zen list has a 10 second TTL. This 
means that a DNS server is ONLY suppose to cache ONLY the last 10 
seconds worth of Spamhaus lookups! Therefore, it would be quite 
reasonable to limit the TTL of such an internal Exim DNS cache... to 
ONLY what was queried in the last 10 seconds!


Given such a tiny amount of time of the caching, and the fact that this 
would be limited in scope to only DNSBLs for which the exim admin goes 
out of their way to implement this feature - I highly doubt that there 
would be that much added memory or CPU overhead resulting from such a 
feature - and it would be totally innocuous to everyone else who didn't 
choose to use this feature.


If you don't particularly care for this feature (should it ever be 
implemented) simply don't go out of your way to use it!


--
Rob McEwen
http://www.invaluement.com
+1 (478) 475-9032



--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Rob McEwen 

On 9/11/2017 3:28 PM, Viktor Dukhovni wrote:
> but I would hope
> that resolver bypass does not become a supported Exim feature.

It would still do an NS lookup on the particular DNSBL, first using the 
standard (current) system. And the implementation would never default to 
this - the end users would have to go out of their way to implement 
this! And even then - it wouldn't impact OTHER queries - which would 
continue to operate as usual. It would only surgically starget the 
particular DNSBL selected, on a case-by-case basis, with the user having 
to purposely go out of their way to implement this for any one DNSBL.


> All the problems the OP would like to solve are best handled via a 
dedicated

> local resolver.  That resolver can forward queries to some more central
> resolver and define stub zones (or an appropriate alternative 
mechanism) for

> whatever RBL domains it would like to bypass the upstream cache.

I wonder if you read my other messages? even with explicit and clear 
instructions - including warnings on step one of our signup form - and 
even though those instructions are almost condescending/insulting to our 
subscribers the way we try hard to clearly spell out how important it is 
for them to use a locally-hosted DNS resolver - even with all of that - 
about 25% of the entire overhead labor hours time running invaluement - 
involves constantly having to spend time contacting customers (and 
potential customers in a trial)... to get them to fix their DNS so that 
the queries stop coming from Google or OpenDNS. Yet for those few spam 
filters which do have a "query DNSBL provider's server directly" feature 
- these problems NEVER happen.


Viktor, if all of our subscribers had your particular expertise and 
knowledge, I'm sure I wouldn't have to bother trying to figure this out. 
But as I had mentioned, it isn't that they are all stupid. Most of the 
time, they are IT admins that have a million other responsibilities 
besides managing the mail server and DNS.  Ideally, you are 100% correct 
about there not being a need to do this - but that is only true in an 
ideal worth that doesn't exit. I have a mountain of evidence that in the 
real world, this feature would very much be of great benefit for many.


Again, if this feature were implemented - as described - it would be 
completely innocuous to those didn't go out of their way to implement 
this. Of course, I would want the implementation by the end user to be 
very very simple too - but it wouldn't be something that someone could 
easily mistakenly do, either.


--
Rob McEwen
http://www.invaluement.com
+1 (478) 475-9032



--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Viktor Dukhovni 

> On Sep 11, 2017, at 2:58 PM, Heiko Schlittermann via Exim-dev 
>  wrote:
> 
>> What I want to accomplish is this: provide subscribers to the invaluement
>> anti-spam blacklist... who use exim... the ability to have their DNS queries
>> to DNSBLs... come directly from Exim, skipping the normal DNS resolver. (and
>> other DNSBLs could benefit from this too!)
>> 
>> The way this would work... is that Exim would do a normal NS lookup on the
>> host name at the root of a DNSBL (eg "zen.spamhaus.org", for example), then
>> collect IP address(es) that those authoritative name servers resolve to, and
>> then do the actual DNSBL lookup *directly* on that DNSBL's authoritative
>> servers, skipping the regular caching DNS server "middleman".
> 
> I'm not sure if I got it. You want to re-invent a caching name service
> inside Exim? What's wrong with installing a caching (and validating)
> resolver next to the host Exim is running on? (Ideally on the same
> machine.)  Bind, Unbound, or even systemd-resolved will do and the
> installations of these tools shouldn't be hard for someone who is about
> to set up a mail system.

I agree that this would be a bad idea.  Of course the OP is free to find
someone to implement this for a custom version of Exim, but I would hope
that resolver bypass does not become a supported Exim feature.

All the problems the OP would like to solve are best handled via a dedicated
local resolver.  That resolver can forward queries to some more central
resolver and define stub zones (or an appropriate alternative mechanism) for
whatever RBL domains it would like to bypass the upstream cache.

-- 
Viktor.


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Heiko Schlittermann via Exim-dev 
Hi,

Rob McEwen  (Sa 09 Sep 2017 20:59:01 CEST):
> What I want to accomplish is this: provide subscribers to the invaluement
> anti-spam blacklist... who use exim... the ability to have their DNS queries
> to DNSBLs... come directly from Exim, skipping the normal DNS resolver. (and
> other DNSBLs could benefit from this too!)
> 
> The way this would work... is that Exim would do a normal NS lookup on the
> host name at the root of a DNSBL (eg "zen.spamhaus.org", for example), then
> collect IP address(es) that those authoritative name servers resolve to, and
> then do the actual DNSBL lookup *directly* on that DNSBL's authoritative
> servers, skipping the regular caching DNS server "middleman".

I'm not sure if I got it. You want to re-invent a caching name service
inside Exim? What's wrong with installing a caching (and validating)
resolver next to the host Exim is running on? (Ideally on the same
machine.)  Bind, Unbound, or even systemd-resolved will do and the
installations of these tools shouldn't be hard for someone who is about
to set up a mail system.

> (Ideally, Exim would internally cache the answer for the NS lookups... so
> that it wouldn't have to do this NS lookup with every single DNSBL lookup.
> But technically, that part is a bit more exotic.)

And even w/o a caching resolver next to Exim the DNS resolver
infrastructure should be mature enough to handle the "non-caching"
behaviour of Exim.

As already explained by others on this list, Exim's architecture
doesn't allow for an easy caching implementation.

> PS - This can be beneficial for other uses besides my "invaluement"
…
> servers - and it becomes a hassle for them to set up their own DNS resolver
> and/or the server provider or datacenter constantly overwrites their DNS
> settings, forcing them back to Google (etc). Some of these organizations

I can imagine a global Exim option to override the name server addresses 
provided
by the resolver library (if there isn't such option already). This would
solve the problem with overridden resolver configuration files
(resolv.conf).

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: PGP signature
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Lena--- via Exim-dev 
> to forward most traffic
> to the main resolver and handle the special zones accordingly.

I already had BIND running in my VPS as the authoritative for my zones.
I modified its config (and deleted /etc/resolv.conf):

options {
directory   "/etc/namedb";
pid-file"/var/run/named/pid";
dump-file   "/var/dump/named_dump.db";
statistics-file "/var/stats/named.stats";

forward only;
forwarders { 8.8.8.8; 8.8.4.4; };
also-notify { 193.201.116.2; };
allow-recursion { localhost; 127.0.0.1; 212.109.198.40; };
max-cache-size 1M;
};
zone "." { type hint; file "named.root"; };

zone "lena.kiev.ua" {type master; file "/etc/namedb/master/lena.kiev.ua";};
... other my zones ...

zone "dnswl.org" { type forward; forwarders { }; };
zone "spamhaus.org" { type forward; forwarders { }; };
zone "spamrats.com" { type forward; forwarders { }; };
// dnswl, spamhaus don't reply via 8.8.8.8
/* If no forwarders statement is present or an empty list is provided then
   no forwarding will be done for the domain canceling the effects of any
   forwarders in the options clause. */

// these reply via 8.8.8.8 though have limits in usage policy:
zone "hostkarma.junkemailfilter.com" { type forward; forwarders { }; };
zone "sorbs.net" { type forward; forwarders { }; };
zone "bl.mailspike.net" { type forward; forwarders { }; };
zone "surbl.org" { type forward; forwarders { }; };


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Phil Pennock 
On 2017-09-11 at 12:25 -0400, Rob McEwen wrote:
> (1) I noticed the following (patch? or fork?):
> "DNS for exim.org" by Nigel Metheringham
> ...could that be related or similar to anything I'm trying to accomplish?

Entirely different repository.  That's for configuring the zonefile for
exim.org itself, back when it was Lua-based and using
GitHub/webhooks/LuaDNS before we switched to something which would
support DNSSEC.

> (2) while your suggestions are excellent - they quickly go beyond my
> specific skills/expertise. (I'm not a C programmer - and I'm not even very
> good at C++). Is there any way I could commission or pay someone (as a
> contractor) to implement this? Do you have any suggestions?

Sure, we're open to people posting looking for contractors for Exim
development.

My own schedule is tight for the next couple of months (minimum) so I
can't take this on.

 is where most of these
are listed; the page is open to editing by anyone with a GitHub account,
so these aren't vetted.  I will note that "Heiko Schlittermann" is one
of the two people (together with Jeremy) currently doing most of the
Exim development work, so if I were looking to hire, I'd start there.

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

[exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Rob McEwen 

On 9/11/2017 12:14 PM, Phil Pennock wrote:
We'd take patches, no need to maintain your own version. It's written 
in C.




Phil,

Thanks for your excellent suggestions. Two questions:

(1) I noticed the following (patch? or fork?):
"DNS for exim.org" by Nigel Metheringham
...could that be related or similar to anything I'm trying to accomplish?

(2) while your suggestions are excellent - they quickly go beyond my 
specific skills/expertise. (I'm not a C programmer - and I'm not even 
very good at C++). Is there any way I could commission or pay someone 
(as a contractor) to implement this? Do you have any suggestions?


--
Rob McEwen
http://www.invaluement.com
+1 (478) 475-9032



--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Phil Pennock 
On 2017-09-11 at 10:45 -0400, Rob McEwen wrote:
> (5) It doesn't sound like this is possible in Exim. THEREFORE - What would I
> need to do to build this as a fork (or addon?) to Exim - If I try to provide
> a custom build of Exim - would I need to maintain multiple versions for
> different OS's? Or could I just provide one single download for my
> customers? What would be involved for them to install it - yet without
> overritting their existing Exim settings... as a sort of drop-in upgrade?
> What language is Exim written in?

We'd take patches, no need to maintain your own version.  It's written
in C.

https://git.exim.org/exim.git
https://github.com/Exim/exim/wiki/EximDevelopment

The git repo, for historical reasons to do with combining multiple other
repositories into one at the time of git conversion, is one level
"higher" than the layout seen in release tarballs, thus `src/src/` to
get to the source code.  Alas.  (We should get around to fixing that,
one of these decades).

The core DNS logic is in dns.c and for the dnslists ACL condition, see
verify_check_dnsbl() in verify.c.  That function is responsible for
parsing out the configure syntax (see
)
and delegates work to one_check_dnsbl() defined just above it.

So you'd need a syntax to specify nameservers, unambiguous and distinct
from the existing syntax; see the docs link just above and the following
sections for what syntax is currently handled.  There are other ways of
suppling "connection"-style information used for stuff like database
connections, but nothing really consistent.

You'd need a custom resolver routine which can be given an explicit
nameserver; I don't know of a way to do this with current system
resolver routines.  A while back I wondered about moving away from the
mess that is this underdocumented system API towards something modern,
like getdns.  .  That's probably the sanest
portable route to doing anything other than "system resolver" with DNS,
even though the project is currently only declaring itself
known-to-work-with Linux, FreeBSD, OSX, and MS Windows.

It looks like getdns includes package-conf support, `getdns.pc` is
bundled.  Unfortunately all the current package-conf support is
lookup-specific and in src/scripts/lookups-Makefile.

Ignore the Exim build integration side of things.  If you get code
working which works if you manually slap getdns in CFLAGS in the
Makefile, I can find time to make sure it's feature-guarded and that the
build-system works with it.

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Jeremy Harris 
On 11/09/17 15:45, Rob McEwen wrote:
>  - What
> would I need to do to build this as a fork (or addon?) to Exim

Start with source code and build system from git.exim.org, probably.

> - If I
> try to provide a custom build of Exim - would I need to maintain
> multiple versions for different OS's?

Depends on whether you can write what you want in portable fashion.

> Or could I just provide one single
> download for my customers?

Depends on your customers.

> What would be involved for them to install it
> - yet without overritting their existing Exim settings... as a sort of
> drop-in upgrade?

Depends on your customers and what package management system they use.
This part belongs to the distros, in the Linux segment of the world
(I'm unsure if there's anyone else left now, but if there is then
there is presumably an equivalent layer of organisations).  It is
not something managed by the Exim project.

> What language is Exim written in?

Mostly 'C'.  There's a few ancillary utilities in Perl.

-- 
Luck,
  Jeremy

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

[exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Rob McEwen 

--replying to multiple messages - wrapped up in one reply--

Thanks for all of the suggestions and information! Here are some follow 
up thoughts:


(1) I already totally understand about how/why this can be done via 
pointing the DNS to a locally server, such as BIND - and then  - *if* 
BIND is forwarding all queries to google - there can be a "conditional 
forwarder" to force the BIND to treat specific types of queries 
differently. HOWEVER I always (mistakenly!) thought that this required 
pointing that conditional forwarder to a specific IP - so I didn't know 
that having that part empty - would get BIND to revert THOSE queries 
back to using root hints and then send the query to authoritative 
servers - so that new knowledge is helpful. (Thanks Lena!)


(2) I also understand the benefit of caching to DNSBLs. This isn't quite 
as big of a deal as it seems... because quality DNSBLs only have a TTL 
that lasts for a few minutes, at most! But this is still an issue since 
caching still helps in rapid fire situations - where multiple messages 
are sent in close succession to multiple users. While this is a valid 
point - the problem I'm trying to solve is such a pain sometimes - that 
this lack-of-caching issue can often be a "lesser of evils".


(3) Stepping back a moment to look at "the big picture" - Sadly, it is 
getting to the point where about 95+% of all email - is handled by about 
a dozen large tech companies - and (I'm guessing that...) 50+% is 
handled by about 6 tech companies. (Microsoft, Google, Godaddy, 
Proofpoint, Mimecast, Sophos) Meanwhile, the number of domains handled 
by those who manage their own server - is distinctly LESS than 5% - and 
shrinking. I think this is very bad for the industry. I fear that one 
day - these large providers will get together and implement standards 
that throw the rest of us overboard. Keep in mind that these dozen of so 
companies operate a small percentage of the overall MTAs - but they 
still handle the mail for 95+% of all domains. Part of the reason for 
this - is that running a mail server is getting too complicated! New 
layers of expertise and knowledge keeps getting added to the mix - 
TLS/SSL, DKIM, SPF, dmarc, PTR records, and customers are starting to 
expect near perfect spam filtering these days - often the email admin 
wears 50 hats besides managing the mail server - so this is  partly the 
reason why so many flock to large cloud providers. So, in general, there 
needs to a paradigm shift where providers of various hardware and 
software run-your-own-mail-server packages... to try to make these 
things easier. The problem that I'm having - is when dealing with these 
sys admins who wear those 50 hats... they either don't want to hassle 
with DNS - or they try to modify the DNS and figure out that their 
hoster has that locked down - or their hoster keeps overwriting their 
settings.


(4) Along those lines (what I said in #3) - for smallish operations 
(perhaps fewer than 5000 mailboxes)... losing a little caching 
efficiency is an acceptable tradeoff for being able to provide the 
option to click a checkbox that says "use DNSBL provider's DNS servers 
directly" - then, if there can be added caching that is limited to the 
last few minutes is added - that is even better!


(5) It doesn't sound like this is possible in Exim. THEREFORE - What 
would I need to do to build this as a fork (or addon?) to Exim - If I 
try to provide a custom build of Exim - would I need to maintain 
multiple versions for different OS's? Or could I just provide one single 
download for my customers? What would be involved for them to install it 
- yet without overritting their existing Exim settings... as a sort of 
drop-in upgrade? What language is Exim written in?


(6) Take a look at step 1 of my subscribe page:
https://www.invaluement.com/subscribe/
First, from a marketing standpoint - this is a real buzzkiller! (if a 
less technical manager-type - who just wants less spam at a minimal cost 
- sees this - they get frustrated and click away!) And this is the new 
and improved "simplified" version. NOW - consider that somewhere between 
25-50% of all invaluement trials - even AFTER reading "step 1" - STILL 
get that wrong! I then have to send a follow-up message trying to 
further explain to them how/why their queries can't come via Google or 
OpenDNS servers - how to fix that - and even then, half the time, they 
still can't get it right. And every month, about 2-3% of existing 
customers start suddenly doing this wrong - due to settings reverting 
back to Google or OpenDNS. When any of these situations happens - then 
the queries from Google or OpenDNS are obviously blocked. But the 
problem is worsened by the fact that Google and OpenDNS often do as many 
as a dozen retries, thus clogging up my logs with lots of failed queries 
- and the sum total of all of these can add up to much resource usage. 
Thus, if this "query DNSBL providers' DNS servers directly"

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-11 Thread Jasen Betts 
On 2017-09-09, Rob McEwen  wrote:
> Hi! This is Rob McEwen from the invaluement.com anti-spam blacklist. (I 
> just joined this list.)
>
> I have a feature request for Exim. Or, since Exim is clearly one of the 
> world's most flexible/configurable MTAs, is... what I'm about to 
> describe... already be possible with existing features?
>
> What I want to accomplish is this: provide subscribers to the 
> invaluement anti-spam blacklist... who use exim... the ability to have 
> their DNS queries to DNSBLs... come directly from Exim, skipping the 
> normal DNS resolver. (and other DNSBLs could benefit from this too!)

no, this will hurt the DNSBLs feeding the requests through caching
resolvers helps the DNSBLs by reducing their infrastructure costs.

> The way this would work... is that Exim would do a normal NS lookup on 
> the host name at the root of a DNSBL (eg "zen.spamhaus.org", for 
> example), then collect IP address(es) that those authoritative name 
> servers resolve to, and then do the actual DNSBL lookup *directly* on 
> that DNSBL's authoritative servers, skipping the regular caching DNS 
> server "middleman".

You can get a better result by installing bind or some other caching
resolver locally with only the root domain servers for reference

this way exim only one set of lookups is made to find the IP address
of the blacklist. and only lookup one per ip-address until it the DNS
response TTL expires.

> (Ideally, Exim would internally cache the answer for the NS lookups... 
> so that it wouldn't have to do this NS lookup with every single DNSBL 
> lookup. But technically, that part is a bit more exotic.)

the exim process is epehemeral, internal caching is done, but each 
instance only sees one connection attempt.

> Is there a way to do this already in Exim? If not, does anyone have any 
> suggestions regarding how this might be implemented? 

exim currently uses gethostent() for DNS lookups, you'd need to
replace that with something that can use different configurations 
on a per-call basis.

personally I'd go with bind on the exim server and have the DNSBL serve
results with a TTL of 1 when I want them to be re-queried every time.

-- 
This email has not been checked by half-arsed antivirus software 

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

Re: [exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-10 Thread Phil Pennock 
On 2017-09-09 at 14:59 -0400, Rob McEwen wrote:
> I have a feature request for Exim. Or, since Exim is clearly one of the
> world's most flexible/configurable MTAs, is... what I'm about to describe...
> already be possible with existing features?

Not quite possible, but could be a fairly small modification to the
dnsdb lookup type.  But a bad idea because of Exim's architecture.

> (Ideally, Exim would internally cache the answer for the NS lookups... so
> that it wouldn't have to do this NS lookup with every single DNSBL lookup.
> But technically, that part is a bit more exotic.)

Exim forks a new process for every accepted connection and every
outbound delivery.  Exim's lookup system isn't really designed for any
kind of programmable write-back caching (redis/whatever).

For the short lifetime of the process, lookup results are remembered in
an LRU, but that's about it.

Further, Exim is currently using OS native libresolv-style interfaces
and knows nothing except how to issue a query using those.

The normal approach for postmasters running mail-servers is to install a
dedicated DNS resolver either on the box, or serving the boxes, so that
all the DNS traffic from the mail-system doesn't push out of cache the
lookups which outbound query folks care about.

Then if a particular zone needs to be handled specially, that can be
configured in the DNS resolver's setup accordingly.  Encapsulate the
special DNS logic in the DNS resolver.

If someone really wants something special but can't run a separate DNS
server (personal development box in colocation somewhere) then I'd look
at using pf/iptables user-based filtering to divert DNS traffic to the
resolver on port 53 to port 63 instead and run a second DNS resolver on
port 63, which is configured with a tiny cache, to forward most traffic
to the main resolver and handle the special zones accordingly.

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##

[exim-dev] feature request for exim: query DNSBL providers' DNS servers directly

2017-09-10 Thread Rob McEwen 
Hi! This is Rob McEwen from the invaluement.com anti-spam blacklist. (I 
just joined this list.)


I have a feature request for Exim. Or, since Exim is clearly one of the 
world's most flexible/configurable MTAs, is... what I'm about to 
describe... already be possible with existing features?


What I want to accomplish is this: provide subscribers to the 
invaluement anti-spam blacklist... who use exim... the ability to have 
their DNS queries to DNSBLs... come directly from Exim, skipping the 
normal DNS resolver. (and other DNSBLs could benefit from this too!)


The way this would work... is that Exim would do a normal NS lookup on 
the host name at the root of a DNSBL (eg "zen.spamhaus.org", for 
example), then collect IP address(es) that those authoritative name 
servers resolve to, and then do the actual DNSBL lookup *directly* on 
that DNSBL's authoritative servers, skipping the regular caching DNS 
server "middleman".


(Ideally, Exim would internally cache the answer for the NS lookups... 
so that it wouldn't have to do this NS lookup with every single DNSBL 
lookup. But technically, that part is a bit more exotic.)


Is there a way to do this already in Exim? If not, does anyone have any 
suggestions regarding how this might be implemented? For example, if it 
can't be done with Exim's current features, is there some kind of way 
that I could write a custom plugin for Exim that could possibly 
accomplish this? (if that is the best option, please point me in the 
right direction for investigating way to write Exim plugins)


Thanks!

PS - This can be beneficial for other uses besides my "invaluement" 
commercial anti-spam blacklist. For example, sometimes, those who host 
their own mail servers... are on remote systems that default to Google's 
DNS servers - and it becomes a hassle for them to set up their own DNS 
resolver and/or the server provider or datacenter constantly overwrites 
their DNS settings, forcing them back to Google (etc). Some of these 
organizations have subscriptions to Spamhaus, which then become useless 
when they have trouble reliably running their own DNS server... then 
their DNSBL queries for Spamhaus (and others) are sometimes blocked. For 
this reason, Kerio Connect has a feature called "use DNSBL provider's 
server directly". I'm now working towards trying to find ways to 
implement this same technology into other MTAs and spam filters. (thus 
this post!)


--
Rob McEwen
http://www.invaluement.com
+1 (478) 475-9032



--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim 
details at http://www.exim.org/ ##