Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-04 14:38, admin--- wrote: > https://bugs.exim.org/show_bug.cgi?id=2369 > > --- Comment #1 from Jeremy Harris --- > My preferred dev platform, currently f28, does not - and building > libcork fails, on an initial attempt. How does it fail for you? It works for me. $ autoreconf -i $ ./configure $ ./make -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-04 14:38, admin--- wrote: > https://bugs.exim.org/show_bug.cgi?id=2369 > > --- Comment #1 from Jeremy Harris --- I much prefer email communication to bugzilla (perhaps not a surprise), so replying here. I hope that's OK; if we reach any conclusions, I'll summarize in the bug log. > Could you craft up some text for doc/doc-txt/experimental-spec.txt ? Okay, I will do that. > Can it be dynamically-loaded? If not we'll want to note that in > EDITME. In principle, it can. It's just that the scaffolding setup is so laborious (and unnecessarily so, IMO) that by the time I got it to link statically I had enough. I plan to start a thread about all that soon. > The feature-macro is missing (see src/macro_predef.c); since this is purely a > lookup, _HAVE_LOOKUP_CORKIPSET seems appropriate. Note that in doc > too. Hmm ... see above, but I'll look into that. > The coding style ought to be massaged towards project policy; it's short > enough > I can do that. Is there a document about the preferred project style? Or better, a tool implementing it, such as a bunch of clang-format settings? > Are the required libraries supported by any common distros? Yes, debian and devuan, at least. These are the two platforms I use for my server type hosts. > My preferred dev platform, currently f28, does not - and building > libcork fails, on an initial attempt. Yeah, I use f28 on the desktop, too. I got the libs from github but have not tried to build them there, yet. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
Turns out the underlying library wants IPv6 addresses colon separated; but in contexts where a host address is being tested for list membership, exim passes it to the lookup as dot-separated. This is of course documented in the Spec, section 10.12. Since IP addresses are the only things worth testing by this lookup, maybe I should internally translate them from dot to colon, before I pass them to libcork? -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] [Bug 2369] New feature: single-key lookup type based on libcorkipset - https://github.com/dcreager/libcorkipset
On 2019-02-18 19:35, Jasen Betts wrote: > > I'd had the loop fixed in my local git but the commit got lost when > > I made a patch :-( Really, it would be much easier for me to just > > work with git. Why is that option not there? > > As I understand it (being an outsider); the exim project is not hosted > on github: it is mirrored on github. So the exim github is effectively > read only. I've always known that github is a read only mirror. But there is git.exim.org, and I set my "origin" to that. > git diff master > thing.patch I did something very similar to generate my patch, but it turned out the upstream branch was not in the right relation to mine. The branch I was working on was based on my local branch from which I build my running server. So first I tried to rebase, and when that proved impossibly hard I had to edit the patch by hand, and that's where the mistake happened. Of course you can say it is all my fault for not starting the development from master in the first place, but it would be much harder to make such a mistake of there was an official development branch for the feature from/to which I could pull/push. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[exim-dev] Test suite unusable?
eximtest@devuan-205f:~/exim/test$ ./runtest 2850 Exim tester exim-4.92-61-g859c1166-dirty Exim binary is `../src/build-Linux-aarch64/exim' You need to have sudo access to root to run these tests. Checking ... Test for sudo OK Probing with config file: /home/itz/exim/test/test-config *** Version mismatch *** Exim binary: 4_92-61-859c1166-XX *** Git: 4.92-61-859c1166-XX aux-fixed file is world-writeable; best to strip them all, recursively the 2nd complaint is probably not a showstopper, but the only world-writeable files are as follows: eximtest@devuan-205f:~/exim/test$ ls -lRa aux-fixed/ | grep 'w[-xsS] ' lrwxrwxrwx 1 itz eximtest 15 Feb 23 13:33 9ec80de3.0 -> ../../CA/CA.pem lrwxrwxrwx 1 itz eximtest 19 Feb 23 13:33 d89e5358.0 -> ../../CA/Signer.pem Sadly I am pretty close to the point of giving up on contributing code to Exim, because it is so laborious. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Test suite unusable?
On 2019-02-23 23:29, Jeremy Harris wrote: > On 23/02/2019 22:44, Ian Zimmerman via Exim-dev wrote: > > the 2nd complaint is probably not a showstopper, but the only > > world-writeable files are as follows: > > > > eximtest@devuan-205f:~/exim/test$ ls -lRa aux-fixed/ | grep 'w[-xsS] ' > > lrwxrwxrwx 1 itz eximtest 15 Feb 23 13:33 9ec80de3.0 -> ../../CA/CA.pem > > lrwxrwxrwx 1 itz eximtest 19 Feb 23 13:33 d89e5358.0 -> > > ../../CA/Signer.pem > > What does "ls -l aux-fixed/0037*" tell you? devuan-205f!2 test$ ls -l aux-fixed/0037* -rw-rw-r-- 1 itz eximtest 1660 Feb 23 13:33 aux-fixed/0037.f-1 -rw-rw-r-- 1 itz eximtest 92 Feb 23 13:33 aux-fixed/0037.f-2 -rw-rw-r-- 1 itz eximtest 92 Feb 23 13:33 aux-fixed/0037.f-3 -rw-rw-r-- 1 itz eximtest 351 Feb 23 13:33 aux-fixed/0037.f-4 -rw-rw-r-- 1 itz eximtest 92 Feb 23 13:33 aux-fixed/0037.f-5 -rw-rw-r-- 1 itz eximtest 457 Feb 23 13:33 aux-fixed/0037.f-6 -rw-rw-r-- 1 itz eximtest 28 Feb 23 13:33 aux-fixed/0037.f-7 -rw-rw-r-- 1 itz eximtest 85 Feb 23 13:33 aux-fixed/0037.f-8 -rw-rw-r-- 1 itz eximtest 21 Feb 23 13:33 aux-fixed/0037.f-9 -rw-rw-r-- 1 itz eximtest 186 Feb 23 13:33 aux-fixed/0037.msg FWIW, I was running not as "itz", but as "eximtest" - the ID whose primary group is "eximtest". -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Test suite unusable?
On 2019-02-24 01:57, Jeremy Harris wrote: > On 24/02/2019 01:16, Ian Zimmerman via Exim-dev wrote: > > devuan-205f!2 test$ ls -l aux-fixed/0037* > > -rw-rw-r-- 1 itz eximtest 1660 Feb 23 13:33 aux-fixed/0037.f-1 > > Thanks! You found a testsuite bug; the message has been > wrong since I first wrote it (and you're the first to > notice!). It should say "group". You presumably are > not running the README guidance of an 022 umask; > fix by "chmod -R g-w aux-fixed". That answers the riddle, but it doesn't help me much, since the very purpose of this setup was to be able to build as myself and then run the tests as the ad-hoc eximtest user who has few other powers. Do you have a suggestion for the version mismatch? -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-10 23:03, Jeremy Harris wrote: > If you can reliably detect the ipv6-ness, yes, that sounds like the > minimally intrusive way. How are the IPv4-mapped IPv6 addresses written in Exim? The straight translation to dots instead of colons would be ambiguous, wouldn't it? Say, ...1.2.3.4 could mean either the mapped address or the normal IPv6 address 0:0:0::0001:0002:0003:0004 . -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-24 15:01, Jeremy Harris wrote: > On 24/02/2019 07:47, Ian Zimmerman via Exim-dev wrote: > > > > How are the IPv4-mapped IPv6 addresses written in Exim? > > > > The straight translation to dots instead of colons would be ambiguous, > > wouldn't it? Say, ...1.2.3.4 could mean either the mapped address > > or the normal IPv6 address 0:0:0::0001:0002:0003:0004 . > > You can write them either way; both are acceptable and the number-base > of the dotted-portion gets translated ( .234 does _not_ become :0234 ). But that was not the question I asked. It's not about one address having multiple representations; it's that -- after a blind colon-dot transliteration -- the same string representation may denote multiple addresses. > But that's ipv6 addresses given to exim, not how they are given to > the place you're coding. I think you need to check, given that a > colon-to-dot translation has been done by the time it hits you. Of course, that is why I'm asking. > I'd expect conversion to unabbreviated form to have been done too. Do you mean I can expect an IPv6 address (mapped or not) to have exactly 7 separators, whatever these might be? If yes, there is no ambiguity. I think I must keep in mind 2 cases: 1. Addresses coming from exim itself, for example $sender_host_address. Unfortunately the spec doesn't say what format this is in (if it is IPv6). Can you enlighten me about this? 2. Addresses explicitly written into the configuration by user. This one is about what I can require from users. I need to document that in the experimental documentation file. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-24 19:51, Jeremy Harris wrote:
> On 24/02/2019 19:17, Jeremy Harris via Exim-dev wrote:
> > I don't know if a lookup done via the list-syntax
> > "hosts = corkipset:/filename"
> > will be different. Probably it will, sigh.
>
> Dots, and always seven.
>
> I tested by adding
>
>"pipeline_advertised_hosts = net-corkipset;DIR/aux-var/TEST.ipset"
>
> to the config, some "exim -bh" to the script and some
> debug_printf() in the code. I didn't look at actual
> results of the lookups though.
Thanks for the help (as always).
I thought I would start by looking what iplsearch does, because it is
the closest lookup to what I'm dreaming about, semantically at least.
And bingo, it seems to not work as documented:
devuan-205f!33 exim$ ls `pwd`/strange-iplist
/home/itz/exim/strange-iplist
devuan-205f!34 exim$ ls `pwd`/exim.conf
/home/itz/exim/exim.conf
devuan-205f!35 exim$ cat strange-iplist
"::::::0102:0203" : data
":::102:203" : also_data
1.2.3.4 : more_data
devuan-205f!37 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{1.2.3.4}iplsearch{/home/itz/exim/strange-iplist}{$value}}'
more_data
devuan-205f!40 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{1.2.3.4}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
good
devuan-205f!38 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{::::::0102:0203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
bad
devuan-205f!39 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{:::102:203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
bad
devuan-205f!41 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{......0102.0203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
Failed: lookup of "......0102.0203" gave DEFER:
"......0102.0203" is not a valid iplsearch key (an IP
address, with optional CIDR mask, is wanted): in a host list, use net-iplsearch
as the search type
So, maybe before trying to merge corkipset, I should focus on fixing
iplsearch :-o
--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim
details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-25 18:19, Ian Zimmerman wrote:
> I thought I would start by looking what iplsearch does, because it is
> the closest lookup to what I'm dreaming about, semantically at least.
> And bingo, it seems to not work as documented:
>
> devuan-205f!33 exim$ ls `pwd`/strange-iplist
> /home/itz/exim/strange-iplist
> devuan-205f!34 exim$ ls `pwd`/exim.conf
> /home/itz/exim/exim.conf
> devuan-205f!35 exim$ cat strange-iplist
> "::::::0102:0203" : data
> ":::102:203" : also_data
> 1.2.3.4 : more_data
> devuan-205f!37 exim$ ./src/build-Linux-aarch64/exim -C
> /home/itz/exim/exim.conf -be
> '${lookup{1.2.3.4}iplsearch{/home/itz/exim/strange-iplist}{$value}}'
> more_data
> devuan-205f!40 exim$ ./src/build-Linux-aarch64/exim -C
> /home/itz/exim/exim.conf -be
> '${lookup{1.2.3.4}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
> good
> devuan-205f!38 exim$ ./src/build-Linux-aarch64/exim -C
> /home/itz/exim/exim.conf -be
> '${lookup{::::::0102:0203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
> bad
> devuan-205f!39 exim$ ./src/build-Linux-aarch64/exim -C
> /home/itz/exim/exim.conf -be
> '${lookup{:::102:203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
> bad
> devuan-205f!41 exim$ ./src/build-Linux-aarch64/exim -C
> /home/itz/exim/exim.conf -be
> '${lookup{......0102.0203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
> Failed: lookup of "......0102.0203" gave DEFER:
> "......0102.0203" is not a valid iplsearch key (an IP
> address, with optional CIDR mask, is wanted): in a host list, use
> net-iplsearch as the search type
>
> So, maybe before trying to merge corkipset, I should focus on fixing
> iplsearch :-o
And I narrowed it down: when I replace the leading hexuplet with
1234, it succeeds. So, leading zeros seem to be the problem. I also tried
to replace instead with a single 0 (thinking redundant leading zeros
might be it) but that still fails.
Doing this now in test 0002 instead from command line. Significantly,
that test didn't try any address with leading zeros before.
--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim
details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-26 11:21, Jeremy Harris wrote:
> > And bingo, it seems to not work as documented:
devuan-205f!35 exim$ cat strange-iplist
"::::::0102:0203" : data
":::102:203" : also_data
1.2.3.4 : more_data
devuan-205f!38 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{::::::0102:0203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
bad
devuan-205f!39 exim$ ./src/build-Linux-aarch64/exim -C
/home/itz/exim/exim.conf -be
'${lookup{:::102:203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
bad
> Which bit are you saying doesn't match docs?
>
> So far, I see you giving a dot-group-hex ipv6 to an iplsearch
> lookup, which is wrong per docs:
> "key for an iplsearch lookup must be an IP address".
Huh? Both commands 38 and 39 (2nd and 3rd from the end in my up thread
post) try looking up a colon-separated IPv6 address which is present in
the file -- in fact the same address in both the abbreviated and full
forms -- and do not find it.
--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim
details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-26 17:28, Jeremy Harris wrote:
> > devuan-205f!35 exim$ cat strange-iplist
> > "::::::0102:0203" : data
> > ":::102:203" : also_data
> > 1.2.3.4 : more_data
>
>
> > devuan-205f!38 exim$ ./src/build-Linux-aarch64/exim -C
> > /home/itz/exim/exim.conf -be
> > '${lookup{::::::0102:0203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
> > bad
> > devuan-205f!39 exim$ ./src/build-Linux-aarch64/exim -C
> > /home/itz/exim/exim.conf -be
> > '${lookup{:::102:203}iplsearch{/home/itz/exim/strange-iplist}{good}{bad}}'
> > bad
>
> The address being looked up, :::102:203, is an ipv6-mapped ipv4
> and we'd expect it to match 1.2.2.3 in the file. But you only have
> 1.2.3.4
But according to the spec (section 10.11) the mapping is done _when
checking an address for membership in a host list_. I read that (IMO
with justification) as meaning a bare lookup does no such translation,
because none is mentioned in the part of the spec about lookups (that
would be section 9.3 I guess). So at least this is a documentation bug.
The fact that the address I tested was a mapped one was intentional, of
course, because those are the ones presenting potential ambiguity when
expressed in a dot delimited form.
--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim
details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
I am still confused by this paragraph in section 10.12: When an IPv6 address is converted to a string, dots are normally used instead of colons, so that keys in lsearch files need not contain colons (which terminate lsearch keys). This was implemented some time before the ability to quote keys was made available in lsearch files. However, the more recently implemented iplsearch files do require colons in IPv6 keys (notated using the quoting facility) so as to distinguish them from IPv4 keys. For this reason, when the lookup type is iplsearch, IPv6 addresses are converted using colons and not dots. In all cases, full, unabbreviated IPv6 addresses are always used. My naive understanding was that lookups are more primitive than host lists, and exim converts the subject into a string _once_ and then matches it against each list item in turn, passing the converted string to the lookup's *find() entry when an item refers to a lookup. But this paragraph seems to imply that exim does the conversion anew for each lookup item; after all there could be multiple lookups of different kinds in the list, some iplsearch and some plain lsearch. In fact it seems that iplsearch lookups are somewhat special when used in lists. And that is what interests me in this situation, because I want corkipset lookup to share the same specialness. I think the dot translation is an awful hack, and basically I want nothing to do with it ;-) Do you know how (where in the code) iplsearch is made special in this way? -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [exim-dev] Bug 2369: single-key lookup type based on libcorkipset
On 2019-02-26 22:26, Ian Zimmerman wrote: > In fact it seems that iplsearch lookups are somewhat special when used > in lists. And that is what interests me in this situation, because I > want corkipset lookup to share the same specialness. I think the dot > translation is an awful hack, and basically I want nothing to do with it > ;-) > > Do you know how (where in the code) iplsearch is made special in this way? Ok, I found it: it is in verify.c , function check_host(). I can deal with it, but my patch will have to touch (minimally) that code. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet and on broken lists which rewrite From, fetch the TXT record for no-use.mooo.com. -- ## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
