Re: [exim] De-tainting with ${sg} expansion
(Apologies if this doesn't thread properly -- I am writing via email and
reading via Lurker...)
On 27/07/2020 19:45, Jamie Barnes via Exim-users wrote:
>> I've been avoiding check_local_user (since it tries to chdir into home
>> directories that the exim user has no access to), so I don't think I have
>> access to $local_part_data (as nothing populates it).
On 2020-07-28 20:25 +100, Jeremy Harris via Exim-users wrote:
> Not so. Any lookup done by a local_parts= condition on a router also sets it.
That's nice to know, thanks, but I'm not doing any lookup in a `local_parts`
router directive either. I'm literally storing everything that comes in, in a
domain/localpart folder tree.
> You didn't show the routers calling these transports...
> Transports are only run when called by a router...
There were a lot of routers, with a lot of logic in between, and I didn't
expect everyone to attempt to parse it all. In summary, when I was putting it
all together originally, I short-circuited all of that routing logic for mail
sent locally (e.g. by daemons to the root user), so I have this one first...
store_local_user:
driver= accept
condition = ${if bool{$acl_c_islocal}}
transport = local_delivery
log_as_local
no_more
...and then, as a fallback to catch all the places where logic hasn't routed
something, the last router is...
unhandled:
driver= accept
transport = local_unhandled
verify= false
no_more
>> local_delivery:
>> driver= appendfile
>> file = ${if or{{bool{$acl_m_localdiscard}}
>>\
>> {bool{${lookup{$local_part}
>>\
>> lsearch{/etc/passwd}
>>\
>> {no}
>>\
>> {yes}
>>\
>>{/dev/null}
>>\
>>
>> {/var/spool/mail/${sg{$local_part}{BADFILECHARS}{_
>
> The discard invoked by the /dev/null choice could be done by a router
> (redirect, to :blackhole:).
> Doing that would simplify this to
>
> file = /var/spool/mail/${sg{$local_part}{BADFILECHARS}{_}}
>
> ... and then we can detaint easily with a lookup rather than the ${sg}
>
> file = /var/spool/mail/${lookup{$local_part}lsearch,ret=key{/etc/passwd}}
>
> (whether you need any further cleaning depends on your policies on username
> creation).
To be honest, despite the documentation, I've never really understood the
redirect router. If there is any reasonable gain to be had from switching this
logic to redirect, I might re-evaluate it.
> Depending on the router calling this transport, you might be able to do the
> lookup there, populating $local_part_data for use here.
>
>> user = ${if or{{bool{$acl_m_localdiscard}}
>>\
>> {eqi {$local_part}{root}}
>>\
>> {bool{${lookup{$local_part}
>>\
>>
>> lsearch{/etc/passwd} \
>> {no}
>>\
>> {yes}
>>\
>>{mail}
>>\
>>{$local_part}}
>
> This looks like it uses the same lookup or value as the file= option.
It is. The irony is that local mail is the least of my worries here, but Exim
wasn't originally the server's default system mail handler and this was mostly
an attempt to get that facility back when we switched away from Postfix. It's
just that the local mail was clogging up the queue (defers, because of the
Tainted errors).
> local_unhandled:
> driver= appendfile
> create_directory = yes
> directory = /var/spool/exim/unhandled/\
> ${sg{$domain}{BADFILECHARS}{_}}/\
> ${sg{$local_part}{BADFILECHARS}{_}}/\
> $tod_logfile
>
> This transport is much harder to deal with. Currently you have to use
> one of the gross hacks presented as a get-out-of-jail-free, to
> obtain untainted strings for that path. I hate them, but there were
> always going to be ways for shooting oneself in the foot.
>
> I fully expect them to be cargo-culted, probably via serverfault.
>
> Hint: Use a lookup which matches anything, and returns the key.
I missed the fact that Exim-users is moderated, so after
Re: [exim] De-tainting with ${sg} expansion
On 27/07/2020 19:45, Jamie Barnes via Exim-users wrote:> I've been avoiding
check_local_user (since it tries to chdir into home directories that the exim
user has no access to), so I don't think I have access to $local_part_data (as
nothing populates it).
Not so. Any lookup done by a local_parts= condition on a router
also sets it.
You didn't show the routers calling these transports.
> in need of an alternative for the following two transports, where I need to
> be able to store *any* received mail (not handled by earlier
> routers/transports)
Transports are only run when called by a router...
> local_delivery:
> driver= appendfile
> file = ${if or{{bool{$acl_m_localdiscard}}
> \
> {bool{${lookup{$local_part}
> \
> lsearch{/etc/passwd}
> \
> {no}
> \
> {yes}
> \
>{/dev/null}
> \
>
> {/var/spool/mail/${sg{$local_part}{BADFILECHARS}{_
The discard invoked by the /dev/null choice could be done by
a router (redirect, to :blackhole:). Doing that would simplify this to
file = /var/spool/mail/${sg{$local_part}{BADFILECHARS}{_}}
... and then we can detaint easily with a lookup rather than the ${sg}
file = /var/spool/mail/${lookup{$local_part}lsearch,ret=key{/etc/passwd}}
(whether you need any further cleaning depends on your policies on username
creation).
Depending on the router calling this transport, you might be able to do the
lookup there, populating $local_part_data for use here.
> user = ${if or{{bool{$acl_m_localdiscard}}
> \
> {eqi {$local_part}{root}}
> \
> {bool{${lookup{$local_part}
> \
> lsearch{/etc/passwd}
> \
> {no}
> \
> {yes}
> \
>{mail}
> \
>{$local_part}}
This looks like it uses the same lookup or value as the file= option.
> local_unhandled:
> driver= appendfile
> create_directory = yes
> directory = /var/spool/exim/unhandled/\
> ${sg{$domain}{BADFILECHARS}{_}}/\
> ${sg{$local_part}{BADFILECHARS}{_}}/\
> $tod_logfile
This transport is much harder to deal with. Currently you have to use
one of the gross hacks presented as a get-out-of-jail-free, to
obtain untainted strings for that path. I hate them, but there were
always going to be ways for shooting oneself in the foot.
I fully expect them to be cargo-culted, probably via serverfault.
Hint: Use a lookup which matches anything, and returns the key.
I'm not happy with the situation either - but I've not been able to
think of a principled way of dealing with it. I do not regard the
above hack as principled because there are zero limits on it.
> Is ${sg} not a suitable expansion to de-taint $local_part or $domain?
No - because there are no limits on it, and I see no way for Exim
to interpret the replacement pattern in a $(sg) to infer useful limits.
Possibly some form of operator which had the semantics of
"anything below this given path is fair game" would work. But it
could not be a general de-tainting string expansion, because taint
not only prevents use as a file path but also prevents further
expansion.
--
Cheers,
Jeremy
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
On 6/29/20 12:18 PM, Kurt Jaeger via Exim-users wrote:
> One thing I'll test is if we hand values over to perl, maybe
> we'll get back untainted value...
>
> Or did me beat someone to that already ? 8-}
I did not test that, I would imagine that should work because how would
it really know what return values you are sending back.
I know that using sg{} or {if match {} {} {}} does not work, string
expansion fails...
Even this fails...
${if match {$local_part}{.*sms[\-\+]([a-z0-9]+).*}{$1}{}}
With expansion failure due to tainted... I'm clearly just pulling how
known safe data, so it should be considered de-tainted
There is literally no difference vs doing some fake lookup...
--
inoc.net!rblayzor
XMPP: rblayzor.AT.inoc.net
PGP: https://pgp.inoc.net/rblayzor/
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
Hi!
> That fact that string sub-sitution and matching parts don't even work
> now is a real problem...
>
> data = ${expand:"|/command -c ${if match
> {$local_part}{.*foo[\-\+]([a-z0-9]+).*}{$1}{}}"}
One thing I'll test is if we hand values over to perl, maybe
we'll get back untainted value...
Or did me beat someone to that already ? 8-}
--
p...@opsec.eu+49 171 3101372Now what ?
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
That fact that string sub-sitution and matching parts don't even work
now is a real problem...
data = ${expand:"|/command -c ${if match
{$local_part}{.*foo[\-\+]([a-z0-9]+).*}{$1}{}}"}
Won't even work because matching only numbers and letters is still
considered "tainted".
Forcing everything to a static lookup is a bit harsh for a dot release
that snuck under the radar and broke everything
There should of been ample deprecation notice with a default to "off"
before this was forced out.
On 6/26/20 5:54 PM, Sebastian Nielsen via Exim-users wrote:
> is NOT expanded, meaning only character that needs to
> be escaped in that circumstance is { and }.
>
> Example:
> ${filter{$phonenumber}{0123456789}}
>
> Would simply strip all characters except 0-9 from $phonenumber and return in
> untainted form, so if $phonenumber contains "hey0do56" then
> ${filter{$phonenumber}{0123456789}} will return 056
--
inoc.net!rblayzor
XMPP: rblayzor.AT.inoc.net
PGP: https://pgp.inoc.net/rblayzor/
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
I agree about this, but a more simpler interface would be a multiuse
interface that suits most usages of user-supplied data.
And that is to give a very simple "filter" interface that will filter out
all invalid characters and also untaint the data.
Like:
${filter{$data_to_be_filtered}{}}
is NOT expanded, meaning only character that needs to
be escaped in that circumstance is { and }.
Example:
${filter{$phonenumber}{0123456789}}
Would simply strip all characters except 0-9 from $phonenumber and return in
untainted form, so if $phonenumber contains "hey0do56" then
${filter{$phonenumber}{0123456789}} will return 056
Of course, the function ${filter must then be built in such a way its 100%
safe to use, basically theres nothing a untrusted user could do to cause
${filter to return any character outside of the administrator-configured
filter list.
Of course you could supply a "unsafe" list of characters, but thats the
system administrator's responsibility.
So instead of separate functions check_file_path or check_sql_data etc, its
better to have one function, that can be administrator-configured to allow
arbitary filtering, which also could be used in other situations where you
want to filter user-supplied data from unwanted characters, even for
functions that do allow tainted data.
And the best with such ${filter function is that it will never throw any
exceptions or fail, it will just filter out whatever danger is supplied. Of
course, if admin wants it to throw exception in case of unpermitted
characters, its easy for system administrator to do a string compare before
and after the filter function, and err out if they don't match (meaning,
something got filtered out).
And in the rare cases you want to filter out certain character sequences,
its easy to, before the filter function, do a sg{} and replace all unwanted
sequences with a character outside the permitted space, and then run the
filter function.
-Ursprungligt meddelande-
Från: Evgeniy Berdnikov via Exim-users
Skickat: den 26 juni 2020 23:12
Till: exim-users@exim.org
Ämne: Re: [exim] de-tainting
Hello.
On Thu, Jun 25, 2020 at 09:16:59PM +0100, Jeremy Harris via Exim-users
wrote:
> On 25/06/2020 20:50, Evgeniy Berdnikov via Exim-users wrote:
> > at least in statement "In all other situations, this variable
> > expands to nothing", because it may be filled if no lookup is done.
>
> Yes, that is no longer true. See the sections starting
>
> http://exim.org/exim-html-current/doc/html/spec_html/ch-domain_host_ad
> dress_and_local_part_lists.html#SECTlistresults
Referenced text have broken record about 'domains' match, I expect some
words about $domain_data variable. Hope you'll supply them.
And I have question about user interface, which becomes very obscure and
inconvenient. All these innovations with tainting bring problems. In some
cases they may be "resolved" by lookups. Obviously, the idea is to restrict
file name patterns to some predefined set of strings, stored somewhere in
database, in hope that this set would be less risky then data from the
wild. Well, it seems good on first glance. But many users want to use file
system as native database with data-driven records, i.e. with file names
constructed from $local_part, $domain, $sender_host and other variables
that can't be predicted. They can't be expressed as a set of predefined
strings. Is there any effective solution for this case?
I can propose awful "untainting" algorithm: 1. construct a tainted string,
2. put it into database, 3. make lookup to get back "untainted"
data. Any protection against "../" inside, or against SQL injection will
be dropped, but it allows construction of arbitrary untainted strings with
minimum efforts. And I'm sure that samples of such "untainting functions"
became available in internet if Exim's interface continue to evolve in
direction of obscurity and inconveniece.
In my opinion, in order to give an effective tool for data protection, one
should supply simple and copletely "intuitive" interface. For files it may
be a function, say, ${check_file_path:...}, which checks its argument for
dangerous patterns (such as "../") and specials, then returns untainted
copy of string if everything is OK, or throws an exception.
For SQL such function may be ${check_sql_data:...}. Moreover, it seems
better to add such functionality to existing quoting functions:
${qoute_ldap:...}, ${quote_mysql:...}, etc.
Jeremy, I kindly propose to think about movement in this direction.
You have done a really great work, but for the present its result is too
complex and too restrictive for use. If you don't supply simple and
convenient interface, Exim would lose its main advantage, power of built-in
scripting language, and inavitably lose its users. IMHO.
--
Eugene Berdnikov
--
## List detail
Re: [exim] de-tainting
Hello.
On Thu, Jun 25, 2020 at 09:16:59PM +0100, Jeremy Harris via Exim-users wrote:
> On 25/06/2020 20:50, Evgeniy Berdnikov via Exim-users wrote:
> > at least in statement "In all other situations, this variable expands
> > to nothing", because it may be filled if no lookup is done.
>
> Yes, that is no longer true. See the sections starting
>
> http://exim.org/exim-html-current/doc/html/spec_html/ch-domain_host_address_and_local_part_lists.html#SECTlistresults
Referenced text have broken record about 'domains' match, I expect some
words about $domain_data variable. Hope you'll supply them.
And I have question about user interface, which becomes very obscure and
inconvenient. All these innovations with tainting bring problems. In some
cases they may be "resolved" by lookups. Obviously, the idea is to restrict
file name patterns to some predefined set of strings, stored somewhere
in database, in hope that this set would be less risky then data from the
wild. Well, it seems good on first glance. But many users want to use
file system as native database with data-driven records, i.e. with
file names constructed from $local_part, $domain, $sender_host and other
variables that can't be predicted. They can't be expressed as a set of
predefined strings. Is there any effective solution for this case?
I can propose awful "untainting" algorithm: 1. construct a tainted
string, 2. put it into database, 3. make lookup to get back "untainted"
data. Any protection against "../" inside, or against SQL injection will
be dropped, but it allows construction of arbitrary untainted strings with
minimum efforts. And I'm sure that samples of such "untainting functions"
became available in internet if Exim's interface continue to evolve in
direction of obscurity and inconveniece.
In my opinion, in order to give an effective tool for data protection,
one should supply simple and copletely "intuitive" interface. For files
it may be a function, say, ${check_file_path:...}, which checks its
argument for dangerous patterns (such as "../") and specials, then returns
untainted copy of string if everything is OK, or throws an exception.
For SQL such function may be ${check_sql_data:...}. Moreover, it seems
better to add such functionality to existing quoting functions:
${qoute_ldap:...}, ${quote_mysql:...}, etc.
Jeremy, I kindly propose to think about movement in this direction.
You have done a really great work, but for the present its result is
too complex and too restrictive for use. If you don't supply simple
and convenient interface, Exim would lose its main advantage, power of
built-in scripting language, and inavitably lose its users. IMHO.
--
Eugene Berdnikov
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
On 25/06/2020 20:50, Evgeniy Berdnikov via Exim-users wrote: > at least in statement "In all other situations, this variable expands > to nothing", because it may be filled if no lookup is done. Yes, that is no longer true. See the sections starting http://exim.org/exim-html-current/doc/html/spec_html/ch-domain_host_address_and_local_part_lists.html#SECTlistresults I'll remove that misleading sentence. -- Cheers, Jeremy -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
On Thu, Jun 25, 2020 at 04:29:21PM +0100, Jeremy Harris via Exim-users wrote:
> You had "xxample.ru:" as your matching line. Key, and zero-length
> data.
Well. After reading docs several times and a series of experiments
with ACLs I came to the following conclusions:
1. Variable $domain_data is set when condition "domains = ..." is satisfied,
regardless of presense/absense of lookups. Particulary, condition
"domains = $domain" is always valid in RCPT ACL and set $domain_data.
2. Variable $domain_data is tainted if right side of "domains = ..."
ACL condition is tainted.
These contradicts to documentation
| $domain_data is also set when the domains condition in an ACL matches
| a domain by means of a lookup. The data read by the lookup is available
| during the rest of the ACL statement. In all other situations,
| this variable expands to nothing.
at least in statement "In all other situations, this variable expands
to nothing", because it may be filled if no lookup is done.
Moreover, if right side of "domains = ..." is a string expansion which
combines lookup data with other data, then $domain_data may be different
from lookup data.
Below is testcase.
--
ACL
test_domain_data:
warn
logwrite= before test domain=\"$domain\" domain_data=\"$domain_data\"
domains = $domain
logwrite= after test domain=\"$domain\" domain_data=\"$domain_data\"
condition = ${if !eq{${lookup {$domain_data} lsearch \
{/etc/exim4/cust.d/$domain_data}}}{}}
debug output:
>>> check logwrite = before test domain=\"$domain\"
>>> domain_data=\"$domain_data\"
>>>= before test domain="xxample.ru" domain_data=""
LOG: 459 [323342] before test domain="xxample.ru" domain_data=""
>>> check domains = $domain
>>> xxample.ru in "xxample.ru"? yes (matched "xxample.ru")
>>> check logwrite = after test domain=\"$domain\" domain_data=\"$domain_data\"
>>>= after test domain="xxample.ru" domain_data="xxample.ru"
LOG: 459 [323342] after test domain="xxample.ru" domain_data="xxample.ru"
LOG: 459 [323342] Tainted filename for search: '/etc/exim4/cust.d/xxample.ru'
>>> warn: condition test error in ACL "test_domain_data"
--
Test was run on Debian, package exim4-daemon-light 4.94-4 for amd64.
This testcase also shows that $domain_data is NOT always untainted.
In my opinion, documentation for $domain_data and $local_part_data
is misleading. It leads to illusion that those variables are filled
on lookup and are bound to lookup data, but they are filled on
condition match and are bound to condition expression.
PS. All written here is for ACL, I did not test routers and trasports yet.
--
Eugene Berdnikov
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
On 2020-06-25 Evgeniy Berdnikov via Exim-users wrote: [...] > I run "Exim version 4.94 #2 built 19-Jun-2020 08:31:26" from Debian. Hello, Judging from the build date this should be 4.94-3, which is exim-4.94+fixes up to and including ecf1e77accda6355ebb745a0a03e97ba7eb298b2 [Taint: fix verify. Bug 2598]. cu Andreas PS: For distro packages something like the intersting lines from dpkg -l exim\* would be more helpful than the version string since it allows checking what patches were included. -- `What a good friend you are to him, Dr. Maturin. His other friends are so grateful to you.' `I sew his ears on from time to time, sure' -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] de-tainting
On 25/06/2020 16:08, Evgeniy Berdnikov via Exim-users wrote: processing "warn" (/var/lib/exim4/config.autogenerated 485) check acl = test_domain_data using ACL "test_domain_data" processing "warn" (/var/lib/exim4/config.autogenerated 490) check logwrite = before lookup domain=\"$domain\" domain_data=\"$domain_data\" = before lookup domain="xxample.ru" domain_data="" > LOG: 287 [313687] before lookup domain="xxample.ru" domain_data="" check domains = lsearch;/etc/exim4/cust.d/domains.list xxample.ru in "lsearch;/etc/exim4/cust.d/domains.list"? yes (matched "lsearch;/etc/exim4/cust.d/domains.list") check logwrite = after lookup domain=\"$domain\" domain_data=\"$domain_data\" = after lookup domain="xxample.ru" domain_data="" http://exim.org/exim-html-current/doc/html/spec_html/ch-file_and_database_lookups.html#SECTsinglekeylookups lsearch: ... line beginning with the search key, terminated by a colon ... The remainder of the line, with leading and trailing white space removed, is the data You had "xxample.ru:" as your matching line. Key, and zero-length data. -- Cheers, Jeremy -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] De-tainting
Ken,
I already started to prepare a section about the motivation of tainting
and about how to de-taint.
Maybe I can share it before we include it into the official docs.
(As it keeps biting me too ;)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
signature.asc
Description: PGP signature
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: [exim] De-tainting
Hi! > The concept "de-tainting" appears in the index, but not in the manual. Yes, but: Jeremy can't do all the heavy lifting all by himself. We need to help him. Write docs etc. -- p...@opsec.eu+49 171 3101372Now what ? -- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
