[pfx-dev] Re: dict_mongodb (projections)
>> We probably don't need to go as far as parsing the JSON query to ensure
>> that '%x' substitutions happen only in values and not in keys...
>
> I think it would be preferable to do this, as it catches human error that
> would result in an insecure system. One just needs to ensure that keys
> keys never have a % that is not followed by another %. JSON syntax rules
> mean that a % cannot appear anywhere else.
This is not too hard to do if you guys think it would make for a safer
implementation.
Maybe something like this can be added at line 409 (before query_string
expansion):
bson_iter_t iter;
const char *key = NULL;
query = bson_new_from_json(dict_mongodb->query_filter, -1, );
if (!query) {
msg_warn("%s:%s: failed to create a query from '%s' : %s",
dict_mongodb->dict.type, dict_mongodb->dict.name,
vstring_str(dict_mongodb->query_filter), error.message);
DICT_MONGODB_LOOKUP_ERR_RETURN(DICT_ERR_RETRY);
}
if (bson_iter_init(, query)) {
while (bson_iter_next()) {
key = bson_iter_key();
if (strchr(key, '%')) {
msg_panic("keys in query can not have %% expansions!");
bson_destroy(query);
DICT_MONGODB_LOOKUP_ERR_RETURN(DICT_ERR_RETRY);
}
}
}
bson_destroy(query);
This code doesn't take into account arrays in query right now. If need be, we
can create
a function to check keys and iterate arrays as well (for example, the $or
operator has
an array of objects as operand, each object has its own keys).
By the way, I have ran all the tests I originally run on my code, and they all
passed
with the code from
https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb
Regards
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On 12/6/23 20:39, Viktor Dukhovni via Postfix-devel wrote: > On Thu, Dec 07, 2023 at 01:06:57AM +, Hamid Maadani wrote: > However, I am concerned about the use of `bson_new_from_json()` and its need to quote the MongoDB operators. This feels completely unnatural. How is there then a distinction between: $or: [...] and "$or": [...] the latter should be a verbatim key called "$or", not a MongoDB operator. How do we avoid having issues with inputs that contain a leading "$", or are the leading "$" signs only special in the JSON object key, rather than the value? This needs to be understood and documented. As well as clarifying any potential confusion around projections... >>> ... >>> I am still uneasy about this. What if one really wanted a key that >>> starts with "$"? Ideally the API would have supported operators without >>> overloading already quoted strings. >> >> Using 'bson_new_from_json' seems to be the easiest way to give admins >> flexibility on what queries/projections they want to have. I actually >> initially wanted to use aggregations, but decided against that to keep >> simplicity. >> >> Mongo 5.0 and above support keys that start with dollar signs according to >> this: >> https://www.mongodb.com/docs/manual/core/dot-dollar-considerations >> > > I am somewhat reassured by the fact that that document consistently only > talks about dollar-prefixed *keys*, and makes no mention of special > concerns for dollar-prefixed values. So I guess, the user will have to > know that despite the formal MongoDB syntax not needing quotes for $or, > the Postfix dictionary driver will require quotes, and the operator will > still work. > > Provided "%s", "%u", and the like always appear on the *value* side of a > MongoDB query, there are no related issues. Anyone using external input > to set a *key* in the JSON query would be asking for trouble... > > We probably don't need to go as far as parsing the JSON query to ensure > that '%x' substitutions happen only in values and not in keys... I think it would be preferable to do this, as it catches human error that would result in an insecure system. One just needs to ensure that keys keys never have a % that is not followed by another %. JSON syntax rules mean that a % cannot appear anywhere else. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On Thu, Dec 07, 2023 at 01:06:57AM +, Hamid Maadani wrote: > >> However, I am concerned about the use of `bson_new_from_json()` and its > >> need to quote the MongoDB operators. This feels completely unnatural. > >> How is there then a distinction between: > >> > >> $or: [...] > >> > >> and > >> > >> "$or": [...] > >> > >> the latter should be a verbatim key called "$or", not a MongoDB > >> operator. How do we avoid having issues with inputs that contain a > >> leading "$", or are the leading "$" signs only special in the JSON > >> object key, rather than the value? This needs to be understood and > >> documented. As well as clarifying any potential confusion around > >> projections... > > ... > > I am still uneasy about this. What if one really wanted a key that > > starts with "$"? Ideally the API would have supported operators without > > overloading already quoted strings. > > Using 'bson_new_from_json' seems to be the easiest way to give admins > flexibility on what queries/projections they want to have. I actually > initially wanted to use aggregations, but decided against that to keep > simplicity. > > Mongo 5.0 and above support keys that start with dollar signs according to > this: > https://www.mongodb.com/docs/manual/core/dot-dollar-considerations > I am somewhat reassured by the fact that that document consistently only talks about dollar-prefixed *keys*, and makes no mention of special concerns for dollar-prefixed values. So I guess, the user will have to know that despite the formal MongoDB syntax not needing quotes for $or, the Postfix dictionary driver will require quotes, and the operator will still work. Provided "%s", "%u", and the like always appear on the *value* side of a MongoDB query, there are no related issues. Anyone using external input to set a *key* in the JSON query would be asking for trouble... We probably don't need to go as far as parsing the JSON query to ensure that '%x' substitutions happen only in values and not in keys... -- Viktor. ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
>> However, I am concerned about the use of `bson_new_from_json()` and its >> need to quote the MongoDB operators. This feels completely unnatural. >> How is there then a distinction between: >> >> $or: [...] >> >> and >> >> "$or": [...] >> >> the latter should be a verbatim key called "$or", not a MongoDB >> operator. How do we avoid having issues with inputs that contain a >> leading "$", or are the leading "$" signs only special in the JSON >> object key, rather than the value? This needs to be understood and >> documented. As well as clarifying any potential confusion around >> projections... > ... > I am still uneasy about this. What if one really wanted a key that > starts with "$"? Ideally the API would have supported operators without > overloading already quoted strings. Using 'bson_new_from_json' seems to be the easiest way to give admins flexibility on what queries/projections they want to have. I actually initially wanted to use aggregations, but decided against that to keep simplicity. Mongo 5.0 and above support keys that start with dollar signs according to this: https://www.mongodb.com/docs/manual/core/dot-dollar-considerations I have not found an example on how to search for one though... Regards Hamid Maadani ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On Wed, Dec 06, 2023 at 07:31:41PM -0500, Viktor Dukhovni via Postfix-devel
wrote:
> However, I am concerned about the use of `bson_new_from_json()` and its
> need to quote the MongoDB operators. This feels completely unnatural.
> How is there then a distinction between:
>
> $or: [...]
>
> and
>
> "$or": [...]
>
> the latter should be a verbatim key called "$or", not a MongoDB
> operator. How do we avoid having issues with inputs that contain a
> leading "$", or are the leading "$" signs only special in the JSON
> object key, rather than the value? This needs to be understood and
> documented. As well as clarifying any potential confusion around
> projections...
It does, however, look overloading:
{ "$operator": ... }
to be the same as:
{ $operator: ... }
is expected practice with MongoDB:
https://github.com/mongodb/mongo-c-driver/blob/54f737ea488caadac0cf9275c4be1fbb37cf5609/src/libmongoc/tests/test-mongoc-matcher.c#L222-L267
So the best we can hope for is that this overloading is restricted to
keys, and never applies to values in queries, so that in:
{ "$or": [ "foo": "$bar" ] }
only "$or" is special, while "$bar" is a literal. Users will then have
to know to let untrusted content leak into query keys, but that should
be obvious regardless of metacharacter issues.
I am still uneasy about this. What if one really wanted a key that
starts with "$"? Ideally the API would have supported operators without
overloading already quoted strings.
--
Viktor.
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
> Sorry about that. I added error checking everywhere, and I
> may have been too pessimistic (getting tru and false wroing).
It was actually my config.. I had projections set as:
{"projection": {...}}
where the should just be a simple JSON object. Guess I was testing
something and forgot to change it back.
all my basic checks seem to work for now. I will run some more tests tomorrow.
Regards
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On Wed, Dec 06, 2023 at 07:06:30PM -0500, Wietse Venema via Postfix-devel wrote:
> I have been adding text to the mongodb_table that any text pasted
> in the place of a %letter directive in result_format will be subject
> to escaping, that is, Postfix inserts a backslash character before
> a double quote or backslash character.
>
> This ensures that the result will have the same structure as
> result_format: each string in the result_format is still exactly
> one string in the result, and each special character {}[], etc. is
> still exactly one in the result. An attacker cannot 'control' how
> the result will be processed.
>
> What about projections? Given
>
> projection = { "_id":0, "mail_path": {"$concat": ["$domain", "/",
> "$local_part"]} }
>
> what if $domains contains
>
> foo"]}, nasty stuff...
>
Here "$domain" is a *field name* from the JSON schema. The `$concat`
operator will use the associated response element as part of
constructing a the value of the "mail_path" element of the response.
I don't think there's a problem here as such.
However, I am concerned about the use of `bson_new_from_json()` and its
need to quote the MongoDB operators. This feels completely unnatural.
How is there then a distinction between:
$or: [...]
and
"$or": [...]
the latter should be a verbatim key called "$or", not a MongoDB
operator. How do we avoid having issues with inputs that contain a
leading "$", or are the leading "$" signs only special in the JSON
object key, rather than the value? This needs to be understood and
documented. As well as clarifying any potential confusion around
projections...
--
Viktor.
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
Sorry, I was confusing query and result processing. The first needs to be secured. The second is garbage in, garbage out. Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
On 7/12/23 04:53, Wietse Venema via Postfix-devel wrote: (major) there is no code to escape special characters when parts or all of a Postfix query are pasted into the MongoDB query filter. I think that at the very least, quotes and backslashes should be escaped with a backslash. I can add a little function for that and update the mongodb_table file. Since it's JSON then escaping according to JSON rules would seem appropriate. Using a JSON library would generally be ideal but probably not worth it here. RFC 4627 basically says to escape quotes, backslashes and any control characters (U+ through U+001F): https://www.rfc-editor.org/rfc/rfc4627#section-2.5 Peter ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
I have been adding text to the mongodb_table that any text pasted
in the place of a %letter directive in result_format will be subject
to escaping, that is, Postfix inserts a backslash character before
a double quote or backslash character.
This ensures that the result will have the same structure as
result_format: each string in the result_format is still exactly
one string in the result, and each special character {}[], etc. is
still exactly one in the result. An attacker cannot 'control' how
the result will be processed.
What about projections? Given
projection = { "_id":0, "mail_path": {"$concat": ["$domain", "/",
"$local_part"]} }
what if $domains contains
foo"]}, nasty stuff...
If an attacker can change the shape of the projection, then that
would be a problem.
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel: > Ok, compiled with driver 1.23.4 and have no more linking issues. > I noticed if I have docker_va_filter parameter, it throws: > Dec 06 22:32:49 mail postfix/proxymap[202]: fatal: bad string length 0 < 1: > docker_va_query_filter = > > I think line 168 needs to be: > dict_mongodb->query_filter = cfg_get_str(p, "query_filter", NULL, 0, 0); > for the fallback to work (from query_filter to filter). Correct. The idea was to introduce the better name without breaking your configs that were using "filter". I am updateing the source code. > The queries are not returning any results tho, need to do some more > debugging... Sorry about that. I added error checking everywhere, and I may have been too pessimistic (getting tru and false wroing). > > Only when config is specified in main.cf. Not when config > > is specified as in the current mongodb_table manpage. > > True. From your's and Viktor's explanation, I understand that the > documentation is in the wrong place. I'll defer to you to move it > to the correct location, still a bit confused on where it should be. I'm adding a section to the mongodb_table manual page. Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Ok, compiled with driver 1.23.4 and have no more linking issues.
I noticed if I have docker_va_filter parameter, it throws:
Dec 06 22:32:49 mail postfix/proxymap[202]: fatal: bad string length 0 < 1:
docker_va_query_filter =
I think line 168 needs to be:
dict_mongodb->query_filter = cfg_get_str(p, "query_filter", NULL, 0, 0);
for the fallback to work (from query_filter to filter).
The queries are not returning any results tho, need to do some more debugging...
> Only when config is specified in main.cf. Not when config
> is specified as in the current mongodb_table manpage.
True. From your's and Viktor's explanation, I understand that the
documentation is in the wrong place. I'll defer to you to move it
to the correct location, still a bit confused on where it should be.
>> now, in my case, I'm using a docker container, and am using parameters
>> in main.cf , a sample below:
>> docker_va_uri = $docker_dburi
>> docker_va_dbname = $docker_dbname
>> docker_va_collection = mailbox
>> docker_va_filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}],
>> "active": 1}
>> docker_va_result_attribute = username
Great, so at least I was using it right in my testing scenario. A bit
confused about Viktor's comment on how I am confusing the layers, but
I would need to learn more about postfix and I'm sure I'll figure that
out. Is there an example on how to do the parameterized config I can
look at?
> Here you have to duplicate the $. That leaves Viktor's question
> about placing quotes around "$or".
This is because bson_new_from_json is used. A proper JSON format is
required for this function to work, and that means $or should come
inside double quotes. If I do something like:
#include
#include
void main() {
bson_error_t err;
bson_t *obj = bson_new_from_json("{$or: [{\"a\": 1}, {\"a\": 2}]}", -1,
);
if (!obj) {
printf("%s\n", err.message);
}
bson_destroy(obj);
}
and run the compiled binary, I will get:
Got parse error at "$", position 1: "SPECIAL_EXPECTED"
Changing the line to:
bson_t *obj = bson_new_from_json("{\"$or\": [{\"a\": 1}, {\"a\": 2}]}", -1,
);
would resolve the issue.
Regards,
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani:
> > If the configuration says $$or, what code shall responsible for
> > treating this as $or? The Postfix cfg_parser does not do that, and
> > dict_mongodb.c will pass $$or into the mongo-c library.
>
> It does not? I might totally be me misunderstanding the documentation,
Only when config is specified in main.cf. Not when config
is specified as in the current mongodb_table manpage.
> now, in my case, I'm using a docker container, and am using parameters
> in main.cf , a sample below:
> docker_va_uri = $docker_dburi
> docker_va_dbname = $docker_dbname
> docker_va_collection = mailbox
> docker_va_filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}],
> "active": 1}
> docker_va_result_attribute = username
Here you have to duplicate the $. That leaves Viktor's question
about placing quotes around "$or".
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Viktor Dukhovni via Postfix-devel:
> On Wed, Dec 06, 2023 at 08:10:22PM +, Hamid Maadani via Postfix-devel
> wrote:
>
> > now, in my case, I'm using a docker container, and am using parameters
> > in main.cf , a sample below:
> > docker_va_uri = $docker_dburi
> > docker_va_dbname = $docker_dbname
> > docker_va_collection = mailbox
> > docker_va_filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}],
> > "active": 1}
> > docker_va_result_attribute = username
>
> You're mixing up the layers. In the legacy flat "main.cf" SQL-like
This means we have to add back the section in legact configuration
configuration, and put the '$$' handlingf there because,
because that really only happens with main.cf parsing.
> And I am aslo rather puzzled by the double quotes you're putting around
> `$or`. These certainly don't appear in the MongoDB documentation. I
> would expect `"$or"` to be treated as a verbatim JSON key and not as a
> MongoDB operator. Otherwise, we potentially have deeper quoting issues
> than just double-quote and backslash characters...
Let's hear from Hamid.
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
On Wed, Dec 06, 2023 at 08:10:22PM +, Hamid Maadani via Postfix-devel wrote:
> now, in my case, I'm using a docker container, and am using parameters
> in main.cf , a sample below:
> docker_va_uri = $docker_dburi
> docker_va_dbname = $docker_dbname
> docker_va_collection = mailbox
> docker_va_filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}],
> "active": 1}
> docker_va_result_attribute = username
You're mixing up the layers. In the legacy flat "main.cf" SQL-like
table syntax, with "tablename_parameter_name" settings, yes "$" needs
to be "$$" to survive *main.cf* parameter expansion, but that is NOT
part of the underlying *table syntax*, which is what users would write
in:
main.cf:
mongodb = proxy:mongodb:${config_directory}/
virtual_alias_maps = ${mongodb}mongo-valias.cf
mongo-valias.cf:
...
filter = { $or: [ {"mail": "%s"}, { "alias": "%s" } ] }
...
And I am aslo rather puzzled by the double quotes you're putting around
`$or`. These certainly don't appear in the MongoDB documentation. I
would expect `"$or"` to be treated as a verbatim JSON key and not as a
MongoDB operator. Otherwise, we potentially have deeper quoting issues
than just double-quote and backslash characters...
--
Viktor.
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
On Wed, Dec 06, 2023 at 02:25:39PM -0500, Wietse Venema via Postfix-devel wrote:
> > This is a good point. Honestly, I didn't think about escaping characters
> > because the queries are meant to be in JSON form and taken literally,
For a lookup key to be taken "literally" its metacharacters MUST be
escaped, so that it does not introduce unintended syntax! The data
interpolated via '%s' and '%u' comes from untrusted sources and MUST NOT
be allowed to introduce an (no)SQL-injection attack:
https://xkcd.com/327/
The documentation should clearly state that all %s/%u/%d/%[1-9]
expansions MUST be enclosed in double quotes to ensure valid JSON
string syntax:
- { "anyaddr": "%s" }
- { "domainaddr": "%u@%d" }
- { "2ld": "%2.%1" }
- ...
There is no mechanism for non-string or structured compound inputs to
the Postfix table lookup layer, so the lookup key is always an
unstructured string, containing untrusted data, and will be escaped for
inclusion in a quoted string, but the enclosing quotes MUST be provided
by the Postfix administrator configuring the lookup table.
[ By the way, db_common_expand() assumes that domain names do not
contain escaped "." characters in labels, and just performs a
naïve split on "." rather than parsing a general presentation
form domain, which might be "foo\.bar.example.com", with
"foo.bar" as its logical first label. I expect that's not
a concern. Since non-RFC1123 names are broadly rejected
by Postfix at various layers. ]
> > > (minor) the database config file parser does not expand $name,
> > > ${name} etc. so '$$' is taken literally, not as '$'. I can remove
> > > that text from the mongodb_table file
> >
> > I think in the mongodb_table file, the expansions like $$ are included for
> > query_filter and projection. "query_filter" is expanded in
> > dict_mongodb_lookup
> > (line 411), but projection is not. would be best to expand projection as
> > well
> > (maybe around line 377?)
>
> What code is supposed to pay attention to '$' characters? The Postfix client?
> The MongoC library?
I don't see any code that expands "$$" to just "$". The referenced
db_common_expand() function called near line 411:
https://github.com/wietse-postfix/postfix-dukhovni/blob/c753d0a358fc6e02ca3bf8b25a2598aedea4dfb8/postfix/src/global/db_common.c#L408-L510
does nothing special with '$' characters. If MongoDB expects "$or" as
an operator, then this is verbatim what needs to be in the query.
Has this code been tested? I don't understand how the "$$or" ever
worked:
https://www.mongodb.com/docs/manual/reference/operator/query/or/
--
Viktor.
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
> If the configuration says $$or, what code shall responsible for
> treating this as $or? The Postfix cfg_parser does not do that, and
> dict_mongodb.c will pass $$or into the mongo-c library.
It does not? I might totally be me misunderstanding the documentation,
and I am by no means a postfix expert, so please correct me if I'm wrong.
I see:
"... The general format of the main.cf file is as follows:
... Specify "$$" to produce a single "$" character. ..."
in https://www.postfix.org/postconf.5.html
now, in my case, I'm using a docker container, and am using parameters
in main.cf , a sample below:
docker_va_uri = $docker_dburi
docker_va_dbname = $docker_dbname
docker_va_collection = mailbox
docker_va_filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}],
"active": 1}
docker_va_result_attribute = username
and then use something like:
postmap -q ha...@dexo.tech proxy:mongodb:docker_va
Is this the wrong approach?
Does cfg_parser or db_common_expand not replace $$ in docker_va_filter
with a single $?
Regards
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel:
> > What code is supposed to pay attention to '$' characters? The Postfix
> > client?
> > The MongoC library?
>
> In MQL, you have operators like "$or", or you can use the value of a field
> like "$field". These are instances that the person configuring postfix would
> need to use a $ character in either query or projection. In this example:
> filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}], "active": 1}
> I am configuring dict_mongo to use an $or operator, and search the collection
> for any records which has a username or alias.address set to the requested
> value.
If the configuration says $$or, what code shall responsible for
treating this as $or? The Postfix cfg_parser does not do that, and
dict_mongodb.c will pass $$or into the mongo-c library.
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Wietse Venema via Postfix-devel: > Hamid Maadani via Postfix-devel: > > > There was a missing update to the makedefs script (in the top-level > > > directory). The updated code is at > > > https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb > > > > hmm.. I'm still getting the same type of error after a rebuild: > > Dec 06 17:32:07 mail postfix/proxymap[2595]: fatal: load_library_symbols: > > dlopen failure loading > > /usr/lib/postfix/postfix-mongodb.so: Error relocating > > /usr/lib/postfix/postfix-mongodb.so: > > mongoc_client_new_from_uri_with_error: symbol not found > > Your original code calls mongoc_client_new() The _with error() > variant was added two years ago. > > How would I find out which mongo-c library version supports the > _with error() variant which gives a better error message? Looks like I will need to do a binary seach on Mongo-C documentaition. It's not the end of the world. What is your Mongo-C library version? I was using 1.24.3: /usr/include/libmongoc-1.0/mongoc/mongoc-version.h:#define MONGOC_VERSION (1.24.3) Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
> Your original code calls mongoc_client_new() The _with error()
> variant was added two years ago.
> How would I find out which mongo-c library version supports the
> _with error() variant which gives a better error message?
> The on-line documentation is not quite clear about that, it seems
> to assume that everyone will be using the latest and greatest.
ah, I see. It was introduced in v1.21.0 with:
https://github.com/mongodb/mongo-c-driver/commit/e7e15002d63cb57424f467c5f21eafa9ec0f018e
I'm using alpine 3.16 which has v1.16.2 , will upgrade to alpine 3.18
and build with v1.23.4 of the driver.
> The "%s" stuff will paste in bits from the Postfix query, which
> can be an email address provided by a hostile SMTP client.
> A hostile client could use an address that contains quotes or
> backslashes, to change the structure of the MongoDB query,
> and exercise database features that you did not intend..
> If the idea is that the "user" would escape the quotes etc,
> then I wodewr what user you have inb mind. It can't be the
> person who configures Postfix or the person who maintains the
> database.
That is fair, I have failed to consider that scenario. MQL is not prone
to SQL Injection type attacks, but escaping would make it more safe.
Absolutely.
> What code is supposed to pay attention to '$' characters? The Postfix client?
> The MongoC library?
In MQL, you have operators like "$or", or you can use the value of a field
like "$field". These are instances that the person configuring postfix would
need to use a $ character in either query or projection. In this example:
filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}], "active": 1}
I am configuring dict_mongo to use an $or operator, and search the collection
for any records which has a username or alias.address set to the requested
value.
Regards
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel:
> > There was a missing update to the makedefs script (in the top-level
> > directory). The updated code is at
> > https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb
>
> hmm.. I'm still getting the same type of error after a rebuild:
> Dec 06 17:32:07 mail postfix/proxymap[2595]: fatal: load_library_symbols:
> dlopen failure loading
> /usr/lib/postfix/postfix-mongodb.so: Error relocating
> /usr/lib/postfix/postfix-mongodb.so:
> mongoc_client_new_from_uri_with_error: symbol not found
Your original code calls mongoc_client_new() The _with error()
variant was added two years ago.
How would I find out which mongo-c library version supports the
_with error() variant which gives a better error message?
The on-line documentation is not quite clear about that, it seems
to assume that everyone will be using the latest and greatest.
> > (major) there is no code to escape special characters when parts
> > or all of a Postfix query are pasted into the MongoDB query filter.
> > I think that at the very least, quotes and backslashes should be
> > escaped with a backslash. I can add a little function for that and
> > update the mongodb_table file.
>
> This is a good point. Honestly, I didn't think about escaping characters
> because the queries are meant to be in JSON form and taken literally, e.g.:
> filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}], "active": 1}
> So I left it to the user to escape. But it might be a good idea to add that
> (maybe by using bson_utf8_escape_for_json after expansion?)
The "%s" stuff will paste in bits from the Postfix query, which
can be an email address provided by a hostile SMTP client.
A hostile client could use an address that contains quotes or
backslashes, to change the structure of the MongoDB query,
and exercise database features that you did not intend..
If the idea is that the "user" would escape the quotes etc,
then I wodewr what user you have inb mind. It can't be the
person who configures Postfix or the person who maintains the
database.
> > (minor) the database config file parser does not expand $name,
> > ${name} etc. so '$$' is taken literally, not as '$'. I can remove
> > that text from the mongodb_table file
>
> I think in the mongodb_table file, the expansions like $$ are included for
> query_filter and projection. "query_filter" is expanded in
> dict_mongodb_lookup
> (line 411), but projection is not. would be best to expand projection as well
> (maybe around line 377?)
What code is supposed to pay attention to '$' characters? The Postfix client?
The MongoC library?
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
> There was a missing update to the makedefs script (in the top-level
> directory). The updated code is at
> https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb
hmm.. I'm still getting the same type of error after a rebuild:
Dec 06 17:32:07 mail postfix/proxymap[2595]: fatal: load_library_symbols:
dlopen failure loading
/usr/lib/postfix/postfix-mongodb.so: Error relocating
/usr/lib/postfix/postfix-mongodb.so:
mongoc_client_new_from_uri_with_error: symbol not found
> (major) there is no code to escape special characters when parts
> or all of a Postfix query are pasted into the MongoDB query filter.
> I think that at the very least, quotes and backslashes should be
> escaped with a backslash. I can add a little function for that and
> update the mongodb_table file.
This is a good point. Honestly, I didn't think about escaping characters
because the queries are meant to be in JSON form and taken literally, e.g.:
filter = {"$$or": [{"username":"%s"}, {"alias.address": "%s"}], "active": 1}
So I left it to the user to escape. But it might be a good idea to add that
(maybe by using bson_utf8_escape_for_json after expansion?)
> (minor) the database config file parser does not expand $name,
> ${name} etc. so '$$' is taken literally, not as '$'. I can remove
> that text from the mongodb_table file
I think in the mongodb_table file, the expansions like $$ are included for
query_filter and projection. "query_filter" is expanded in dict_mongodb_lookup
(line 411), but projection is not. would be best to expand projection as well
(maybe around line 377?)
Regards
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel:
> > Can you copy the dict_mongodb.* files into your environment and see
> > what I broke? They should still build with postfix-3.8-20220527.
>
> I just did, and they build fine. However, I get a linking error when trying
> to use the library:
> "Dec 05 23:45:05 mail postfix/proxymap[29029]: fatal:
> load_library_symbols: dlopen failure loading
> /usr/lib/postfix/postfix-mongodb.so: Error relocating
> /usr/lib/postfix/postfix-mongodb.so: mongoc_collection_find_with_opts:
> symbol not found"
There was a missing update to the makedefs script (in the top-level
directory). The updated code is at
https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb
Could you comment on the following:
(major) there is no code to escape special characters when parts
or all of a Postfix query are pasted into the MongoDB query filter.
I think that at the very least, quotes and backslashes should be
escaped with a backslash. I can add a little function for that and
update the mongodb_table file.
(minor) the database config file parser does not expand $name,
${name} etc. so '$$' is taken literally, not as '$'. I can remove
that text from the mongodb_table file.
A liitle script to quickly build a throw-away MongoDB database
for testing would also be helpful, but that can be done outside
the mailing list.
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel: > > Can you copy the dict_mongodb.* files into your environment and see > > what I broke? They should still build with postfix-3.8-20220527. > > I just did, and they build fine. However, I get a linking error when trying > to use the library: > "Dec 05 23:45:05 mail postfix/proxymap[29029]: fatal: load_library_symbols: > dlopen failure loading /usr/lib/postfix/postfix-mongodb.so: Error relocating > /usr/lib/postfix/postfix-mongodb.so: mongoc_collection_find_with_opts: symbol > not found" > I get a different error (Fedora) with shared=yes dynamicmaps=yes: postmap: fatal: load_library_symbols: dlopen failure loading /usr/lib/postfix/3.9-20231112/postfix-mongodb.so: /usr/lib/postfix/3.9-20231112/postfix-mongodb.so: undefined symbol: mongoc_client_set_error_api But it is likely for a similar reason. I';; look at this tomorrow. Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
> Can you copy the dict_mongodb.* files into your environment and see > what I broke? They should still build with postfix-3.8-20220527. I just did, and they build fine. However, I get a linking error when trying to use the library: "Dec 05 23:45:05 mail postfix/proxymap[29029]: fatal: load_library_symbols: dlopen failure loading /usr/lib/postfix/postfix-mongodb.so: Error relocating /usr/lib/postfix/postfix-mongodb.so: mongoc_collection_find_with_opts: symbol not found" I do not have this issue with the one building from my repo. Tried it with version 3.7.8 (default for alpine 3.16 which is my testing platform at the moment), and also tried it on the build box with version 3.9-20231112, same issue. Will look into that, run some tests as soon as that is fixed and let you know. Regards Hamid Maadani December 3, 2023 5:01 PM, "Wietse Venema via Postfix-devel" wrote: > Hamid Maadani via Postfix-devel: > >> Yes sir. the last two commit in this branch should cover everything needed: >> https://github.com/21stcaveman/postfix/tree/mongodb > > I forked Viktor's repo and added a version of your code and > documentation relative to the last Postfix 3.9 development release. > > The code is untested and there are some comments with XXX(Wietse) > for items that need attention. One is whether we need to quote > metacharacters in the Postfix lookup key. > > Can you copy the dict_mongodb.* files into your environment and see > what I broke? They should still build with postfix-3.8-20220527. > > https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb > > Wietse > ___ > Postfix-devel mailing list -- postfix-devel@postfix.org > To unsubscribe send an email to postfix-devel-le...@postfix.org ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel: > Yes sir. the last two commit in this branch should cover everything needed: > https://github.com/21stcaveman/postfix/tree/mongodb I forked Viktor's repo and added a version of your code and documentation relative to the last Postfix 3.9 development release. The code is untested and there are some comments with XXX(Wietse) for items that need attention. One is whether we need to quote metacharacters in the Postfix lookup key. Can you copy the dict_mongodb.* files into your environment and see what I broke? They should still build with postfix-3.8-20220527. https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Hamid Maadani via Postfix-devel: > Yes sir. the last two commit in this branch should cover everything needed: > https://github.com/21stcaveman/postfix/tree/mongodb Got it. The diffs of this code against postfix-3.8-20220527 will apply easily to Postfix 3.9. I'm making first pass over the code and docs. If you hear nothing then it will very likely in Postfix 3.9. Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Yes sir. the last two commit in this branch should cover everything needed: https://github.com/21stcaveman/postfix/tree/mongodb Regards Hamid Maadani November 29, 2023 4:55 PM, "Wietse Venema via Postfix-devel" wrote: > Viktor Dukhovni (27 Jun 2022): > >> After that, Wietse and I will have to find some time for code >> review. This may take a bit of time, but should ideally happen in >> time for 3.8.0, and so naturally would need to be complete a few >> snapshots earlier. > > hamid via Postfix-devel: >> Was doing some server maintenance and was reminded of this. Anything >> you guys need from me on this?RegardsHamid Maadani > > I must have been really busy, because there is a thread with 41 > messages between June 15-27 2022, and I replied only to the initial > posting. And if I do not start work on simething then it never happens. > > Is this still the location? > https://github.com/21stcaveman/postfix/blob/mongodb/postfix/src/global/dict_mongodb.c > > Wietse > > ___ > Postfix-devel mailing list -- postfix-devel@postfix.org > To unsubscribe send an email to postfix-devel-le...@postfix.org ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
Viktor Dukhovni (27 Jun 2022): > After that, Wietse and I will have to find some time for code > review. This may take a bit of time, but should ideally happen in > time for 3.8.0, and so naturally would need to be complete a few > snapshots earlier. hamid via Postfix-devel: > Was doing some server maintenance and was reminded of this. Anything > you guys need from me on this?RegardsHamid Maadani I must have been really busy, because there is a thread with 41 messages between June 15-27 2022, and I replied only to the initial posting. And if I do not start work on simething then it never happens. Is this still the location? https://github.com/21stcaveman/postfix/blob/mongodb/postfix/src/global/dict_mongodb.c Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb
> After that, Wietse and I will have to find some time for code review. This > may take a bit of time, but should ideally happen in time for 3.8.0, and so > naturally would need to be complete a few snapshots earlier.Was doing some > server maintenance and was reminded of this. Anything you guys need from me > on this?RegardsHamid Maadani ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
