[pfx-dev] Re: dict_mongodb (projections)
>> We probably don't need to go as far as parsing the JSON query to ensure
>> that '%x' substitutions happen only in values and not in keys...
>
> I think it would be preferable to do this, as it catches human error that
> would result in an insecure system. One just needs to ensure that keys
> keys never have a % that is not followed by another %. JSON syntax rules
> mean that a % cannot appear anywhere else.
This is not too hard to do if you guys think it would make for a safer
implementation.
Maybe something like this can be added at line 409 (before query_string
expansion):
bson_iter_t iter;
const char *key = NULL;
query = bson_new_from_json(dict_mongodb->query_filter, -1, );
if (!query) {
msg_warn("%s:%s: failed to create a query from '%s' : %s",
dict_mongodb->dict.type, dict_mongodb->dict.name,
vstring_str(dict_mongodb->query_filter), error.message);
DICT_MONGODB_LOOKUP_ERR_RETURN(DICT_ERR_RETRY);
}
if (bson_iter_init(, query)) {
while (bson_iter_next()) {
key = bson_iter_key();
if (strchr(key, '%')) {
msg_panic("keys in query can not have %% expansions!");
bson_destroy(query);
DICT_MONGODB_LOOKUP_ERR_RETURN(DICT_ERR_RETRY);
}
}
}
bson_destroy(query);
This code doesn't take into account arrays in query right now. If need be, we
can create
a function to check keys and iterate arrays as well (for example, the $or
operator has
an array of objects as operand, each object has its own keys).
By the way, I have ran all the tests I originally run on my code, and they all
passed
with the code from
https://github.com/wietse-postfix/postfix-dukhovni/tree/mongodb
Regards
Hamid Maadani
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On 12/6/23 20:39, Viktor Dukhovni via Postfix-devel wrote: > On Thu, Dec 07, 2023 at 01:06:57AM +, Hamid Maadani wrote: > However, I am concerned about the use of `bson_new_from_json()` and its need to quote the MongoDB operators. This feels completely unnatural. How is there then a distinction between: $or: [...] and "$or": [...] the latter should be a verbatim key called "$or", not a MongoDB operator. How do we avoid having issues with inputs that contain a leading "$", or are the leading "$" signs only special in the JSON object key, rather than the value? This needs to be understood and documented. As well as clarifying any potential confusion around projections... >>> ... >>> I am still uneasy about this. What if one really wanted a key that >>> starts with "$"? Ideally the API would have supported operators without >>> overloading already quoted strings. >> >> Using 'bson_new_from_json' seems to be the easiest way to give admins >> flexibility on what queries/projections they want to have. I actually >> initially wanted to use aggregations, but decided against that to keep >> simplicity. >> >> Mongo 5.0 and above support keys that start with dollar signs according to >> this: >> https://www.mongodb.com/docs/manual/core/dot-dollar-considerations >> > > I am somewhat reassured by the fact that that document consistently only > talks about dollar-prefixed *keys*, and makes no mention of special > concerns for dollar-prefixed values. So I guess, the user will have to > know that despite the formal MongoDB syntax not needing quotes for $or, > the Postfix dictionary driver will require quotes, and the operator will > still work. > > Provided "%s", "%u", and the like always appear on the *value* side of a > MongoDB query, there are no related issues. Anyone using external input > to set a *key* in the JSON query would be asking for trouble... > > We probably don't need to go as far as parsing the JSON query to ensure > that '%x' substitutions happen only in values and not in keys... I think it would be preferable to do this, as it catches human error that would result in an insecure system. One just needs to ensure that keys keys never have a % that is not followed by another %. JSON syntax rules mean that a % cannot appear anywhere else. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On Thu, Dec 07, 2023 at 01:06:57AM +, Hamid Maadani wrote: > >> However, I am concerned about the use of `bson_new_from_json()` and its > >> need to quote the MongoDB operators. This feels completely unnatural. > >> How is there then a distinction between: > >> > >> $or: [...] > >> > >> and > >> > >> "$or": [...] > >> > >> the latter should be a verbatim key called "$or", not a MongoDB > >> operator. How do we avoid having issues with inputs that contain a > >> leading "$", or are the leading "$" signs only special in the JSON > >> object key, rather than the value? This needs to be understood and > >> documented. As well as clarifying any potential confusion around > >> projections... > > ... > > I am still uneasy about this. What if one really wanted a key that > > starts with "$"? Ideally the API would have supported operators without > > overloading already quoted strings. > > Using 'bson_new_from_json' seems to be the easiest way to give admins > flexibility on what queries/projections they want to have. I actually > initially wanted to use aggregations, but decided against that to keep > simplicity. > > Mongo 5.0 and above support keys that start with dollar signs according to > this: > https://www.mongodb.com/docs/manual/core/dot-dollar-considerations > I am somewhat reassured by the fact that that document consistently only talks about dollar-prefixed *keys*, and makes no mention of special concerns for dollar-prefixed values. So I guess, the user will have to know that despite the formal MongoDB syntax not needing quotes for $or, the Postfix dictionary driver will require quotes, and the operator will still work. Provided "%s", "%u", and the like always appear on the *value* side of a MongoDB query, there are no related issues. Anyone using external input to set a *key* in the JSON query would be asking for trouble... We probably don't need to go as far as parsing the JSON query to ensure that '%x' substitutions happen only in values and not in keys... -- Viktor. ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
>> However, I am concerned about the use of `bson_new_from_json()` and its >> need to quote the MongoDB operators. This feels completely unnatural. >> How is there then a distinction between: >> >> $or: [...] >> >> and >> >> "$or": [...] >> >> the latter should be a verbatim key called "$or", not a MongoDB >> operator. How do we avoid having issues with inputs that contain a >> leading "$", or are the leading "$" signs only special in the JSON >> object key, rather than the value? This needs to be understood and >> documented. As well as clarifying any potential confusion around >> projections... > ... > I am still uneasy about this. What if one really wanted a key that > starts with "$"? Ideally the API would have supported operators without > overloading already quoted strings. Using 'bson_new_from_json' seems to be the easiest way to give admins flexibility on what queries/projections they want to have. I actually initially wanted to use aggregations, but decided against that to keep simplicity. Mongo 5.0 and above support keys that start with dollar signs according to this: https://www.mongodb.com/docs/manual/core/dot-dollar-considerations I have not found an example on how to search for one though... Regards Hamid Maadani ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On Wed, Dec 06, 2023 at 07:31:41PM -0500, Viktor Dukhovni via Postfix-devel
wrote:
> However, I am concerned about the use of `bson_new_from_json()` and its
> need to quote the MongoDB operators. This feels completely unnatural.
> How is there then a distinction between:
>
> $or: [...]
>
> and
>
> "$or": [...]
>
> the latter should be a verbatim key called "$or", not a MongoDB
> operator. How do we avoid having issues with inputs that contain a
> leading "$", or are the leading "$" signs only special in the JSON
> object key, rather than the value? This needs to be understood and
> documented. As well as clarifying any potential confusion around
> projections...
It does, however, look overloading:
{ "$operator": ... }
to be the same as:
{ $operator: ... }
is expected practice with MongoDB:
https://github.com/mongodb/mongo-c-driver/blob/54f737ea488caadac0cf9275c4be1fbb37cf5609/src/libmongoc/tests/test-mongoc-matcher.c#L222-L267
So the best we can hope for is that this overloading is restricted to
keys, and never applies to values in queries, so that in:
{ "$or": [ "foo": "$bar" ] }
only "$or" is special, while "$bar" is a literal. Users will then have
to know to let untrusted content leak into query keys, but that should
be obvious regardless of metacharacter issues.
I am still uneasy about this. What if one really wanted a key that
starts with "$"? Ideally the API would have supported operators without
overloading already quoted strings.
--
Viktor.
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
On Wed, Dec 06, 2023 at 07:06:30PM -0500, Wietse Venema via Postfix-devel wrote:
> I have been adding text to the mongodb_table that any text pasted
> in the place of a %letter directive in result_format will be subject
> to escaping, that is, Postfix inserts a backslash character before
> a double quote or backslash character.
>
> This ensures that the result will have the same structure as
> result_format: each string in the result_format is still exactly
> one string in the result, and each special character {}[], etc. is
> still exactly one in the result. An attacker cannot 'control' how
> the result will be processed.
>
> What about projections? Given
>
> projection = { "_id":0, "mail_path": {"$concat": ["$domain", "/",
> "$local_part"]} }
>
> what if $domains contains
>
> foo"]}, nasty stuff...
>
Here "$domain" is a *field name* from the JSON schema. The `$concat`
operator will use the associated response element as part of
constructing a the value of the "mail_path" element of the response.
I don't think there's a problem here as such.
However, I am concerned about the use of `bson_new_from_json()` and its
need to quote the MongoDB operators. This feels completely unnatural.
How is there then a distinction between:
$or: [...]
and
"$or": [...]
the latter should be a verbatim key called "$or", not a MongoDB
operator. How do we avoid having issues with inputs that contain a
leading "$", or are the leading "$" signs only special in the JSON
object key, rather than the value? This needs to be understood and
documented. As well as clarifying any potential confusion around
projections...
--
Viktor.
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
Sorry, I was confusing query and result processing. The first needs to be secured. The second is garbage in, garbage out. Wietse ___ Postfix-devel mailing list -- postfix-devel@postfix.org To unsubscribe send an email to postfix-devel-le...@postfix.org
[pfx-dev] Re: dict_mongodb (projections)
I have been adding text to the mongodb_table that any text pasted
in the place of a %letter directive in result_format will be subject
to escaping, that is, Postfix inserts a backslash character before
a double quote or backslash character.
This ensures that the result will have the same structure as
result_format: each string in the result_format is still exactly
one string in the result, and each special character {}[], etc. is
still exactly one in the result. An attacker cannot 'control' how
the result will be processed.
What about projections? Given
projection = { "_id":0, "mail_path": {"$concat": ["$domain", "/",
"$local_part"]} }
what if $domains contains
foo"]}, nasty stuff...
If an attacker can change the shape of the projection, then that
would be a problem.
Wietse
___
Postfix-devel mailing list -- postfix-devel@postfix.org
To unsubscribe send an email to postfix-devel-le...@postfix.org
