I didn't understand at first either. The code has a comment[1] with a link
providing justification[2].
TLDR: There are 2 issues:
1. Something in our stack juju->mgo->mongo (it hasn't been explained to me what
is doing this) rewrites keys which contain dots into new nested structures.
E.g.: {foo.bar: baz} -> {foo: {bar: baz}}.
2. Storing keys which begin with $ signs can lead to injection attacks if
you're using that data while executing these statements: {$where, mapReduce,
group}. This is better explained in [2].
I hope that helps. Please, keep asking questions until this is better
understood. It looks like a huge blind-spot for most of our developers.
[1] -
https://github.com/juju/juju/blob/master/mongo/utils/data_cleansing.go#L31-L32
[2] - http://docs.mongodb.org/manual/faq/developers/#faq-dollar-sign-escaping
Matthew Williams <[email protected]> writes:
> I seem to be missing something. Why do we need this?
>
> Matty
>
> On 24 Jun 2016 17:14, "Nate Finch" <[email protected]> wrote:
>
> It seems as though we should be cleansing all the keys.... since
> we never know what queries we might want to make in the future.
>
>
>
> On Fri, Jun 24, 2016 at 12:04 PM Katherine Cox-Buday
> <[email protected]> wrote:
>
>
> As I have only just discovered the need to cleanse mongo data,
> I can't say for sure, but it looks like we may have been
> cleansing things in the parts of Juju that need it. William
> may know more.
>
> If not, I imagine a small upgrade step would make short work
> of any problems.
>
> roger peppe <[email protected]> writes:
>
> > This is useful, thanks.
> >
> > Note that's it's not necessary to cleanse *all* keys that go
> into Mongo,
> > just the ones that might be used in queries.
> >
> > But one thought... what about keys that already contain
> full-width
> > dollar and dot?
> >
> > cheers,
> > rog.
> >
> > On 23 June 2016 at 21:09, Katherine Cox-Buday
> > <[email protected]> wrote:
> >> Hey all,
> >>
> >> William gave me a good review and it came up that I wasn't
> cleansing
> >> some of
> >> the data being placed in Mongo. I wasn't aware this had to
> be done,
> >> and
> >> after talking to a few other folks it became apparent that
> maybe not
> >> many
> >> people know we should be doing this.
> >>
> >> At any rate, William also pointed me to some existing code
> which did
> >> this.
> >> I've pulled it out into the mongo/utils package for general
> >> consumption. The
> >> comments do a pretty good job of elucidating why this is
> necessary.
> >>
> >>
> https://github.com/juju/juju/blob/master/mongo/utils/data_cleansing.go
>
> >>
> >> -
> >> Katherine
> >>
> >> --
> >> Juju-dev mailing list
> >> [email protected]
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>
>
> --
> Katherine
>
> --
> Juju-dev mailing list
> [email protected]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
> --
> Juju-dev mailing list
> [email protected]
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
>
--
Katherine
--
Juju-dev mailing list
[email protected]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev
