On Thu, Apr 21, 2016 at 05:55:44PM -0400, Phillip Hallam-Baker wrote:
> On Wed, Apr 20, 2016 at 6:25 PM, Ron <[email protected]> wrote:
> > On Tue, Apr 19, 2016 at 02:51:27PM -0400, Phillip Hallam-Baker wrote:
> >> In the meeting, I proposed that we make the use of JSON in ACME
> >> something that can be easily shared across multiple Web Services.
> >>
> >> In a few years time we are quite likely to have multiple JSON Web
> >> services that people want to use together. Implementing them as a
> >> suite is going to be easiest if there is a consistent style. Some of
> >> the encoding choices in ACME right now make implementation in
> >> statically typed languages rather less convenient than they could be.
> >> In particular it is really difficult to parse data streams when the
> >> type descriptor for an object is a field inside the object that can
> >> occur last rather than being first on the wire.
> >
> > I was one of the remote people who thought several of the things you
> > said at the meeting had merit and were worth discussing further here,
> > so thanks for writing this up.
> >
> > But I also thought you were talking about something more than what you
> > have proposed here now - so could you perhaps clarify a little - did
> > some of us read more into what you were saying than you actually had
> > in mind, or have you backed away from some of those suggestions after
> > further discussion after the meeting?
>
> Yes and no. When I originally got the speaking slot, 'submit to last
> call' was not on the agenda. So the proposal I am making is slightly
> more modest.
>
> If we were 6 months out from last call then I would absolutely propose
> splitting the draft into two documents, a platform document and an
> ACME protocol document. In fact I already have such a draft:
>
> https://tools.ietf.org/html/draft-hallambaker-json-web-service-02
>
> But I don't want my proposal to be critical path for last call. And so
> what I have done instead is to implement my version of ACME according
> to the layered model I describe and work out what the minimal changes
> that would be necessary to make ACME consistent.
I'm less sure about splitting the document at this stage (if there are
real reuse points to be had elsewhere a separate document can still
be produced later) - but I also don't think this is the biggest blocker
for moving to last call any time soon ...
Right now we have exactly zero implementations of the -02 draft (I have
an almost complete client implementation, but there is no server that
I can test it with nor am I aware of other client implementations), and
we do have a number of known troublesome issues with the current draft,
some of which moving more of this into the JSON would help to address.
(Some of which I've already raised here, some which I still have queued
up since there's no point flooding this list with them now while the
first lot are still unresponded to and while I'm still running into more
of them while polishing and testing the implementation I have).
> What I would like to avoid is what hit use with SAML where the people
> who insisted on one approach to layering in Version 1.0 discovered it
> didn't really work and it forced us to a Version 2.0. Ooops.
Even without the 'layering' changes you are proposing here, we already
have a number of "oops it doesn't really work" things to address still.
I'd love to see this go to last call as soon as possible. But rushing
that doesn't seem wise when we still have both implementation and
security model issues shaking out that are yet to be resolved.
> > In particular one of the things that I thought seemed important was
> > moving _more_ of the protocol into the JSON. Right now we have a
> > bit of a mish-mash where part of the data you need to complete some
> > task is in the JSON, and part of it is squirrelled away in the HTTP
> > headers.
>
> I am trying to work out where things are happening, right now it looks
> as if we have things happening in the HTTP headers that are not
> actually covered by the authentication at all. Or that could just be
> me misreading the draft, either way it is confusing.
Well, they are covered by the TLS connection to the server. And the
server currently does not sign any of the other content it returns
separately to that. Only client submissions contain protected content.
The more immediate problem is we have things like Link headers being
used to indicate where to obtain issuer certificates from, which are
both underspecified, and it would seem, insufficient for that task in
general.
> > And there's a few places at least where that creates real
> > problems (like where an issued certificate might have more than one
> > valid issuer chain, just to pick one that JSHA and I had looked at).
> >
> > I got the impression you were hoping to make this more transport
> > agnostic (which seems like a useful and fairly easy thing to do if
> > we are going to change things to the degree you are talking here).
>
> That is still my objective. Though I agree that the first cut might
> actually be more of a 'discover bits of the spec that were assumed a
> bit too much'.
Right now, boulder only implements part of the -01 draft, and some
things differently to how the -01 draft says they should be ...
Since migrating it to -02 (let alone -03) is going to be a notable
transition (and will surely break many of the extant clients that
don't actually implement the spec at all but rather some approximation
of it RE'd from what the LE client did), I think it's worth spending
the time to get as much of this right as we already know is troublesome
before it starts that transition, otherwise we're just going to force
two (or more) transitions on the existing users ...
I'm certainly happy to propose and write up things based on what I've
learned from my implementation of this - but it would be nice to get
a little more feedback on some of the things I'd already raised here ...
I don't want to spend a lot of time on that unless we have some at
least in principle agreement about the problem(s) and preferred
solutions from the document authors.
> > I understand the idea of what you are aiming for with this sort of
> > change - but the idea of a protocol full of JSON objects that are
> > mandated to strictly have only one field worries me a bit.
> >
> > It seems more like an admission that "JSON is wrong way to encode
> > this" (or perhaps "I'm using the wrong tool to decode JSON") than
> > a genuine advantage to the encoding. And I'm not of the opinion
> > that JSON is the wrong format to encode this in.
>
> I don't see that as being the case. Rather we have a series of
> protocol messages, each protocol message is of a specific type. We
> should state the type in JSON. So my ideal would be:
>
> "tag" : { ... object ... }
>
> But it seems to be easier to interface these to JSON support code
> wrapped up as an object.
>
> { "tag" : { ... object ... } }
>
>
> Now in a future version of the spec, I could well imagine that we
> might want to relax that one field requirement so that one
> authentication could cover two or more messages or we might have
> arrays of objects instead of singletons. So say we were requesting a
> thousand device certs at a time, we could do that.
The problem I see with what you've proposed here (based on the specific
examples you gave), is that you're just shuffling the unknowns around,
and not in a way that I see improving the general robustness.
If I have something like:
"field": { "type": "x", "whatever": ... }
Then the spec can concretely define what "type" means, what the
allowable grammar for its value may be, and what I should do if my
code doesn't recognise a value it has. It likewise knows that if
I don't see the mandatory fields then the object is invalid.
If I instead have something like:
"field: { "randomtag1": {...}, "randomtag2": ... }
Then I have no context at all to know what I should do with tags
that I don't already have hardcoded. And the solution to that
requires either only have strictly one member of "field", so that
I know "randomtag1" is implicitly the "type" field, or having a
separate 'critical' flag (which experience says nobody will respect
or implement correctly anyway) and makes most parsers more complex.
> So I am saying that for purposes of ACME/1.0 we only need singleton
> requests and responses.
I think if we _require_ that, then this is no longer JSON and we
lose a lot of the nice advantages using JSON gives us. And if we
don't require that, then we're introducing nastier problems than
it it solving ...
> > JSON objects can have multiple fields, and those field are unordered.
> > I don't think you can really escape that, and I don't think we can
> > reasonably "fix" that, without inventing something that "Isn't JSON".
>
> Or you can use nesting because the tag always precedes the value.
>
> > There may be smarter ways we can arrange some of these structures,
> > for good reasons, but I'm not sure that reinventing ASN.1 in JSON is
> > a plan I'd happily endorse.
>
> I see ASN.1 as a terrible implementation of a good idea. The idea of
> having one data encoding is good. The one they chose is horrible and
> DER worst of all.
Don't get me wrong, ASN.1 and TLV in general is fine for what they
were designed for, carrying extensible binary data where bandwidth
is limited.
But this isn't ISDN, and we're using a text protocol, which means
you *have* to parse and validate the text *before* you can even think
about deciding whether what is in it is valid or what it means.
We don't need to compress implicit meaning into the field names,
because we can have explicit fields, that all parsers must understand,
which make any extensions explicitly distinguishable from transmission
errors.
I think I'd actually like to understand why your system has trouble
with parsing this, because I'm sure there's an interesting clue there
somewhere, but I'm less sure it's a sign of trouble in the parts of
this protocol you gave as examples.
> > And I do think there *are* ways that we need to rearrange some of
> > these request structures. In particular, overloading "reg" to both
> > make requests and request changes is troublesome. I think it needs
> > to be split into separate query and change resources.
> >
> > The obvious problem case that evidenced this is in testing of a real
> > implementation is that if you send:
> >
> > { "resource": "reg", "delete": true }
> >
> > to the current Boulder implementation, it will happily return a 202
> > status, with the current registration resource as if you'd submitted
> > a "trivial payload" to actually query that state - with no indication
> > that something went wrong and the request you actually issued had
> > failed.
> >
> > I think that's a serious impediment to ever being able to extend the
> > registration resource functions in any backward compatible way and
> > we should address that now while we still have the chance. A request
> > to change something should never be misinterpreted as a request to do
> > something entirely different to that "successfully".
>
> I think that is spot on and comes up to 'criticality', one of those
> issues that people hate to raise because people have misunderstood it
> so badly in the past. I am pretty sure that a large fraction of those
> BULLRUN dollars were spent muddying the water with chatter about
> criticality.
>
> There are two ways that you might want a legacy application to react
> when faced with an extension:
>
> 1) Ignore the additional fields and continue as normal
> 2) Reject the request, do not attempt to process it.
>
> This is of course what the X.509 'CRITICAL' bit is intended to
> achieve. But people misinterpreted 'CRITICAL' and meaning 'important'
> and have as a result come up with idiotic specs that require CAs to
> cause legacy systems to break just for the hell of it. And probably
> not coincidentally, that idiot spec delayed deployment of a scheme
> that could have blocked the FLAME attack on Microsoft's internal CA.
>
> When we faced this issue in SAML, I had to disguise the criticality
> bit as 'Conditions' so that folk didn't complain.
I think in the example I gave above, the problem is much simpler,
(and one that I worry your syntax proposal might make worse :) ...
We've overloaded "reg" to mean more than one thing (both query and
modify), and the extension made which meaning was intended ambiguous.
I don't think we need a critical flag to fix that case, we just need
the request to do One Thing, unambiguously.
If "reg" only meant "modify this registration object", then the case
above could have just returned an "I didn't modify anything" status
even if it didn't recognise "delete", which wouldn't be misleading
at the client as to what happened.
I think we can actually build "criticality" into the protocol quite
reasonably here without needing a separate flag. For instance:
A "modify reg" request should quite reasonably be rejected if there
are fields which are not understood.
An unrecognised identifier "type" value can safely be "ignored" -
which may still lead to failure of the operation if no recognised
type is also included - but explicitly marks the mandatory field(s)
where optional extensions may be declared.
> Sometimes the desired legacy behavior is to have it reject the
> request, sometimes it is to carry on. This maps very nicely to the
> nested JSON scheme:
>
> * To cause additional fields to be ignored, keep the object type
> identifier tag the same.
> * To cause legacy software to reject, change the object type identifier tag
I think we can do that without the nesting, and I don't think the
nesting adds anything in general to avoiding this problem.
In many languages, JSON maps neatly to native data structures,
so a JSON object is trivially imported into a hash (or dictionary
or whatever your pet language calls them).
This makes looking for mandatory fields with well known names (like
"type") and deciding whether you recognise the value both trivial
to do and trivial to audit.
If we start telling people you have to iterate over every hash key
of every structure you receive, and reject if there is an element
that you don't recognise - then we're probably going to fail at
this, because nobody is going to do that in practice. Their code
is just never going to notice an element they aren't expecting.
We might get away with insisting on that for one or two critical
structures, but in general, if we want this to be widely implemented,
then if we want to extend something that can't be safely ignored, we
need to use a new indicator value in a well known mandatory field
like "type" or "resource".
That's the only way it will be noticed by "the simplest code", which
is what (some/many/most) people will write whatever we do.
> > The example you give here highlights the concern I expressed above.
> > In:
> >
> > "identifier": { "dns": { "value": "example.org"} },
> >
> > by getting rid of the "type" field, you may have made things easier
> > for your parser (while it is hardcoded with *all* of the valid field
> > names that might appear in an "identifier" object) but you've in turn
> > made it harder for everyone else to know what to do if they don't
> > recognise that field name, and difficult or impossible for any future
> > extension to that structure to occur.
>
> The rule 'ignore field names you don't understand' seems to fit the
> JSON approach. If a field is mandatory, you need to understand it.
I think where we disagree here is on the "bootstrap problem".
You are proposing that adding any new type immediately becomes a
field name that existing implementations won't recognise (and in
practice will probably never even notice is there). That makes
it harder than it needs to be to ever add new mandatory elements
because the "natural" behaviour of existing code will be to just
ignore them.
I'm thinking more along the lines of, we start with a fixed set
of mandatory fields, and a defined set of values they can have.
In that model, when we can define which of the mandatory field
values an implementation MUST recognise (like "resource"), and
which of them may contain values that MAY be ignored (like the
identifier "type") -- which is basically what the existing spec
does, modulo a few glitches we should iron out in it.
> Lets say that we want to introduce a new identifier type.
>
> "identifier": { "new": { "value": "example:id"} },
>
> The service can't understand the identifier type that it needs to
> understand to process the request, so the response here is 'I don't
> support that request'.
>
> If however we wanted to add some additional information into the
> identifier, such as a DNSSEC trust root (no, I don't know why you
> would, pretend), we have:
>
> "identifier": { "dns": { "value": "example.org",
> "dnssec" : "wh2iiy24iy2iuwrywiuy3riu2h3riuh=="} },
>
> Legacy services can simply ignore the additional information.
>
> I think this works very nicely with the challenge types. I can imagine
> us adding lots of additional fields over time that are not intended to
> break backwards compatibility. But we do have that option if it is
> needed.
I don't really see the advantage of that over, say:
"identifier": { "type": "dns", "value": "example.org",
"dnssec": "wh2iiy24iy2iuwrywiuy3riu2h3riuh==" },
You still need to parse all of the *text* of "identifier" before you
can assume anything contained in it is valid or means what you think
it means. Once you've done that, you have "type" and can use the
content to fill your static data types, without needing to recurse
through multiple objects to get at it.
The difference is, we can mandate up front that "type" MUST exist,
and how you must deal with any particular value in it (known or
unknown).
> > My code can no longer just go 'there's a value in the type field that
> > I don't recognise, but I know what "type" means and I can safely just
> > ignore that'. Now I have a field name that I don't recognise, or have
> > any idea what to do with, and have no "critical" flag to tell me if
> > I can safely ignore that or not.
>
> So we need to add the above explanation into section 5 and then
> identify the areas where we might want the option of breaking legacy
> systems rather than suffer their best efforts.
>
>
> > And I have a structure to parse with lots of redundancy. Why have
> > "dns" be an object that itself only has one field in it? Why bother
> > to wrap it in "identifier" if "dns" defines it as an identifier ...
>
> Question is whether this is an extension point or not. If it isn't
> then we can remove the hierarchy.
>
> { "blah" : "....",
> "identifier": { "dns": { "value": "example.org"} },
>
> becomes
>
> { "blah" : "....",
> "dns": { "value": "example.org"} },
I'd even be thinking:
{ "blah": "....", "dns": "example.org" }
If you are going to mandate the "one field" thing and that you must
understand what "dns" is, then "value" alone in an object adds nothing.
But this is why I think we're better off mandating "identifier" as a
well known and well defined field, with "type" as the mandatory field
that lets you recognise this is a valid identifier object and lets you
know if you support that type of identifier or not.
In your example above, if the new dnssec field was optional, then it
could still use "type": "dns" - and if it was mandatory, we'd add a
new "type": "dnssec" and define that requirement in the IANA spec for
that new type.
> > It wouldn't be impossible to fix those problems, but if the fix for
> > them is to define lots of objects that may only have one field, then
> > I don't really see that as being an improvement, and this isn't
> > really JSON anymore, it's more like Yet Another Text Representation
> > Of ASN.1, which I think would burn bridges for future extension that
> > we'd be better off not burning.
>
> I think we just need to understand where we want to put extension
> points. Right now, anywhere where we have a "type" tag approach seems
> right.
Yes, I think we can handle 'criticality' by defining whether an
unrecognised type can be ignored or should be cause for rejection
for each object where that is needed.
> ASN.1 certainly has a terrible approach to extensibility. The scheme I
> outline looks to me as if maintains the JSON approach of 'add fields
> at will' while still having enough mechanism there to be able to tell
> legacy services when they should not attempt to do their best with
> stuff they do not understand.
I might still be missing something that you're _thinking_ (though I
think I understand well enough what you've _said_ and the examples
you have given) - but my understanding of the problem you are trying
to solve with this syntax change is that your code has trouble with
deciding the type of something unless that is encoded implicitly in
its field name.
My biggest problem with the solution to that you are suggesting is
that it takes away the ability in general to know the type of something
(and whether it's a valid object in the general case), by replacing
well known defined fields which carry that information explicitly, with
arbitrary field names that may or may not contain it implicitly,
depending on whether the field name is valid but unrecognised, or just
line noise corruption ...
If I see a "type" I don't recognise in an otherwise valid object, then
I can report to the user "Unknown identifier type 'blah'" and give them
some useful indication of what to go look for.
If I can't distinguish that from a simply corrupted structure, then all
I can tell them is something more like "Error 184276, corrupt data", and
OpenSSL already inflicts enough of that sort of thing on them to want
more of it where it is easily avoidable.
I would like us to have a good look at the current JSON objects, and
what in them is and isn't mandatory and the failure modes, and what is
still underspecified, and what other data belongs in the JSON rather
than in the transport metadata. I think that's an important step
(along with getting more actual implementation experience) before we
can serious consider pushing this spec to last call.
I'm not seeing the concrete changes you propose above directly address
that (yet) though. I think it would be helpful if you more directly
directly describe the *problem* your code has with the existing syntax
though. So far I'm seeing a proposed solution to it, and some
rationalisation for that, but I'm not seeing the compelling benefit
that outweighs the problems it in turn introduces in other aspects.
This isn't a problem in general with parsing JSON into static data
types, so there's some other more specific issue if this is giving
you headaches trying to implement it in your framework, and it would
be nice to identify that clearly if we're going to try to accommodate
it better.
Basically, I don't think you can win at the sort of "layering to
define type" methodology you are proposing in this case, there is
always going to be some point in parsing JSON where you don't really
know the high level 'type' of something until you've first parsed the
text (and I say this as someone who is a fan of static typing and has
written code to mmap TIFFs into static structures to parse them ... ).
That isn't a model JSON fits into well, and I think the real choice
here is either "Don't use JSON", or "Work to the strengths of JSON"
and the latter option seems the better one to me at present. That's
the part where I see we are falling short by splitting some of the
protocol into the transport layer.
Cheers,
Ron
_______________________________________________
Acme mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/acme
