On Mon, Mar 14, 2016 at 6:14 PM, Tim Kientzle <kient...@gmail.com> wrote:

>
> > On Mar 14, 2016, at 11:50 AM, Feng Xiao <xiaof...@google.com> wrote:
> >
> >
> >
> > On Thu, Mar 3, 2016 at 6:37 PM, Tim Kientzle <kient...@gmail.com> wrote:
> >
> > I think your current Any JSON design requires that protobuf
> serialization be able to fail.  This is definitely a major change from
> proto2 and something the conformance test does not explore.
> >
> > 1. Decode a valid JSON-encoded message with an Any field in an unknown
> schema.
> >
> > 2. Attempt to encode it in protobuf format.
> >
> > If the descriptor cannot be fetched, then step #2 here must fail.
> >
> > (The alternative would be for #1 to fail in this case, but that would
> imply that JSON-to-JSON transcoding must fail in some circumstances.)
> > With the current Any JSON design, #1 must fail. It's the same for the
> other direction. Encoding an Any message with an unknown schema to JSON
> must fail as well.
>
> Interesting.  I knew that was true of the current C++ implementation; I
> did not realize that was an intentional part of the coding design.
>
> This should certainly be carefully documented in any.proto, since I expect
> some people will be quite surprised to find that decoding valid JSON and
> encoding the same data back to JSON will fail unless you have schema for
> the contents of every Any field.
>
We will publish a more detailed proto3 JSON spec which will describe the
expected behavior of JSON parsers/serializers for scenarios like this.
There are still quite some conformance test failures for our current
implementations though. We are working on fixing them right now and only
after that we can be sure the spec is finalized.


>
>
> > If Any had been defined slightly differently, then transcoding would be
> trivial.  If you are unable to change the current definition of Any, please
> consider introducing an alternative:
> >
> > package google.protobuf;
> >
> > message Other {
> >   string type_url = 1;
> >   oneof encoded {
> >      bytes protobuf = 2;
> >      google.protobuf.Value json = 3;
> >   }
> > }
> > This and similar proposals (like storing JSON in the value field and
> append a ":json" suffix to type URL to tell consumers the payload is JSON)
> had been considered in the design process, and it's decided to stay with
> the current Any representation. +liujisi may be able to provide more
> context around this decision.
> >
> > I think the advantage of the current Any representation is that the JSON
> to protobuf is done once in the proxy without the end users to worry about
> whether the Any payload is protobuf or JSON (i.e., it simplifies the end
> user APIs). With your proposal, the conversion from JSON to protobuf will
> spread out to all places. Every reader of the data must be able to do the
> conversion in order to read the data. That will complicate things quite a
> bit in a large system.
>
> With my proposal, the decision of how Any gets encoded is a contract
> between the originator of the data and the final consumer.  Intermediate
> systems can store, transcode, and forward the data without having any
> knowledge of that schema.
>
I can't say for sure for your environment, but inside Google, the same
piece of data will be read by not only yourself but a lot of others as
well. Actually some protos are used so widely that not a single team can
claim itself as the owner of the proto. There are almost always multiple
parties that are generating and consuming the data for a same proto file.
It's also not uncommon to see intermediate systems wanting to know some
more details in the payload. Having two formats in Any will definitely
increase the complexity.


>
> I was considering Any for cases such as logging, data storage, and other
> systems where intermediaries do not need to access or understand the
> additional data.  This was a common use of extensions in proto2:
> Intermediate handlers would decode and act on a limited set of well-known
> fields; additional data would be simply carried through until a final
> consumer decoded and acted on it.  Any satisfies this use case for
> protobuf-only systems but not for systems that use JSON.

If you only use protobuf format, Any is mostly the same as extensions. If
you are using JSON, proto2 extension doesn't help solve the issue at all. I
think the problem is not that Any does not allow intermediate proxies to
pass through the data without knowing the type info, but that you are
trying to mix JSON and protobuf format in the same system. In google, we
are using a front-end proxy to convert JSON to/from protobuf, and
everything else behind it is using protobuf only. The current Any design
works with this approach very well. Not sure whether this works for your
case, but it's something worth considering.




>
> Tim
>
>
>
> >  This is trivial to transcode:  it only requires that the final consumer
> be able to decode whatever format the originator attached.  Intermediate
> systems can freely transcode without needing to understand the schema at
> all, since the ‘protobuf’ field here can be represented in JSON and the
> ‘json’ field can be encoded to protobuf.  The actual JSON serialization is
> very similar to the current Any serialization (with the addition of the
> extra “json” tag).
> >
> > Cheers,
> >
> > Tim
> >
> >
> >
> > > On Feb 5, 2016, at 1:53 PM, Josh Haberman <haber...@google.com> wrote:
> > >
> > > Hi Tim,
> > >
> > > I think your analysis is correct. The answer to your question is that
> it's not possible to transcode Any from JSON <-> protobuf unless you have
> the schema. That is one of the reasons why the type URL exists as a way to
> fetch schemas for types you otherwise wouldn't know about.
> > >
> > > I'm glad the conformance suite was helpful!
> > >
> > > Best,
> > > Josh
> > >
> > > On Wednesday, February 3, 2016 at 1:37:40 PM UTC-8, Tim Kientzle wrote:
> > > I’m trying to implement Any, including proper JSON and protobuf
> encoding, and the following scenario has me stumped:
> > >
> > >
> > >
> > > I’m looking at the any.proto file, which includes this example:
> > >
> > >
> > >
> > > Sample proto file:
> > >
> > > //     package google.profile;
> > >
> > > //     message Person {
> > >
> > > //       string first_name = 1;
> > >
> > > //       string last_name = 2;
> > >
> > > //     }
> > >
> > >
> > >
> > > Resulting JSON encoding:
> > >
> > > //     {
> > >
> > > //       "@type": "type.googleapis.com/google.profile.Person",
> > >
> > > //       "firstName": <string>,
> > >
> > > //       "lastName": <string>
> > >
> > > //     }
> > >
> > >
> > >
> > > The scenario I’m trying to understand involves three systems:
> > >
> > >
> > >
> > > Originating system:
> > >
> > > * Knows google.profile and writes out the JSON above
> > >
> > >
> > >
> > > Middle system:
> > >
> > > * Reads the JSON example above
> > >
> > > * Does *not* know google.profile (doesn’t have the descriptor and has
> no way to get it)
> > >
> > > * Writes out protobuf
> > >
> > >
> > >
> > > Receiving system:
> > >
> > > * Knows google.profile and reads the protobuf
> > >
> > >
> > >
> > > In particular:  What is the protobuf serialization between the middle
> and receiving systems?
> > >
> > >
> > >
> > > Without the descriptor, the middle system cannot poduce the “real"
> protobuf binary format (the field numbers aren’t in that format), so the
> only option would seem to be to serialize the JSON data as a protobuf
> Struct representation (which in turn has interesting implications for the
> receiving system). Am I missing some other option?
> > >
> > >
> > >
> > > Tim
> > >
> > >
> > >
> > > P.S.  Nice work on the conformance test suite, by the way!  That
> clarified a *lot* of details about the expected handling for protobuf JSON
> format.
> > >
> > >
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+unsubscr...@googlegroups.com.
> > > To post to this group, send email to protobuf@googlegroups.com.
> > > Visit this group at https://groups.google.com/group/protobuf.
> > > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+unsubscr...@googlegroups.com.
> > To post to this group, send email to protobuf@googlegroups.com.
> > Visit this group at https://groups.google.com/group/protobuf.
> > For more options, visit https://groups.google.com/d/optout.
> >
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Reply via email to