> On Mar 14, 2016, at 11:50 AM, Feng Xiao <xiaof...@google.com> wrote:
> 
> 
> 
> On Thu, Mar 3, 2016 at 6:37 PM, Tim Kientzle <kient...@gmail.com> wrote:
> 
> I think your current Any JSON design requires that protobuf serialization be 
> able to fail.  This is definitely a major change from proto2 and something 
> the conformance test does not explore.
> 
> 1. Decode a valid JSON-encoded message with an Any field in an unknown schema.
> 
> 2. Attempt to encode it in protobuf format.
> 
> If the descriptor cannot be fetched, then step #2 here must fail.
> 
> (The alternative would be for #1 to fail in this case, but that would imply 
> that JSON-to-JSON transcoding must fail in some circumstances.)
> With the current Any JSON design, #1 must fail. It's the same for the other 
> direction. Encoding an Any message with an unknown schema to JSON must fail 
> as well.

Interesting.  I knew that was true of the current C++ implementation; I did not 
realize that was an intentional part of the coding design.

This should certainly be carefully documented in any.proto, since I expect some 
people will be quite surprised to find that decoding valid JSON and encoding 
the same data back to JSON will fail unless you have schema for the contents of 
every Any field.


> If Any had been defined slightly differently, then transcoding would be 
> trivial.  If you are unable to change the current definition of Any, please 
> consider introducing an alternative:
> 
> package google.protobuf;
> 
> message Other {
>   string type_url = 1;
>   oneof encoded {
>      bytes protobuf = 2;
>      google.protobuf.Value json = 3;
>   }
> }
> This and similar proposals (like storing JSON in the value field and append a 
> ":json" suffix to type URL to tell consumers the payload is JSON) had been 
> considered in the design process, and it's decided to stay with the current 
> Any representation. +liujisi may be able to provide more context around this 
> decision.
> 
> I think the advantage of the current Any representation is that the JSON to 
> protobuf is done once in the proxy without the end users to worry about 
> whether the Any payload is protobuf or JSON (i.e., it simplifies the end user 
> APIs). With your proposal, the conversion from JSON to protobuf will spread 
> out to all places. Every reader of the data must be able to do the conversion 
> in order to read the data. That will complicate things quite a bit in a large 
> system.

With my proposal, the decision of how Any gets encoded is a contract between 
the originator of the data and the final consumer.  Intermediate systems can 
store, transcode, and forward the data without having any knowledge of that 
schema.

I was considering Any for cases such as logging, data storage, and other 
systems where intermediaries do not need to access or understand the additional 
data.  This was a common use of extensions in proto2:  Intermediate handlers 
would decode and act on a limited set of well-known fields; additional data 
would be simply carried through until a final consumer decoded and acted on it. 
 Any satisfies this use case for protobuf-only systems but not for systems that 
use JSON.

Tim



>  This is trivial to transcode:  it only requires that the final consumer be 
> able to decode whatever format the originator attached.  Intermediate systems 
> can freely transcode without needing to understand the schema at all, since 
> the ‘protobuf’ field here can be represented in JSON and the ‘json’ field can 
> be encoded to protobuf.  The actual JSON serialization is very similar to the 
> current Any serialization (with the addition of the extra “json” tag).
> 
> Cheers,
> 
> Tim
> 
> 
> 
> > On Feb 5, 2016, at 1:53 PM, Josh Haberman <haber...@google.com> wrote:
> >
> > Hi Tim,
> >
> > I think your analysis is correct. The answer to your question is that it's 
> > not possible to transcode Any from JSON <-> protobuf unless you have the 
> > schema. That is one of the reasons why the type URL exists as a way to 
> > fetch schemas for types you otherwise wouldn't know about.
> >
> > I'm glad the conformance suite was helpful!
> >
> > Best,
> > Josh
> >
> > On Wednesday, February 3, 2016 at 1:37:40 PM UTC-8, Tim Kientzle wrote:
> > I’m trying to implement Any, including proper JSON and protobuf encoding, 
> > and the following scenario has me stumped:
> >
> >
> >
> > I’m looking at the any.proto file, which includes this example:
> >
> >
> >
> > Sample proto file:
> >
> > //     package google.profile;
> >
> > //     message Person {
> >
> > //       string first_name = 1;
> >
> > //       string last_name = 2;
> >
> > //     }
> >
> >
> >
> > Resulting JSON encoding:
> >
> > //     {
> >
> > //       "@type": "type.googleapis.com/google.profile.Person",
> >
> > //       "firstName": <string>,
> >
> > //       "lastName": <string>
> >
> > //     }
> >
> >
> >
> > The scenario I’m trying to understand involves three systems:
> >
> >
> >
> > Originating system:
> >
> > * Knows google.profile and writes out the JSON above
> >
> >
> >
> > Middle system:
> >
> > * Reads the JSON example above
> >
> > * Does *not* know google.profile (doesn’t have the descriptor and has no 
> > way to get it)
> >
> > * Writes out protobuf
> >
> >
> >
> > Receiving system:
> >
> > * Knows google.profile and reads the protobuf
> >
> >
> >
> > In particular:  What is the protobuf serialization between the middle and 
> > receiving systems?
> >
> >
> >
> > Without the descriptor, the middle system cannot poduce the “real" protobuf 
> > binary format (the field numbers aren’t in that format), so the only option 
> > would seem to be to serialize the JSON data as a protobuf Struct 
> > representation (which in turn has interesting implications for the 
> > receiving system). Am I missing some other option?
> >
> >
> >
> > Tim
> >
> >
> >
> > P.S.  Nice work on the conformance test suite, by the way!  That clarified 
> > a *lot* of details about the expected handling for protobuf JSON format.
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Protocol Buffers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to protobuf+unsubscr...@googlegroups.com.
> > To post to this group, send email to protobuf@googlegroups.com.
> > Visit this group at https://groups.google.com/group/protobuf.
> > For more options, visit https://groups.google.com/d/optout.
> 
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Reply via email to