Re: [protobuf] Embedding arbitrary JSON in a message
If you are using JavaScript, there may not be a good solution. Ironically,
the JavaScript implementation of protos does not support the canonical JSON
representation for protobuf types (which would be needed to convert
arbitrary JSON into a google.protobuf.Value).
Most other languages, though, should work fine. Here's a sketch in Go:
*example.proto*
syntax = "proto3";
import "google/protobuf/struct.proto";
message SomeMessage {
google.protobuf.Value arbitrary_json = 1;
}
*example.go*
package example
import "github.com/golang/protobuf/ptypes/struct"
import "github.com/golang/protobuf/jsonpb"
func example(json string) (*SomeMessage, error) {
msg := {ArbitraryJson: {}}
jsm := jsonpb.Marshaler{}
if err := jsm.Unmarshal(json, msg.ArbitraryJson); err != nil {
return nil, err
}
return msg, nil
}
You could also use the "encoding/json" package to then convert any kind of
Go value into a google.protobuf.Value: you just have to marshal to JSON
first, and from there into the protobuf. While using JSON as intermediate
format is not efficient, you do get the benefit that a
google.protobuf.Value created this way is guaranteed to be well-formed,
whereas just using a string or bytes field could be garbage that is not
proper JSON.
*Josh Humphries*
jh...@bluegosling.com
On Wed, Sep 26, 2018 at 9:44 AM Jimit Modi wrote:
> Hello @Siddharth, @John,
>
> Did you guys found any solution around it. We are also stuck at very same
> thing.
>
> On Friday, August 17, 2018 at 3:36:29 PM UTC+5:30, Siddharth Kherada wrote:
>>
>> Can I please get an example of how to do this is any language with proto
>> file?
>>
>> I desperately need help with this. I am stuck for a week now on this
>> issue.
>>
>> Thanks,
>> Sid
>>
>> On Thursday, 21 June 2018 15:32:46 UTC-4, Josh Humphries wrote:
>>>
>>> Oops, I meant to point you to google.protobuf.Value:
>>> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L63
>>>
>>> It can represent *any* kind of JSON value. The Struct type is what is
>>> used to represent JSON *objects* (there is also ListValue, for arrays,
>>> as well as support for JS primitive types and null).
>>>
>>>
>>>
>>> *Josh Humphries*
>>> jh...@bluegosling.com
>>>
>>> On Thu, Jun 21, 2018 at 3:25 PM, Josh Humphries
>>> wrote:
>>>
Hi, John,
Take a look at the well-known type google.protobuf.Struct. It is
basically a JSON value, modeled as a proto. It's JSON representation is
exactly what you want, too:
https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L52
*Josh Humphries*
jh...@bluegosling.com
On Thu, Jun 21, 2018 at 3:20 PM, John Lilley wrote:
> Disclaimer: I am totally new to protobuf, engaged in an exploratory
> POC. Please forgive dumb questions :-)
>
> We are looking at migrating an existing, JSON-based protocol in which
> hand-coded C++ is written to perform serdes between objects and JSON. We
> want to replace the hand-coding with an automated approach that can be
> shared between C++ and Java. However, a stumbling block I see is that
> some
> messages have an arbitrary field full of JSON like:
>
> {
>"name":"john",
>"address":"123 main st",
>"attributes":{ any JSON can go here }
> }
>
> While I realize that we could stringify the JSON, this breaks our
> published API. Is there any way I can use protobuf to perform serdes
> between message like this and some struct like:
>
> {
>string name;
>string address;
>json attributes;
> }
>
> I'm even OK if the internal data is stringified JSON:
>
> {
>string name;
>string address;
>string attributes;
> }
>
> So long as the exchanged JSON isn't stringified. In other words, this
> is bad:
> {
>"name":"john",
>"address":"123 main st",
>"attributes":"{ \"attr1\":\"value1\", \"attr2\":[\"elem1\",
> \"elem2\"] }"
> }
>
> It needs to be exchanged like
> {
>"name":"john",
>"address":"123 main st",
>"attributes":{ "attr1":"value1", "attr2":["elem1", "elem2"] }"
> }
>
> Is this possible?
>
> Thanks
> john
>
> --
> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop
Re: [protobuf] Embedding arbitrary JSON in a message
Hello @Siddharth, @John,
Did you guys found any solution around it. We are also stuck at very same
thing.
On Friday, August 17, 2018 at 3:36:29 PM UTC+5:30, Siddharth Kherada wrote:
>
> Can I please get an example of how to do this is any language with proto
> file?
>
> I desperately need help with this. I am stuck for a week now on this issue.
>
> Thanks,
> Sid
>
> On Thursday, 21 June 2018 15:32:46 UTC-4, Josh Humphries wrote:
>>
>> Oops, I meant to point you to google.protobuf.Value:
>> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L63
>>
>> It can represent *any* kind of JSON value. The Struct type is what is
>> used to represent JSON *objects* (there is also ListValue, for arrays,
>> as well as support for JS primitive types and null).
>>
>>
>>
>> *Josh Humphries*
>> jh...@bluegosling.com
>>
>> On Thu, Jun 21, 2018 at 3:25 PM, Josh Humphries > > wrote:
>>
>>> Hi, John,
>>> Take a look at the well-known type google.protobuf.Struct. It is
>>> basically a JSON value, modeled as a proto. It's JSON representation is
>>> exactly what you want, too:
>>>
>>>
>>> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto#L52
>>>
>>>
>>>
>>>
>>> *Josh Humphries*
>>> jh...@bluegosling.com
>>>
>>> On Thu, Jun 21, 2018 at 3:20 PM, John Lilley >> > wrote:
>>>
Disclaimer: I am totally new to protobuf, engaged in an exploratory
POC. Please forgive dumb questions :-)
We are looking at migrating an existing, JSON-based protocol in which
hand-coded C++ is written to perform serdes between objects and JSON. We
want to replace the hand-coding with an automated approach that can be
shared between C++ and Java. However, a stumbling block I see is that
some
messages have an arbitrary field full of JSON like:
{
"name":"john",
"address":"123 main st",
"attributes":{ any JSON can go here }
}
While I realize that we could stringify the JSON, this breaks our
published API. Is there any way I can use protobuf to perform serdes
between message like this and some struct like:
{
string name;
string address;
json attributes;
}
I'm even OK if the internal data is stringified JSON:
{
string name;
string address;
string attributes;
}
So long as the exchanged JSON isn't stringified. In other words, this
is bad:
{
"name":"john",
"address":"123 main st",
"attributes":"{ \"attr1\":\"value1\", \"attr2\":[\"elem1\",
\"elem2\"] }"
}
It needs to be exchanged like
{
"name":"john",
"address":"123 main st",
"attributes":{ "attr1":"value1", "attr2":["elem1", "elem2"] }"
}
Is this possible?
Thanks
john
--
You received this message because you are subscribed to the Google
Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to protobuf+u...@googlegroups.com .
To post to this group, send email to prot...@googlegroups.com
.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Documentation Riddle
On Wed, Sep 26, 2018 at 3:42 AM omid pourhadi
wrote:
> I have some questions :
>
> 1. Is it a good idea to use Any for transferring a serialized/deserialized
> java object from client to server for example consider this ?
>
A google.protobuf.Any encapsulates a *protobuf message *whose type can
vary, not an arbitrary type from an arbitrary language. So, no, you can't
use it to represent a serialized Java object because you won't have a valid
type URL for that object. The type URL is meant to refer to a protobuf type.
Instead, you would need to use bytes. Or you could create your own type
that is similar to Any:
message Object {
string type = 1;
bytes value = 2;
}
Other options are to use google.protobuf.Struct, but that is limited to
what can be represented by a JSON object. If your Java objects have a
canonical JSON representation then this might be a useful solution.
The final alternative is to create your own set of generic messages, very
similar to the types in "google/protobuf/struct.proto" but for modeling
Java objects in a higher-fidelity manner than just as serialized bytes.
>
> import "google/protobuf/empty.proto";
> import "google/protobuf/any.proto";
> import "google/protobuf/type.proto";
>
> service AcceptorService {
> rpc search(google.protobuf.Any) returns (google.protobuf.ListValue) {};
>
> }
>
> 2. How can I use ListValue ?
>
The google.protobuf.ListValue type can only represent the same category of
values that you can represent with a JSON Array. So it can't represent a
list of strongly-typed values. (Instead, the best you could do is a list of
google.protobuf.Struct to represent a List). So if you instead are
trying to represent a Java List, you'd be better off creating your
own type:
message ListOfObject {
// this example uses bytes, serialized Java objects, as the values
// but could reference other types as mentioned above for how you
// might represent an arbitrary Java object in protobuf
repeated bytes objects = 1;
}
You could then serialize Java objects into elements of the objects field.
>
> 3. How can I define a Map in a message (I found
> MapOfObjects in maps.proto but it is using string)?
>
I don't know what maps.proto is. Perhaps you are working with a set of
message types where someone else has already tried to model much of your
Java domain objects already? Anyhow, you'd need to do something like
mentioned above: map where values are serialized Java
objects (or some other value type, based on how you decide to model an
arbitrary Java object).
FWIW, a generally better idea is to limit the domain of what kind of object
can be transferred and model all of them as messages in proto. That would
allow you to use google.protobuf.Any instead of bytes. You could then
define your list and maps as:
message dummy {
repeated google.protobuf.Any list = 1;
map map = 2;
}
Note however that maps in protobuf are limited to integral, bool, and
string keys: you cannot use messages or bytes values as the key for a
protobuf map (unlike in Java, where anything can be a map key). If you have
a data structure like that in Java, you'd instead have to model it like so:
// this example is for Map
message MapObjectStringEntry {
google.protobuf.Any key = 1;
string value = 2;
}
message dummy {
repeated MapObjectStringEntry map = 1;
}
Hope this helps. Good luck.
> message dummy{
> map filters = 1;
> }
>
> Thanks in advanced
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Documentation Riddle
I have some questions :
1. Is it a good idea to use Any for transferring a serialized/deserialized
java object from client to server for example consider this ?
import "google/protobuf/empty.proto";
import "google/protobuf/any.proto";
import "google/protobuf/type.proto";
service AcceptorService {
rpc search(google.protobuf.Any) returns (google.protobuf.ListValue) {};
}
2. How can I use ListValue ?
3. How can I define a Map in a message (I found
MapOfObjects in maps.proto but it is using string)?
message dummy{
map filters = 1;
}
Thanks in advanced
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
