Re: [protobuf] protoc --decode_raw produces incorrect output

[Ilia Mirkin]

Wire types 3 and 4 are for "groups", which were only a thing in
proto1, and deprecated ever since. These were a lot like messages, but
could not stand on their own, and could only be embedded at the
appropriate points as I recall.

Some more info here:
https://developers.google.com/protocol-buffers/docs/encoding

Cheers,

  -ilia

On Wed, May 19, 2021 at 11:25 AM Barry waldbaum
 wrote:
>
> Hi! Thanks for your response!
>
> Wouldn't wire type 3 be the embedded logic? wire type 2 length delimited data 
> shouldn't be parsed and just printed out.
>
> thanks
> -Barry
>
>
>
>
> On Wednesday, May 19, 2021 at 11:12:01 AM UTC-4 Ilia Mirkin wrote:
>>
>> I haven't looked at the decode_raw logic either, but wire type 2 is
>> used for all length-delimited values. That's strings, bytes, and
>> embedded messages (and packed repeated fields). decode_raw doesn't
>> know when it's an embedded message and when it's a string/bytes. So it
>> has to guess somehow, based on some heuristics (like how well the
>> values decode, presumably). And here it guesses wrong.
>>
>> Cheers,
>>
>> -ilia
>>
>> On Wed, May 19, 2021 at 10:40 AM Barry waldbaum
>>  wrote:
>> >
>> > I've been building tools to generate generic protobuf data from json (yes, 
>> > I'm aware I lose context, this is for testing)
>> >
>> > I've found a message I can parse properly, but protoc --decode_raw can't:
>> >
>> > \n \a 3 4 5 0 0 0 0
>> >
>> > protoc
>> > 1 {
>> > 6 {
>> > }
>> > 6: 0x30303030
>> > }
>> >
>> > to reproduce it:
>> >
>> > echo -n $'\x0a' > binary.dat; echo -n $'\x07' >> binary.dat; echo -n 
>> > $'345' >> binary.dat ; protoc --decode_raw < binary.dat
>> >
>> > I'm able to parse this properly using protowire as:
>> > 1: "345"
>> >
>> > also if I change the first character in the bytes to a '1' I get a valid 
>> > output:
>> >
>> > $ echo -n $'\x0a' > binary.dat; echo -n $'\x07' >> binary.dat; echo -n 
>> > $'145' >> binary.dat ; protoc --decode_raw < binary.dat
>> > 1: "145"
>> >
>> > $ protoc --version
>> > libprotoc 3.15.8
>> >
>> > I've worked really hard to keep the reproduction as simple as possible, I 
>> > haven't dug into the code for decode_raw yet, that's my next step..
>> >
>> > thanks
>> > -Barry
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an 
>> > email to protobuf+u...@googlegroups.com.
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/protobuf/2ccfbf93-6b0e-49d5-ae8e-3b3dc1b55218n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/protobuf/a44d0cc7-ac54-4ad5-830c-d0cc8ab42573n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAKb7UvgBaO3HUGS%2ByM45pP2o3h2ttA6zqUxB2GJPUZO5PA%3DRQg%40mail.gmail.com.

Re: [protobuf] protoc --decode_raw produces incorrect output

[Ilia Mirkin]

I haven't looked at the decode_raw logic either, but wire type 2 is
used for all length-delimited values. That's strings, bytes, and
embedded messages (and packed repeated fields). decode_raw doesn't
know when it's an embedded message and when it's a string/bytes. So it
has to guess somehow, based on some heuristics (like how well the
values decode, presumably). And here it guesses wrong.

Cheers,

  -ilia

On Wed, May 19, 2021 at 10:40 AM Barry waldbaum
 wrote:
>
> I've been building tools to generate generic protobuf data from json (yes, 
> I'm aware I lose context, this is for testing)
>
> I've found a message I can parse properly, but protoc --decode_raw can't:
>
>  \n  \a   3   4   5   0   0   0   0
>
> protoc
> 1 {
>   6 {
>   }
>   6: 0x30303030
> }
>
> to reproduce it:
>
> echo -n $'\x0a' > binary.dat; echo -n $'\x07' >> binary.dat; echo -n 
> $'345' >> binary.dat ; protoc --decode_raw < binary.dat
>
> I'm able to parse this properly using protowire as:
> 1: "345"
>
> also if I change the first character in the bytes to a '1' I get a valid 
> output:
>
> $ echo -n $'\x0a' > binary.dat; echo -n $'\x07' >> binary.dat; echo -n 
> $'145' >> binary.dat ; protoc --decode_raw < binary.dat
> 1: "145"
>
> $ protoc --version
> libprotoc 3.15.8
>
> I've worked really hard to keep the reproduction as simple as possible, I 
> haven't dug into the code for decode_raw yet, that's my next step..
>
> thanks
> -Barry
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/protobuf/2ccfbf93-6b0e-49d5-ae8e-3b3dc1b55218n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAKb7Uvj1OkLK%3D%2BRf%3D5A9%3Dhkj4OBw03QvHd7aR%3DsaL%3Dv9jS1Kgw%40mail.gmail.com.

Re: [protobuf] Protocol Buffers Encoding Guide erratum

[Ilia Mirkin]

On Thu, May 6, 2021 at 5:06 PM Gregory Bourassa
 wrote:
>
> Hi folks,
>
> Has anyone noticed that the example message which begins the document is 
> erroneous.   At one point (under the Message Structure heading) the authors 
> claim that:
>
> 96 01 = 1001 0110   0001
>
> which it cannot,  since 1001 0110 in radix 10 is 150.

But it's 96 in radix 16. You always use hex when talking about file /
bit encodings, never base 10...

Cheers,

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAKb7UvgnrFCCxB7fyUsAT3T7e0ShJoK5ggsjm82Y4koRm1yV5A%40mail.gmail.com.

Re: [protobuf] Understanding the binary encoding: Why is there a "01" after certain keys?

[Ilia Mirkin]

What if the field number were 32? (32 << 3) | 2 = 0x102, which won't
fit in a byte. The field number is varint-encoded, and since the high
bit is set with 16+ (i.e. 0x80), the tag id ends up taking 2 (or more)
bytes -- with varint encoding, if the high bit is set, that means the
remaining bits are in the next byte. If space is a big concern, use
tag ids under 16 for the most-often used data.

Cheers,

  -ilia

On Wed, Sep 30, 2020 at 5:04 PM Juan Cruz Viotti  wrote:
>
> I'm trying to understand the binary encoding as explained in 
> https://developers.google.com/protocol-buffers/docs/encoding.
>
> I have the following schema that models a GitHub user as per the GitHub API:
>
> syntax = "proto3";
>
> message GitHubUser {
>   string login = 1;
>   uint32 id = 2;
>   string node_id = 3;
>   string avatar_url = 4;
>   string gravatar_id = 5;
>   string url = 6;
>   string html_url = 7;
>   string followers_url = 8;
>   string following_url = 9;
>   string gists_url = 10;
>   string starred_url = 11;
>   string subscriptions_url = 12;
>   string organizations_url = 13;
>   string repos_url = 14;
>   string events_url = 15;
>   string received_events_url = 16;
>   string type = 17;
>   bool site_admin = 18;
> }
>
> And the following JSON document:
>
> {
>   "login": "octocat",
>   "id": 1,
>   "node_id": "MDQ6VXNlcjE=",
>   "avatar_url": "https://github.com/images/error/octocat_happy.gif";,
>   "gravatar_id": "",
>   "url": "https://api.github.com/users/octocat";,
>   "html_url": "https://github.com/octocat";,
>   "followers_url": "https://api.github.com/users/octocat/followers";,
>   "following_url": 
> "https://api.github.com/users/octocat/following{/other_user}";,
>   "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}";,
>   "starred_url": 
> "https://api.github.com/users/octocat/starred{/owner}{/repo}";,
>   "subscriptions_url": "https://api.github.com/users/octocat/subscriptions";,
>   "organizations_url": "https://api.github.com/users/octocat/orgs";,
>   "repos_url": "https://api.github.com/users/octocat/repos";,
>   "events_url": "https://api.github.com/users/octocat/events{/privacy}";,
>   "received_events_url": 
> "https://api.github.com/users/octocat/received_events";,
>   "type": "User",
>   "site_admin": false
> }
>
> The produced payload using the Python library is the following:
>
> : 0a07 6f63 746f 6361 7410 011a 0c4d 4451  ..octocatMDQ
> 0010: 3656 584e 6c63 6a45 3d22 3168 7474 7073  6VXNlcjE="1https
> 0020: 3a2f 2f67 6974 6875 622e 636f 6d2f 696d  ://github.com/im
> 0030: 6167 6573 2f65 7272 6f72 2f6f 6374 6f63  ages/error/octoc
> 0040: 6174 5f68 6170 7079 2e67 6966 3224 6874  at_happy.gif2$ht
> 0050: 7470 733a 2f2f 6170 692e 6769 7468 7562  tps://api.github
> 0060: 2e63 6f6d 2f75 7365 7273 2f6f 6374 6f63  .com/users/octoc
> 0070: 6174 3a1a 6874 7470 733a 2f2f 6769 7468  at:.https://gith
> 0080: 7562 2e63 6f6d 2f6f 6374 6f63 6174 422e  ub.com/octocatB.
> 0090: 6874 7470 733a 2f2f 6170 692e 6769 7468  https://api.gith
> 00a0: 7562 2e63 6f6d 2f75 7365 7273 2f6f 6374  ub.com/users/oct
> 00b0: 6f63 6174 2f66 6f6c 6c6f 7765 7273 4a3b  ocat/followersJ;
> 00c0: 6874 7470 733a 2f2f 6170 692e 6769 7468  https://api.gith
> 00d0: 7562 2e63 6f6d 2f75 7365 7273 2f6f 6374  ub.com/users/oct
> 00e0: 6f63 6174 2f66 6f6c 6c6f 7769 6e67 7b2f  ocat/following{/
> 00f0: 6f74 6865 725f 7573 6572 7d52 3468 7474  other_user}R4htt
> 0100: 7073 3a2f 2f61 7069 2e67 6974 6875 622e  ps://api.github.
> 0110: 636f 6d2f 7573 6572 732f 6f63 746f 6361  com/users/octoca
> 0120: 742f 6769 7374 737b 2f67 6973 745f 6964  t/gists{/gist_id
> 0130: 7d5a 3b68 7474 7073 3a2f 2f61 7069 2e67  }Z;https://api.g
> 0140: 6974 6875 622e 636f 6d2f 7573 6572 732f  ithub.com/users/
> 0150: 6f63 746f 6361 742f 7374 6172 7265 647b  octocat/starred{
> 0160: 2f6f 776e 6572 7d7b 2f72 6570 6f7d 6232  /owner}{/repo}b2
> 0170: 6874 7470 733a 2f2f 6170 692e 6769 7468  https://api.gith
> 0180: 7562 2e63 6f6d 2f75 7365 7273 2f6f 6374  ub.com/users/oct
> 0190: 6f63 6174 2f73 7562 7363 7269 7074 696f  ocat/subscriptio
> 01a0: 6e73 6a29 6874 7470 733a 2f2f 6170 692e  nsj)https://api.
> 01b0: 6769 7468 7562 2e63 6f6d 2f75 7365 7273  github.com/users
> 01c0: 2f6f 6374 6f63 6174 2f6f 7267 7372 2a68  /octocat/orgsr*h
> 01d0: 7474 7073 3a2f 2f61 7069 2e67 6974 6875  ttps://api.githu
> 01e0: 622e 636f 6d2f 7573 6572 732f 6f63 746f  b.com/users/octo
> 01f0: 6361 742f 7265 706f 737a 3568 7474 7073  cat/reposz5https
> 0200: 3a2f 2f61 7069 2e67 6974 6875 622e 636f  ://api.github.co
> 0210: 6d2f 7573 6572 732f 6f63 746f 6361 742f  m/users/octocat/
> 0220: 6576 656e 7473 7b2f 7072 6976 6163 797d  events{/privacy}
> 0230: 8201 3468 7474 7073 3a2f 2f61 7069 2e67  ..4https://api.g
> 0240: 6974 6875 622e 636f 6d2f 7573 6572 732f  ithub.com/users/
> 0250: 6f63 746f 6361 742f 7265 6365 6976 6564  octocat/re

Re: [protobuf] has() methods in Java

[Ilia Mirkin]

These are gone in proto3. The has* stuff is available with proto2.

On Thu, Jul 11, 2019 at 4:54 PM Janac Meena  wrote:
>
> Hello,
>
> I see in the documentation that there is a way to check for the existence of 
> a field in a message, i.e.
>
> message Foo {
>  string name = 1;
> }
>
> In my Java code, I was hoping to do
>
> // Assuming object "foo" has been initialized
>
> if (foo.hasName()) {
> // do stuff
> }
>
> But I am unable to find the hasX() methods.
>
> What's going on?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/protobuf/1d700a63-f677-4c56-8c47-fa50d3d5468f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAKb7UvibG7u14Fpgnz-Em6Pdv_248h5jOc0wQ7k2bZxbnS89DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Find the message Type from Name of the message

[Ilia Mirkin]

You can get the ->message_type() (assuming the ->type() is
TYPE_MESSAGE), which gets you a Descriptor which in turn has a
->name() / ->full_name() function.

Cheers,

  -ilia

On Tue, May 21, 2019 at 7:13 PM Kiran Kumar  wrote:
>
> I have protobuf messages defined as below. I need to find the message type 
> from the attribute name. For example, when the input is "cfgMsg" the output 
> should be ConfigMsg or CfgServerMsg.ConfigMsg (full name).
>
>
> message CfgServerMsg {
>   string name = 1;
>   ConfigMsg cfgMsg = 2;
> }
>
> message ConfigMsg {
>   string cfgName = 1;
>   uint32 msgId = 2;
> }
>
>
> I have the below code. However, this is working for well defined types like 
> string, int, float etc. and for messages it just prints "message" as the 
> output.
>
> I removed some code and presented only that is relevant to this question. So 
> this is obviously not the complete code.
>
>
> google::protobuf::Message *modObj = new ModObj();
>
> const google::protobuf::Descriptor *outModDesc
> =  modObj->GetDescriptor();
> const Reflection *outModRefl = modObj->GetReflection();
> const FieldDescriptor *field;
>
> // Loop to iterate over all the fields
> {
>   field = outModDesc->FindFieldByName(tmp_name);
>   std::string type = field->type_name();
>   std::cout << "Type:" << type << std::endl;
> }
>
>
> Output:
>
> Type:string
>
> Type:message
>
>
> However, I want to get the actual message type which is "ConfigMsg" instead 
> of just "message". Is there any such API available from protobuf ?
>
>
> I did check this page 
> https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor#FileDescriptor.name.details
>  thoroughly, but couldn't find any thing useful for this.
>
>
> If anyone has done similar thing or know some thing around this, it would be 
> useful.
>
>
> Thanks,
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/protobuf/68cf3dd3-8b5e-4f5c-8b87-b708b4e787da%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAKb7Uvi-xrwhRSZc%2BRXrro%2B70hr9-mb7pB5HmHumtA92fEE9Zw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf Java Generated Code field name has underscore at the end.

[Ilia Mirkin]

Why not just store the serialized data of the protobuf instead? That's
kind of the whole point of protobuf...

On Thu, Sep 6, 2018 at 1:27 PM, Chris Zhang  wrote:
> Hi Adam,
>
> Thanks for the response.
>
> We are trying to persist the protobuf generated java object into mongoDB
> using Spring framework.
> However, when doing the querying from database, the spring framework does
> not support any field name with underscore.
>
> Is there anyway we can work around?
>
> Thanks.
>
>
> On Thursday, September 6, 2018 at 1:05:38 PM UTC-4, Adam Cozzette wrote:
>>
>> There is no way to remove the underscores without changing protoc. But why
>> do you want to get rid of the underscores anyway? Those variables are just a
>> private implementation detail and make no difference to the public API.
>>
>> On Wed, Sep 5, 2018 at 1:07 PM Chris Zhang  wrote:
>>>
>>> Hi,
>>>
>>> I am new to Protobuf, and recently I found out that the java generated
>>> code by protobuf has underscore by the end of each field names.
>>>
>>> For example,
>>>
>>> protobuf message file look like this:
>>>
>>> message DummyMessage [
>>>
>>> string some_id = 1;
>>> bool is_active = 2;
>>> }
>>>
>>> The generated java code is like this:
>>>
>>> Class DummyMessage {
>>>
>>> String someId_;
>>> boolean isActive_;
>>>
>>> }
>>>
>>> Is there any way to get rid of the underscore of each field?
>>>
>>> Thanks,
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: framing header message -- how to know length?

[Ilia Mirkin]

So like I said... you have to write it as



To decode it, you can stick the header length as a varint, all via the
coded input/output stream. Sample code for encoding:

makePacket(Message message) {
  String name = ... (there has gotta be a way of getting the typename
from the message)
  length = message.getSerializedSize()
  Header h = Header.newBuilder().setName(name).setLength(length);
  CodedOutputStream os = newInstance();
  os.writeInt32(h.getSerializedSize());
  os.writeMessageNoTag(h);
  os.writeMessageNoTag(message);
}

And to decode:

Message readPacket() {
  CodedInputStream is;
  hlen = is.readInt32();
  limit = is.pushLimit(hlen);
  Header h = Header.parseFrom(is);
  is.popLimit(limit);
  is.pushLimit(h.getLength());
  messageFactory.getBuilderForType(h.getName()).mergeFrom(is);
}

And you're done. Protobuf is meant for serializing data, not framing
it. Your complaint about lack of framing is well taken, but ... it's
just not part of what protobuf does.

You could just as well have a

message TheOneTrueMessage {
  string name;
  bytes data;
}

And then always decode that -- this gives you your framing. But this
ends up causing lots of copies of the data.

  -ilia

On Tue, Jul 10, 2018 at 11:20 AM, John Lilley  wrote:
> Just to wrap this up, here's what I found:
> You really have to know how big the header message is, because if you simply
> wrap an input stream around the entire concatenated buffer, the
> deserialization doesn't know where the end of the header is.  This strikes
> me as a shortcoming of protobuf, but it is what it is.  I used code
> something like(in java):
>
> protected static byte[] makePacket(Message header, Message message) throws
> IOException {
>// We cannot simply concatenate the two messages; protobuf will fail to
> deserialize them.
>// So we add the header length at the front.  But we don't know it yet...
>ByteArrayOutputStream os = new ByteArrayOutputStream();
>os.write(new byte[4]);   // we will update this later
>CodedOutputStream cos = CodedOutputStream.newInstance(os);
>header.writeTo(cos);
>cos.flush();
>// NOW we know the header length
>int headerLength = os.size() - 4;
>message.writeTo(cos);
>cos.flush();
>byte[] result = os.toByteArray();
>// Set header size
>ByteBuffer bb = ByteBuffer.wrap(result);
>bb.order(ByteOrder.LITTLE_ENDIAN);
>bb.putInt(0, headerLength);
>return result;
> }
>
> Then for deserialization:
>
> protected static Message getResponseMessage(MessageFactory messageFactory,
> byte[] packet) throws Exception {
>ByteBuffer bb = ByteBuffer.wrap(packet);
>bb.order(ByteOrder.LITTLE_ENDIAN);
>int headerLength = bb.getInt(0);
>CommonWrapper.ResponseHeader header;
>{
>   InputStream is = new ByteArrayInputStream(packet, 4, headerLength);
>   CodedInputStream cis = CodedInputStream.newInstance(is);
>   header = CommonWrapper.ResponseHeader.parseFrom(cis);
>}
>InputStream is = new ByteArrayInputStream(packet, 4 + headerLength,
> packet.length - (4 + headerLength));
>return messageFactory.createMessage(header.getResponseMessageType(), is);
> }
>
> The MessageFactory is a hand-rolled class for creating messages from full
> name; but that's for a different thread.
>
> john
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Java API: creating message by name

[Ilia Mirkin]

Right ... I'm lazy too ...

add("FooType", FooType.getDefaultInstance())

...

map.get("FooType").newBuilderForType().mergeFrom(bla).build()

Right?

On Mon, Jul 9, 2018 at 8:45 PM, John Lilley  wrote:
> Because I am lazy!  I only want to add the wrapper once.
> john
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Java API: creating message by name

[Ilia Mirkin]

Why not just call add() with a Message instance instead?

On Mon, Jul 9, 2018 at 7:39 PM, John Lilley  wrote:
> Darn, there's only one snag.  I need the protobuf package name, not the java
> package classname, in order to construct the message name that corresponds
> to the result of full_name() in the C++ generated code.  Of course, I can
> make them the same thing, but I'm not finding any way to interrogate the
> class to find out the package to which it belongs.
> john
>
> On Mon, Jul 9, 2018 at 5:17 PM John Lilley  wrote:
>>
>> Success!  This works nicely (although, lacking polish)
>>
>> public class MessageFactory {
>>
>>private final Map defaultMessages = new HashMap<>();
>>
>>public void add(Class wrapperClass) {
>>   for (Class nestedClass : wrapperClass.getDeclaredClasses()) {
>>  if (!Message.class.isAssignableFrom(nestedClass)) {
>> continue;
>>  }
>>  try {
>> String fullName = wrapperClass.getSimpleName() + "." +
>> nestedClass.getSimpleName();
>> Message message =
>> (Message)nestedClass.getMethod("getDefaultInstance").invoke(null);
>> defaultMessages.put(fullName, message);
>>  } catch (Exception ex) {
>> ex.printStackTrace();
>>  }
>>   }
>>}
>>
>>public Message.Builder getBuilder(String fullName) {
>>   Message message = defaultMessages.get(fullName);
>>   if (message == null) {
>>  throw new IllegalArgumentException("Unknown message name '" +
>> fullName + "'");
>>   }
>>   return message.newBuilderForType();
>>}
>>
>>public Message createMessage(String fullName, InputStream is) throws
>> Exception {
>>   Message.Builder builder = getBuilder(fullName);
>>   CodedInputStream cis = CodedInputStream.newInstance(is);
>>   return builder.mergeFrom(cis).build();
>>}
>>
>>
>>public Message createMessage(String fullName, byte[] buffer) throws
>> Exception {
>>   return createMessage(fullName, new ByteArrayInputStream(buffer));
>>}
>> }
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Java API: creating message by name

[Ilia Mirkin]

newBuilderForType: "Constructs a new builder for a message of the same
type as this message."

That's if you have a message, and you want to create a builder for
that type of message. Not what you want.

The actual types are outside of the library, and need to be looked up
via a type registry. I could have sworn there was such a thing, but
can't find it now. May be good to have a look through the generated
code, and see where it registers itself -- should give you a pointer.

Or you can maintain the registry yourself - a map of string -> default
instance for the type, from which you can call newBuilderForType().

  -ilia

On Mon, Jul 9, 2018 at 5:45 PM, John Lilley  wrote:
> Well, apparently I am really off base.  Given a Descriptor I cannot figure
> out how to create the right message. I *thought* this was the right
> approach:
>  Descriptors.Descriptor desc = // look up the descriptor
>  Builder builder = desc.toProto().newBuilderForType();
>  Message message = builder.mergeFrom(requestBytes).build();
>
>
> But no.  Can anyone help me with this?  I need to go from the full name of a
> message to its builder and I'm not finding anything like MessageFactory() or
> DescriptorPool in Java.
> Thanks
> john
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] framing header message -- how to know length?

[Ilia Mirkin]

It's been nearly a decade since I've looked at those APIs closely,
hopefully someone else can elaborate. If what you want is a data
buffer you can then copy somewhere else, you either need something
dynamically sizeable, or just decree what the max size is, and
allocate that. Even better would be to avoid the additional copy, but
perhaps your message broker won't allow that.

On Sat, Jul 7, 2018 at 4:34 PM, John Lilley  wrote:
> If I want to write header+body to an array of bytes (in C++), is the easiest
> thing to use StringOutputStream, then copy its buffer when finished?
> I also looked at ArrayOutputStream, but at first read it appears to require
> knowledge of the output size before constructing the stream.  True?
> Thanks
> john
>
>
> On Sat, Jul 7, 2018 at 2:18 PM John Lilley  wrote:
>>
>> Got it, thanks!
>> john
>>
>> On Sat, Jul 7, 2018 at 2:13 PM Ilia Mirkin  wrote:
>>>
>>> CodingInputStream/OutputStream have all that. readInt32/etc.
>>>
>>> There's no strict advantage... but presumably you're using protobuf to
>>> make your life easier, and this will make your life easier. (With a
>>> string, you have to include the length, etc. And if the header ever
>>> changes and you want to have back/forward compat, it can be
>>> convenient.)
>>>
>>> On Sat, Jul 7, 2018 at 4:03 PM, John Lilley  wrote:
>>> > Does protobuf include utility methods for direct ser/deser on varint,
>>> > string, etc?
>>> > Thanks
>>> > john
>>> >
>>> > On Sat, Jul 7, 2018 at 2:02 PM John Lilley  wrote:
>>> >>
>>> >> Thanks!
>>> >> Given that, is there any advantage to a "header message" as opposed to
>>> >> just hand-serializing everything in the header?
>>> >>
>>> >>
>>> >> On Sat, Jul 7, 2018 at 12:45 PM Ilia Mirkin 
>>> >> wrote:
>>> >>>
>>> >>> You need explicit lengths. Usually this is done as >> >>> varint>. And the header contains the body length in it.
>>> >>> In Java, there's a CodedInputStream/OutputStream which makes it easy
>>> >>> to consume fixed lengths (push/popLimit) as well as raw varints (as
>>> >>> for the initial header length). Other languages have similar
>>> >>> abstractions.
>>> >>>
>>> >>> On Sat, Jul 7, 2018 at 2:26 PM, John Lilley 
>>> >>> wrote:
>>> >>> > I am posting protobuf messages to a message broker, and in order to
>>> >>> > identify
>>> >>> > them, I prefix the message bytes with the serialized result of a
>>> >>> > "header"
>>> >>> > message:
>>> >>> >
>>> >>> > message Header {
>>> >>> >int version = 1;
>>> >>> >string message_type = 2;
>>> >>> > }
>>> >>> >
>>> >>> > It is easy, to concatenate the header+actual message bytes and post
>>> >>> > the
>>> >>> > resulting block to a queue. But how do I take these apart on the
>>> >>> > receiving
>>> >>> > end? Suppose I get a byte-buffer consisting of:
>>> >>> >
>>> >>> > ---
>>> >>> > | header  |
>>> >>> > ---
>>> >>> > | body|
>>> >>> > ---
>>> >>> >
>>> >>> > Is it OK to throw this oversized buffer at the Header
>>> >>> > deserialization?
>>> >>> > Will
>>> >>> > the extra bytes hurt anything?
>>> >>> >
>>> >>> > Then, once I extract the Header message, how do I know where the
>>> >>> > body
>>> >>> > begins? I could turn around and ask the Header object "how big
>>> >>> > would
>>> >>> > you be
>>> >>> > if serialized?".  Is that reliable?  Is there a better way?
>>> >>> >
>>> >>> > Thanks
>>> >>> > john
>>> >>> >
>>> >>> > --
>>> >>> > You received this message because you are subscribed to the Google
>>> >>> > Groups
>>> >>> > "Protocol Buff

Re: [protobuf] framing header message -- how to know length?

[Ilia Mirkin]

https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.coded_stream

On Sat, Jul 7, 2018 at 4:16 PM, John Lilley  wrote:
> OK in Java I've found the classes UInt32Value, StringValue, etc.
> C++ isn't quite so obvious. Where should I look for those classes?
> Thanks
> john
>
> On Sat, Jul 7, 2018 at 2:03 PM John Lilley  wrote:
>>
>> Does protobuf include utility methods for direct ser/deser on varint,
>> string, etc?
>> Thanks
>> john
>>
>> On Sat, Jul 7, 2018 at 2:02 PM John Lilley  wrote:
>>>
>>> Thanks!
>>> Given that, is there any advantage to a "header message" as opposed to
>>> just hand-serializing everything in the header?
>>>
>>>
>>> On Sat, Jul 7, 2018 at 12:45 PM Ilia Mirkin  wrote:
>>>>
>>>> You need explicit lengths. Usually this is done as >>> varint>. And the header contains the body length in it.
>>>> In Java, there's a CodedInputStream/OutputStream which makes it easy
>>>> to consume fixed lengths (push/popLimit) as well as raw varints (as
>>>> for the initial header length). Other languages have similar
>>>> abstractions.
>>>>
>>>> On Sat, Jul 7, 2018 at 2:26 PM, John Lilley  wrote:
>>>> > I am posting protobuf messages to a message broker, and in order to
>>>> > identify
>>>> > them, I prefix the message bytes with the serialized result of a
>>>> > "header"
>>>> > message:
>>>> >
>>>> > message Header {
>>>> >int version = 1;
>>>> >string message_type = 2;
>>>> > }
>>>> >
>>>> > It is easy, to concatenate the header+actual message bytes and post
>>>> > the
>>>> > resulting block to a queue. But how do I take these apart on the
>>>> > receiving
>>>> > end? Suppose I get a byte-buffer consisting of:
>>>> >
>>>> > ---
>>>> > | header  |
>>>> > ---
>>>> > | body|
>>>> > ---
>>>> >
>>>> > Is it OK to throw this oversized buffer at the Header deserialization?
>>>> > Will
>>>> > the extra bytes hurt anything?
>>>> >
>>>> > Then, once I extract the Header message, how do I know where the body
>>>> > begins? I could turn around and ask the Header object "how big would
>>>> > you be
>>>> > if serialized?".  Is that reliable?  Is there a better way?
>>>> >
>>>> > Thanks
>>>> > john
>>>> >
>>>> > --
>>>> > You received this message because you are subscribed to the Google
>>>> > Groups
>>>> > "Protocol Buffers" group.
>>>> > To unsubscribe from this group and stop receiving emails from it, send
>>>> > an
>>>> > email to protobuf+unsubscr...@googlegroups.com.
>>>> > To post to this group, send email to protobuf@googlegroups.com.
>>>> > Visit this group at https://groups.google.com/group/protobuf.
>>>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] framing header message -- how to know length?

[Ilia Mirkin]

CodingInputStream/OutputStream have all that. readInt32/etc.

There's no strict advantage... but presumably you're using protobuf to
make your life easier, and this will make your life easier. (With a
string, you have to include the length, etc. And if the header ever
changes and you want to have back/forward compat, it can be
convenient.)

On Sat, Jul 7, 2018 at 4:03 PM, John Lilley  wrote:
> Does protobuf include utility methods for direct ser/deser on varint,
> string, etc?
> Thanks
> john
>
> On Sat, Jul 7, 2018 at 2:02 PM John Lilley  wrote:
>>
>> Thanks!
>> Given that, is there any advantage to a "header message" as opposed to
>> just hand-serializing everything in the header?
>>
>>
>> On Sat, Jul 7, 2018 at 12:45 PM Ilia Mirkin  wrote:
>>>
>>> You need explicit lengths. Usually this is done as >> varint>. And the header contains the body length in it.
>>> In Java, there's a CodedInputStream/OutputStream which makes it easy
>>> to consume fixed lengths (push/popLimit) as well as raw varints (as
>>> for the initial header length). Other languages have similar
>>> abstractions.
>>>
>>> On Sat, Jul 7, 2018 at 2:26 PM, John Lilley  wrote:
>>> > I am posting protobuf messages to a message broker, and in order to
>>> > identify
>>> > them, I prefix the message bytes with the serialized result of a
>>> > "header"
>>> > message:
>>> >
>>> > message Header {
>>> >int version = 1;
>>> >string message_type = 2;
>>> > }
>>> >
>>> > It is easy, to concatenate the header+actual message bytes and post the
>>> > resulting block to a queue. But how do I take these apart on the
>>> > receiving
>>> > end? Suppose I get a byte-buffer consisting of:
>>> >
>>> > ---
>>> > | header  |
>>> > ---
>>> > | body|
>>> > ---
>>> >
>>> > Is it OK to throw this oversized buffer at the Header deserialization?
>>> > Will
>>> > the extra bytes hurt anything?
>>> >
>>> > Then, once I extract the Header message, how do I know where the body
>>> > begins? I could turn around and ask the Header object "how big would
>>> > you be
>>> > if serialized?".  Is that reliable?  Is there a better way?
>>> >
>>> > Thanks
>>> > john
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "Protocol Buffers" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to protobuf+unsubscr...@googlegroups.com.
>>> > To post to this group, send email to protobuf@googlegroups.com.
>>> > Visit this group at https://groups.google.com/group/protobuf.
>>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] framing header message -- how to know length?

[Ilia Mirkin]

You need explicit lengths. Usually this is done as . And the header contains the body length in it.
In Java, there's a CodedInputStream/OutputStream which makes it easy
to consume fixed lengths (push/popLimit) as well as raw varints (as
for the initial header length). Other languages have similar
abstractions.

On Sat, Jul 7, 2018 at 2:26 PM, John Lilley  wrote:
> I am posting protobuf messages to a message broker, and in order to identify
> them, I prefix the message bytes with the serialized result of a "header"
> message:
>
> message Header {
>int version = 1;
>string message_type = 2;
> }
>
> It is easy, to concatenate the header+actual message bytes and post the
> resulting block to a queue. But how do I take these apart on the receiving
> end? Suppose I get a byte-buffer consisting of:
>
> ---
> | header  |
> ---
> | body|
> ---
>
> Is it OK to throw this oversized buffer at the Header deserialization?  Will
> the extra bytes hurt anything?
>
> Then, once I extract the Header message, how do I know where the body
> begins? I could turn around and ask the Header object "how big would you be
> if serialized?".  Is that reliable?  Is there a better way?
>
> Thanks
> john
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Using ParseFromString to Differentiate Message Types

[Ilia Mirkin]

There's no distinction between

message Foo {
  int32 foo = 1;
}

and

message Bar {
  int32 bar = 1;
}

It's all encoded as tags and values, the type and field names aren't
on the wire. Unknown tags are ignored (or put into a "unknown tag"
list). The framing is entirely up to you. One non-horrible thing to do
is to create e.g. a

message Header {
  string type = 1;
  int32 length = 2;
}

Then write a len(header) varint, header, payload. Then you can decode
them sequentially and get the right data on the other end. (The
horrible thing to do is to create a "container" message which has the
the payload data in a "bytes" field, since you end up
double-encoding/decoding the data.)

  -ilia

On Mon, Apr 30, 2018 at 3:49 PM, Andrew Bouchard  wrote:
> I'm using the MOOS communications framework, which allows me to send
> messages between modules in a binary, string, or numerical format using a
> centralized database. In order to minimize the number of new interfaces that
> I need to create, I'd like to send multiple messages over the same
> interface, then differentiate between them on the other end. The method I
> struck on to do this was to use ParseFromString to see whether the received
> message parsed successfully as a given type. Here's a simple example:
>
> void CModule::ProcessMessages( CMOOSMsg &msg )
> {
> // Declare message types
> MyMessages::TypeA type_a;
> MyMessages::TypeB type_b;
>
> // Parse a message of type A
> if( type_a.ParseFromString( msg.GetString() ) )
> {
> printf( "Received a message of type A\n" );
> }
> // Parse a message of type B
> else if( type_b.ParseFromString( msg.GetString() ) )
> {
> printf( "Received a message of type B\n" );
> }
> else
> {
> printf( "ERROR: Unrecognized message type" );
> }
> }
>
> I think that I have misunderstood how ParseFromString works, because instead
> of the first call returning FALSE when msg is a serialized message of type
> B, it attempts to shoehorn the fields from type B into type A; as you might
> guess, this is causing all sorts of problems.
>
> I've been looking into the MessageLite class and its ParseFromCodedStream
> method. I think that it may do what I want, but I'm hesitant to go down the
> rabbit hole of learning a new tool if I can talk to someone else and learn
> if this is really what I want first. Does anyone here have any advice?
> Thanks very much!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Variable-Width Integer Encoding

[Ilia Mirkin]

I doubt you're going to get a nice clean answer. Chances are it's
"whatever Sanjay was thinking at the time" which led to the current
encoding, maintained throughout the proto versions for backwards
compatibility with existing data. While APIs have changed over time,
the wire encoding has remained extremely stable.

While we now live in the future, and storage/memory/bandwidth are free
and infinite, and all CPUs are 64-bit, that was not always the case.
Your encoding would not be a clear win for smaller values, and would
be an obvious waste of 4 bits when encoding int32 values.
Additionally, your encoding would not be cleanly extendable to 128-bit
integers in a backwards-compatible way.

The neat thing about the current encoding is that the width of the
integer doesn't really matter -- it could be a 4096-bit integer for
all you know, you just keep reading bytes until you hit one without a
high bit set. Which means it's easy to adjust protos as values change
in allowed ranges, esp between 32 and 64 bits.

But that's just a post-facto justification. It's unlikely that the
original rationale from 15+ years ago exists anywhere.

  -ilia


On Tue, Jan 2, 2018 at 3:54 PM,   wrote:
> What is the rationale behind the current variable-width integer encoding?
>
> As I understand it, an integer is terminated by a byte that's 
> most-significant bit is equal to zero.  Thus, bytes must be read one at a 
> time, and this condition must be checked after reading each one to determine 
> whether to read another.  Why was this encoding chosen over a variable-width 
> encoding that would require at most two reads -- that is, an encoding that 
> specifies the number of subsequent bytes to read in the first byte?
>
> No, I don't mean for the first byte's value to be the length of the rest of 
> the integer.  Rather, the number of leading ones in the first byte could be 
> the number of following bytes.  This would still allow 7 bits of a value to 
> be stored per byte, with the added bonus of a full 64-bit value being encoded 
> in 9 bytes instead of 10.
>
> Examples:
>
> 0 leading ones followed by a terminating zero and then 7 bits:
>
> 0b0...
>
> 1 leading one followed by a terminating zero, then 6 bits, and then 1 byte:
>
> 0b10.. 
>
> 7 leading ones followed by a terminating zero and then 7 bytes:
>
> 0b1110       
>
> 8 leading ones followed by 8 bytes:
>
> 0b        
> 
>
> So, such an encoding is clearly possible.  Why does Protocol Buffers use 
> something different?  Is this to provide some level of protection against 
> dropped bytes?  Has all of the data already been read into a buffer by the 
> time that it is to be decoded, and so reducing the number of reads does not 
> provide much of a speed boost?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is protobuf3 a superset of protobuf2?

[Ilia Mirkin]

The main thing that proto3 lacks and keeps coming up every time I use
it is field availability information. All the has_* API is gone for
basic types, so you have to supply that information separately, or
restructure your logic to not rely on it.

  -ilia

On Fri, Dec 15, 2017 at 6:40 PM, ajcurtis84  wrote:
> Hello,
>
> I have not found anything in the documentation that explicitly says this.
> The only indication is that the proto3 documentation refers to proto2.
> Things like extensions are available in the proto3 syntax?
>
> thanks
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Any way to "extract all" using protoc or any other tool?

[Ilia Mirkin]

On Sun, Dec 10, 2017 at 11:23 AM, Jim Baldwin  wrote:
> Perhaps it might help if I understood the output of protoc --decode_raw.
>
> Here's an example of a .caffemodel file I'm trying to inspect.  Is there a
> description of what the numbers mean in this file?

As I said... it's a sequence of (tag, value) pairs, so...

>
> 1: "VGG_ILSVRC_16_layers"

Tag id 1: string value "VGG_bla"

> 100 {

Tag id 100: sub-message of some length (not printed here, but it's in
the encoded proto)

>   1: "input-data"

Tag id 1 of sub-message: "input-data"

(I won't annotate the rest, hopefully you get the gist of it.)

So for example, you might have a proto which is

message foo {
  string a = 1;
  repeated message bar = 100;
};

that would lead to that sort of encoding. Nowhere is type information
encoded on the wire unless you explciitly put it there yourself.
There's no way to know which proto it is, and there can be any number
of protos that would yield a valid decoding. So you have to give it
the name of the proto it is with --decode.

Protobuf is a structure encoding/decoding mechanism. Anything extra,
like type names or framing have to come from surrounding logic.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Ilia Mirkin]

On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin  wrote:
> I have a protobuf file, and a .proto file that describes the schema.
>
> The .proto describes dozens of different messages that may be in the
> protobuf file.
>
> I would like to know what messages can be found in the file.  I do a protoc
> --decode_raw and get something out, but I don't see how to use that to
> figure out how to extract messages from the file.
>
> I assume there's something I don't get about protobufs, but it seems to me I
> should be able to take a protobuf data file and corresponding .proto and
> turn it into a file that lets me see what the message hierarchy is in the
> file.  JSON would be a great way to do that.
>
> What am I missing?

An encoded protobuf is just a sequence of (tag, value) pairs. If you
don't know which proto it is, decode_raw is the best you can do. If
you do know which proto it is, you can use --decode instead and pass
it a proto name to use for the decoding.

Cheers,

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] could anyone give me the source code of protobuffer 2.0.3

[Ilia Mirkin]

Not easy to find anymore it seems... looks like all the old releases
have been lost in the move to github. That's unfortunate.

http://ftp.slackware.com/pub/gsb/gsb64-2.28_slackware64-13.1/source/c/protobuf/protobuf-2.0.3.tar.bz2
http://www.java2s.com/Code/Jar/p/Downloadprotobufjava203jar.htm

No clue how authoritative these are. Use with care.


On Wed, Aug 9, 2017 at 11:54 PM,   wrote:
> My project is using this old version, but the source code of protobuffer
> 2.0.3 is missing. Anyone can help me ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Has Anyone Used protobuf on ucLinux?

[Ilia Mirkin]

By the way, you may be interested in the nanopb and protobuf-c
implementations, which likely have fewer dependencies.

On Tue, Mar 14, 2017 at 4:50 PM, Doug Lewis  wrote:
> Ah I see that in the full output of the make process now.
>
> Yes all I really need is to get the libraries built with the cross compiler.
>
> Thank you!
>
> Doug
>
> On Tuesday, March 14, 2017 at 2:56:04 PM UTC-5, Doug Lewis wrote:
>>
>> I've been tasked with seeing if we can use protobuf on ucLinux (micro c
>> Linux).  We had previously been using an embedded version of Debian Linux
>> and we had protobuf working very well on it.
>>
>> The specific issue I am having is as I'm building the protobuf code I get
>> this compile error:
>>
>> google/protobuf/compiler/subprocess.cc:304: error: 'fork' was not declared
>> in this scope
>>
>> ucLinux does not support 'fork' but it does have 'vfork'.  I do not relish
>> the idea of modifying the protobuf source code to change any 'fork' calls to
>> 'vfork' and was wondering if anyone else has "ported" protobuf to ucLinux.
>>
>> Thanks,
>> Doug
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Has Anyone Used protobuf on ucLinux?

[Ilia Mirkin]

So then you don't need the code that's giving you errors - it's part
of protoc. It's been way too many moons since I've looked at the
details for how to only build the lib and not protoc, or even if it's
still possible, but you might find that that's the path of least
resistance.

On Tue, Mar 14, 2017 at 4:35 PM, Doug Lewis  wrote:
> My host platform is an Ubuntu Linux box where the protoc compiler will run
> to generate the C++ code that will be cross compiled to run on the uClinux
> target.
>
> Doug
>
>
> On Tuesday, March 14, 2017 at 2:56:04 PM UTC-5, Doug Lewis wrote:
>>
>> I've been tasked with seeing if we can use protobuf on ucLinux (micro c
>> Linux).  We had previously been using an embedded version of Debian Linux
>> and we had protobuf working very well on it.
>>
>> The specific issue I am having is as I'm building the protobuf code I get
>> this compile error:
>>
>> google/protobuf/compiler/subprocess.cc:304: error: 'fork' was not declared
>> in this scope
>>
>> ucLinux does not support 'fork' but it does have 'vfork'.  I do not relish
>> the idea of modifying the protobuf source code to change any 'fork' calls to
>> 'vfork' and was wondering if anyone else has "ported" protobuf to ucLinux.
>>
>> Thanks,
>> Doug
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Has Anyone Used protobuf on ucLinux?

[Ilia Mirkin]

This is in the protobuf compiler. Do you plan on compiling protobuf
proto definitions into source code on your ucLinux platform itself?

On Tue, Mar 14, 2017 at 3:56 PM, Doug Lewis  wrote:
> I've been tasked with seeing if we can use protobuf on ucLinux (micro c
> Linux).  We had previously been using an embedded version of Debian Linux
> and we had protobuf working very well on it.
>
> The specific issue I am having is as I'm building the protobuf code I get
> this compile error:
>
> google/protobuf/compiler/subprocess.cc:304: error: 'fork' was not declared
> in this scope
>
> ucLinux does not support 'fork' but it does have 'vfork'.  I do not relish
> the idea of modifying the protobuf source code to change any 'fork' calls to
> 'vfork' and was wondering if anyone else has "ported" protobuf to ucLinux.
>
> Thanks,
> Doug
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] String max length is 255. Correct?

[Ilia Mirkin]

The length is stored as a varint; you should be able to store strings
of any length, up to 64-bit.

On Wed, Dec 14, 2016 at 8:43 AM, David Mullineux  wrote:
> The encoding suggests that strings are encoded with a 1 byte length prefix. 
> This means my strings cannot be larger than 255 in size. How can I store 
> longer strings please?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Reserved keyword usage in protobuf 2.5.0

[Ilia Mirkin]

On Wed, May 25, 2016 at 5:44 PM, Marcos Juarez  wrote:

> I ran into a situation today in which I needed to use the "reserved"
> keyword in a protobuf definition, so that in the future, nobody would use
> those fields.  I've attached a screenshot of what the docs say, from
> https://developers.google.com/protocol-buffers/docs/proto#reserved
>
> [image: Inline image 1]
>
> However, when I try to use that keyword in an actual message with field
> definitions, protoc always complains with the following error:
>
> *Expected "required", "optional", or "repeated".*
>
> My protobuf definition looks like this:
>
> message MyMessage {
>   // Some comments
>   reserved, 4, 108, 109;
>

Did you mean "reserved 4, 108, 109;"?

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Has Anyone Successfully Defined "smaller" integers for protocol buffers (C++ or Java)?

[Ilia Mirkin]

If your varint is < 128, it will only take a single byte to get
encoded. Even if it's declared as an int64. (Also, consider that what
matters isn't data size but number of frames sent.)

Anyways, if you wanted to created fixed 8- and 16-bit types, you'd
have to add new field types for those, and pipe through the new types
everywhere. Just grep for FIXED32 and Uint32. However you really
should consider just using one of the varint-encoded types -- they
will likely serve you better.

  -ilia

On Tue, May 24, 2016 at 12:24 PM, Douglas Lewis  wrote:
> Well the solution we want to implement is with 16 bit and 8 bit elements
> when that is all the resolution of the integer variable we need.  This is an
> embedded application and we are sending the messages over a wireless
> connection so data packet size is critical to achieving our desired
> throughput.
>
> - Doug
>
> On Tuesday, May 24, 2016 at 10:06:50 AM UTC-5, Douglas Lewis wrote:
>>
>> I'm working on a project and we are hoping to use protocol buffers for
>> messages between embedded and PC based applications.
>>
>> One issue I've immediately run into is protocol buffers only has 32 and 64
>> bit integers, I'm needing 16 and 8 bit integers for fields in the messages.
>> Has anyone been able to successfully add integer types to protocol buffers?
>> We are using C++ on the embedded side and Java on the PC side.
>>
>> Thanks,
>> Doug
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Has Anyone Successfully Defined "smaller" integers for protocol buffers (C++ or Java)?

[Ilia Mirkin]

Why not just use a varint? You can use the zigzag variants if you want
to store things like -1 efficiently. The only time you'd benefit from
having a fixed8/fixed16 field is if the high bits are set, and it'd
only be by 1 byte.

  -ilia

On Tue, May 24, 2016 at 11:32 AM, Douglas Lewis  wrote:
> I'm talking about fixed integers.  We have certain elements of the messages
> that we only need/want resolution to 16 bits or 8 bits.
>
> - Doug
>
>
> On Tuesday, May 24, 2016 at 10:06:50 AM UTC-5, Douglas Lewis wrote:
>>
>> I'm working on a project and we are hoping to use protocol buffers for
>> messages between embedded and PC based applications.
>>
>> One issue I've immediately run into is protocol buffers only has 32 and 64
>> bit integers, I'm needing 16 and 8 bit integers for fields in the messages.
>> Has anyone been able to successfully add integer types to protocol buffers?
>> We are using C++ on the embedded side and Java on the PC side.
>>
>> Thanks,
>> Doug
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Has Anyone Successfully Defined "smaller" integers for protocol buffers (C++ or Java)?

[Ilia Mirkin]

On Mon, May 23, 2016 at 9:55 AM, Douglas Lewis  wrote:
> I'm working on a project and we are hoping to use protocol buffers for
> messages between embedded and PC based applications.
>
> One issue I've immediately run into is protocol buffers only has 32 and 64
> bit integers, I'm needing 16 and 8 bit integers for fields in the messages.
> Has anyone been able to successfully add integer types to protocol buffers?
> We are using C++ on the embedded side and Java on the PC side.

Non-fixed integers have varint encoding, so it doesn't matter how many
bits there are... or are you talking about fixed integers?

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Ilia Mirkin]

You can't distinguish an empty repeated from one that's not there at
all. If you need that, you'll need a manual presence field.

On Tue, Mar 29, 2016 at 2:02 AM, Yoav H  wrote:
> How do they handle collections (repeated, non packed) in this case?
> The absence of the tag is not conclusive.
> Actually, even packed collection (and strings, and binary data) suffer from
> that, as you are "expected" to not include a packed collection with zero
> bytes.
>
>
> On Saturday, March 26, 2016 at 1:08:23 PM UTC-7, Ilia Mirkin wrote:
>>
>> Encoding is identical... just the API is different. In proto2, you
>> have (in C++) FooMessage->has_field() which will tell you whether a
>> field was present in the encoded version (or has been set prior if
>> you're building a new message). The Java API has something rather
>> similar... hasField() I think?
>>
>> On Sat, Mar 26, 2016 at 4:00 PM, Yoav H  wrote:
>> > Thanks all,
>> >
>> > Do you know where I can find the proto2 encoding guide?
>> > The proto site has only the proto3 encoding described.
>> >
>> > On Saturday, March 26, 2016 at 12:21:39 PM UTC-7, Tim Kientzle wrote:
>> >>
>> >>
>> >> > On Mar 26, 2016, at 11:43 AM, Yoav H  wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > I wanted ask regarding the decision to populate fields with default
>> >> > values, even if they do not appear in the encoded message.
>> >> > If I want to send a "patch" message, where I want to update just the
>> >> > provided fields, how can I do that with protobuf (without adding
>> >> > IsXXXSet
>> >> > for every field)?
>> >> >
>> >> > Why not add another type, representing a default value?
>> >> > So the schematics would be, if the field is missing, it is null, and
>> >> > if
>> >> > the field exists, but with this "missing value" type, it will get the
>> >> > default value?
>> >>
>> >> As Ilia pointed out, proto2 still exists, is still supported, and can
>> >> be
>> >> used for
>> >> cases where you require these particular semantics.
>> >>
>> >> For proto3, you might look at google.protobuf.FieldMask, which is a new
>> >> standard message (one of the "well-known types") specifically designed
>> >> to store a set of field names.  You might be able to achieve what you
>> >> want
>> >> by
>> >> providing a FieldMask with your data listing the specific fields to
>> >> be updated.
>> >>
>> >> Tim
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Ilia Mirkin]

Encoding is identical... just the API is different. In proto2, you
have (in C++) FooMessage->has_field() which will tell you whether a
field was present in the encoded version (or has been set prior if
you're building a new message). The Java API has something rather
similar... hasField() I think?

On Sat, Mar 26, 2016 at 4:00 PM, Yoav H  wrote:
> Thanks all,
>
> Do you know where I can find the proto2 encoding guide?
> The proto site has only the proto3 encoding described.
>
> On Saturday, March 26, 2016 at 12:21:39 PM UTC-7, Tim Kientzle wrote:
>>
>>
>> > On Mar 26, 2016, at 11:43 AM, Yoav H  wrote:
>> >
>> > Hi,
>> >
>> > I wanted ask regarding the decision to populate fields with default
>> > values, even if they do not appear in the encoded message.
>> > If I want to send a "patch" message, where I want to update just the
>> > provided fields, how can I do that with protobuf (without adding IsXXXSet
>> > for every field)?
>> >
>> > Why not add another type, representing a default value?
>> > So the schematics would be, if the field is missing, it is null, and if
>> > the field exists, but with this "missing value" type, it will get the
>> > default value?
>>
>> As Ilia pointed out, proto2 still exists, is still supported, and can be
>> used for
>> cases where you require these particular semantics.
>>
>> For proto3, you might look at google.protobuf.FieldMask, which is a new
>> standard message (one of the "well-known types") specifically designed
>> to store a set of field names.  You might be able to achieve what you want
>> by
>> providing a FieldMask with your data listing the specific fields to
>> be updated.
>>
>> Tim
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Default Values vs Missing Values

[Ilia Mirkin]

Use proto2, which has the has_* checks per field. (Using get_* you
still get the default value, of course.) It's extremely unfortunate
that this functionality was removed in proto3, I see that making
proto3 unattractive for all but the simplest uses of protos. I know in
almost every protobuf use-case I've had, the presence accessors were
imperative to proper operation.

On Sat, Mar 26, 2016 at 2:43 PM, Yoav H  wrote:
> Hi,
>
> I wanted ask regarding the decision to populate fields with default values,
> even if they do not appear in the encoded message.
> If I want to send a "patch" message, where I want to update just the
> provided fields, how can I do that with protobuf (without adding IsXXXSet
> for every field)?
>
> Why not add another type, representing a default value?
> So the schematics would be, if the field is missing, it is null, and if the
> field exists, but with this "missing value" type, it will get the default
> value?
>
> Thanks,
> Yoav.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] [Java][Python][TCP] Reading messages that where written with writeDelimited and viceversa

[Ilia Mirkin]

On Tue, Sep 29, 2015 at 9:31 AM, Mike White  wrote:
> aloha everyone,
>
> i stumbled across this post as i'm hitting this error as well. since i'm a
> newbie to python i had a question on how to implement this. when doing these
> steps,
>
> # Read a message from Java's writeDelimitedTo:
> import google.protobuf.internal.decoder as decoder

buffer = open(sys.argv[1], "rb").read()

>
> # Read length
> (size, position) = decoder._DecodeVarint(buffer, 0)
> # Read the message
> message_object.ParseFromString(buffer[position:position+size])

And then this should work, I think.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: How to check empty message object

[Ilia Mirkin]

On Thu, Sep 17, 2015 at 11:18 AM, Simon Brandhof
 wrote:
> What are the side-effects to define wrapper messages as a workaround ?
>
> message Foo {
>   Int32 bar = 1;
> }
>
> message Int32 {
>   int32 value = 1;
> }
>
> In Java it allows to have Foo.hasBar() and Foo.getBar().getValue().

Sub-message involves a lot more encoding overhead, both cpu time
(creating objects) as well as larger data encoding (extra tag + length
get encoded -- an extra 2 bytes).

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Question about data type from binary stream

[Ilia Mirkin]

https://developers.google.com/protocol-buffers/docs/encoding#signed-integers

This is all documented. Please read the docs.

On Sat, Sep 5, 2015 at 11:08 PM, TheAceInfinity  wrote:
> I wrote these helper macros for s types:
> #define encode_sint32(n) (((n) << 1) ^ ((n) >> 31))
> #define encode_sint64(n) (((n) << 1) ^ ((n) >> 63))
>
> Do you have any information on how to go about decoding the bytes if my
> original type was sint32 or sint64?
>
> On Saturday, September 5, 2015 at 7:41:05 PM UTC-6, Ilia Mirkin wrote:
>>
>> On Sat, Sep 5, 2015 at 9:36 PM, TheAceInfinity  wrote:
>> > If I know that the bytes are a type of varint from a byte stream, is
>> > there
>> > then no way to determine whether the 150 should be calculated and
>> > interpreted as an int32 or an sint32?
>>
>> Nope, no information beyond the wire type. Also no way to tell the
>> difference between, say, a string or a submessage, since those are
>> both just length-delimited fields.
>>
>> > That's my question. What exactly is the TAG ID for?
>>
>> To tell which field is which.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Question about data type from binary stream

[Ilia Mirkin]

On Sat, Sep 5, 2015 at 9:36 PM, TheAceInfinity  wrote:
> If I know that the bytes are a type of varint from a byte stream, is there
> then no way to determine whether the 150 should be calculated and
> interpreted as an int32 or an sint32?

Nope, no information beyond the wire type. Also no way to tell the
difference between, say, a string or a submessage, since those are
both just length-delimited fields.

> That's my question. What exactly is the TAG ID for?

To tell which field is which.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Question about data type from binary stream

[Ilia Mirkin]

$ perl -ane 'foreach (@F) { print pack "C", hex($_) }' | protoc --decode_raw
08 96 01
1: 150

Or did you want to see how one might do this by hand? The 96 is
misleading since one might be tempted to think that 0x96 == 150 (which
it is), but really it's 0x16 + high bit set, which indicates that the
next bit should be taken from the next byte, which happens to just be
01, i.e. only the low bit is set, aka 0x96 in the end. If the stream
were 08 96 02, then you should end up with a value of 0x116, i.e. 278:

$ perl -ane 'foreach (@F) { print pack "C", hex($_) }' | protoc --decode_raw
08 96 02
1: 278

Everything in a protobuf stream is pairs of (tagid, value). The first
byte has the tagid (if it's low enough, it's varint encoded starting
with bit 3, so if it's > 4 bits it spills into the next byte) as well
as the wiretype. The wiretype determines how to parse the value. Of
course there's no way to tell the difference between, say, an int32 or
sint32 (which uses zigzag encoding), at least not from the information
on the wire.

All this and more explained at
https://developers.google.com/protocol-buffers/docs/encoding?hl=en#structure

Hope this helps.

  -ilia

On Sun, Aug 30, 2015 at 9:49 PM, TheAceInfinity  wrote:
> Say we take the bytes from the example in the docs.
>
> 0x08, 0x96, 0x01
>
> For 0x08, the wire_type is 0, which indicates VarInt, the message before
> encoding the data shows that the VarInt is a type int32. The field_number
> from 0x08 is 1 though. Where does this 1 come into play when it comes to
> decoding the data from the binary stream?
>
> I read this: "As you know, a protocol buffer message is a series of
> key-value pairs. The binary version of a message just uses the field's
> number as the key"
>
> What is the definition of *key* in this context? Does it mean that I can use
> this to determine the type from the specific wire_type as a key to be used
> for a 1-based associative array? (i.e. 1 = int32, 2 = int64, etc...)
>
> If someone could just clarify how I would determine the specific VarInt out
> of these 3 bytes, that would be much appreciated!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protocol Buffer for parsing binary message format

[Ilia Mirkin]

No, protobuf has a fixed wire format. You can't use it to describe
arbitrary binary formats, only arbitrary data to be passed around.

On Wed, Aug 26, 2015 at 3:26 AM, tomcat  wrote:
> In my application, I need to parse a relatively complex binary message
> exchange format, i.e. I receive messages in a binary format and need a
> parser to parse the messages to bring it into a Java object representation.
> Is protobuf suitable for this kind of task?
>
> kind regards
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protov3 empty fields

[Ilia Mirkin]

Just to echo this, I think that having empty fields be
distinguishables from ones set to the default value is an immensely
useful feature, irrespective of whether the field is a primitive type
or not. I know I've used it lots of times.

On Wed, Jul 1, 2015 at 4:28 PM, John C.  wrote:
> My understanding is that if a primitive field is empty in a v3 message then
> it will give 0 which is indistinguishable from a field that is actually 0.
> From what I have read is this is done to enable support of a wide range of
> languages, but how is this a well reasoned solution to this problem?  The
> response to me might be "well just explicitly include a 'has_field' boolean
> to indicate the presence of a field", but I have messages with many optional
> fields and adding a "has_xxx" field for each makes the message definition so
> ugly that I feel overcome by waves of nausea.  What initially attracted me
> to protocol buffers is the ability to define messages in a single place
> rather than using say JSON where the protocol can easily end up being
> distributed over multiple source files.
>
> My solution for this big problem is an ugly hack but better than any other
> ugly hack:  for primitive fields code the value as (val*2) | 1.  This
> guarantees all present fields are non-zero so empty fields can be detected.
> I think a better solution would be to, at least, have an API that I can use
> to determine if a field is present or not.  Since the v3 work is still in
> alpha, my hope is that someone comes to their senses.
>
> If not for the much better Ruby performance with v3 I would stick with v2
> but the improvement is undeniable so v3 with ugly hack it is for now.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to get ByteArray from Protobuf in Java

[Ilia Mirkin]

These are the right steps. Make sure that the sender and receiver see the
same data. Should be obvious, but protobuf has no framing, so if you want
to send a bunch in one stream, you need to handle that.
On Jun 5, 2015 2:34 AM,  wrote:

> I am new to Protobuf.
>
> Here is my problem
>
>
> I am using toByteArray() method on a protobuf object in java to get the
> binary representation of teh object
> Then i send the object via web socket to the client
> When the client receives the Message, i convert from binary to Protobuf
> object using parseFrom.
>
> *Parsing and Serialization*
> Finally, each protocol buffer class has methods for writing and reading
> messages of your chosen type using the protocol buffer *binary format*
> . These
> include:
>
>- byte[] toByteArray();: serializes the message and returns a byte
>array containing its raw bytes.
>- static Person parseFrom(byte[] data);: parses a message from the
>given byte array.
>- void writeTo(OutputStream output);: serializes the message and
>writes it to an OutputStream.
>- static Person parseFrom(InputStream input);: reads and parses a
>message from an InputStream.
>
> The sender only send 9 bytes and receives gets a huge message. Basically
> the sender and receivers are receiving different size message for the
> Protobuf object
>
> Is toByteArray() the proper way to generate Byte ARRAY from the Protobuf
> object? If so, how to parse it on the client side?
>
> Thank you
>
> Mark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf encoded message

[Ilia Mirkin]

BTW, you might be interested in reading through
https://developers.google.com/protocol-buffers/docs/encoding. This is
how all the encoding/decoding works.

12 = wire type 2 (0x12 & 0x7) (length-delimited), field number 2 (0x12 >> 3)
40 = the length (i.e. 64)
the next 64 bytes make up the value for field 2
 = would be decoded for wire type + field number again.

No idea where that 01 byte is coming from, but it probably shouldn't be there.

An alternate interpretation, with the 01 byte:

12 = wire type 2, field number 2
01 = length (i.e. 1 byte)
40 = the data for field 2
30 = wire type 0, field number 6
37 = value of field 6
62 = wire type 2, field number 12
65 = length
... not enough bytes to decode.

Hence the failure by protoc --decode_raw.

On Thu, May 21, 2015 at 2:09 PM, Ilia Mirkin  wrote:
> Hm, that 01 byte seems to be extra somehow. Or there are bytes after
> this which are missing.
>
> $ perl -ane 'foreach (@F) { print pack "C", hex($_) }' | protoc --decode_raw
> 12 01 40 30 37 62 65 37 36 30 34 32 33 35 32 37
> 33 30 64 64 63 37 38 35 39 39 38 39 34 66 31 31
> 37 65 30 37 34 35 36 61 37 64 30 37 66 62 36 31
> 64 39 38 32 31 62 32 36 61 38 33 34 61 34 30 66
> 64 62 38
> Failed to parse input.
> $ perl -ane 'foreach (@F) { print pack "C", hex($_) }' | protoc --decode_raw
> 12 40 30 37 62 65 37 36 30 34 32 33 35 32 37
> 33 30 64 64 63 37 38 35 39 39 38 39 34 66 31 31
> 37 65 30 37 34 35 36 61 37 64 30 37 66 62 36 31
> 64 39 38 32 31 62 32 36 61 38 33 34 61 34 30 66
> 64 62 38
> 2: "07be76042352730ddc78599894f117e07456a7d07fb61d9821b26a834a40fdb8"
>
> On Thu, May 21, 2015 at 6:59 AM, Mason Cubes  wrote:
>> I am trying to understand a protobuf encoded message. the message is given
>> below, it is apparently a 64 byte long hash value, but I cannot understand
>> the encoding process
>>
>> 12 01 40 30 37 62 65 37 36 30 34 32 33 35 32 37
>> 33 30 64 64 63 37 38 35 39 39 38 39 34 66 31 31
>> 37 65 30 37 34 35 36 61 37 64 30 37 66 62 36 31
>> 64 39 38 32 31 62 32 36 61 38 33 34 61 34 30 66
>> 64 62 38
>>
>>
>> The first byte says the following is a byte or string wire type and the
>> second byte says only one byte in the byte or string array. Then the third
>> byte 0x40 it must be the length of the field i.e 64. If I consider this byte
>> as a varint tag content then the rest of the contents does not make sense,
>> unless it is a repeated field with integers only 1 byte values all the way
>> to the end.
>>
>> I cannot understand how it was decoded. I don't have the proto file used to
>> encode the message.
>>
>> Can someone give me a hint?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at http://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf encoded message

[Ilia Mirkin]

Hm, that 01 byte seems to be extra somehow. Or there are bytes after
this which are missing.

$ perl -ane 'foreach (@F) { print pack "C", hex($_) }' | protoc --decode_raw
12 01 40 30 37 62 65 37 36 30 34 32 33 35 32 37
33 30 64 64 63 37 38 35 39 39 38 39 34 66 31 31
37 65 30 37 34 35 36 61 37 64 30 37 66 62 36 31
64 39 38 32 31 62 32 36 61 38 33 34 61 34 30 66
64 62 38
Failed to parse input.
$ perl -ane 'foreach (@F) { print pack "C", hex($_) }' | protoc --decode_raw
12 40 30 37 62 65 37 36 30 34 32 33 35 32 37
33 30 64 64 63 37 38 35 39 39 38 39 34 66 31 31
37 65 30 37 34 35 36 61 37 64 30 37 66 62 36 31
64 39 38 32 31 62 32 36 61 38 33 34 61 34 30 66
64 62 38
2: "07be76042352730ddc78599894f117e07456a7d07fb61d9821b26a834a40fdb8"

On Thu, May 21, 2015 at 6:59 AM, Mason Cubes  wrote:
> I am trying to understand a protobuf encoded message. the message is given
> below, it is apparently a 64 byte long hash value, but I cannot understand
> the encoding process
>
> 12 01 40 30 37 62 65 37 36 30 34 32 33 35 32 37
> 33 30 64 64 63 37 38 35 39 39 38 39 34 66 31 31
> 37 65 30 37 34 35 36 61 37 64 30 37 66 62 36 31
> 64 39 38 32 31 62 32 36 61 38 33 34 61 34 30 66
> 64 62 38
>
>
> The first byte says the following is a byte or string wire type and the
> second byte says only one byte in the byte or string array. Then the third
> byte 0x40 it must be the length of the field i.e 64. If I consider this byte
> as a varint tag content then the rest of the contents does not make sense,
> unless it is a repeated field with integers only 1 byte values all the way
> to the end.
>
> I cannot understand how it was decoded. I don't have the proto file used to
> encode the message.
>
> Can someone give me a hint?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Using CodedOutputStream to send many messages across network

[Ilia Mirkin]

On Tue, Mar 17, 2015 at 10:15 PM, Sayan Goswami  wrote:
> Hi All,
>
> I am using CodedOutputStream and CodedInputStream to transfer messages
> across a network like so:
>
> Client requests:
>
> Socket serverSocket = new Socket(hosts, ports);
>
> CodedOutputStream outputStream =
> CodedOutputStream.newInstance(serverSocket.getOutputStream());
>
> outputStream.flush();
>
> CodedInputStream inputStream =
> CodedInputStream.newInstance(serverSocket.getInputStream());
>
>
> // the following happens inside a loop
>
>  Request request = Request.newBuilder()
>
> .setType(Request.RequestType.GET)
>
> .setKey(key)
>
> .build();
>
>
>  outputStream.writeRawVarint32(request.getSerializedSize());
>
>  request.writeTo(outputStream);
>
>  outputStream.flush();
>
>
> Server Processes Request:
>
>
> CodedOutputStream output =
> CodedOutputStream.newInstance(clientSocket.getOutputStream());
>
> output.flush();
>
> CodedInputStream input =
> CodedInputStream.newInstance(clientSocket.getInputStream());
>
>
> int length = input.readRawVarint32();
>
> byte[] message = input.readRawBytes(length);

Ew. You can just do a

limit = input.pushLimit(length)
request = Request.Builder.mergeFrom(input).build();
input.popLimit(limit);

>
> Request request = Request.parseFrom(message);
>
> // do things with request
>
>
> Now, each request of mine, has a serialized size of 56.. so I can send many
> messages up the stream, but what can I do when the size limit of 64MB is
> exceeded? Is there any way to reset the buffer? Following is the error
> message I am getting:
>
>
> Protocol message was too large.  May be malicious.  Use
> CodedInputStream.setSizeLimit() to increase the size limit.

>From the docs:

If you want to read several messages from a single CodedInputStream,
you could call resetSizeCounter() after each one to avoid hitting the
size limit.

Cheers,

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Multibye character in protobuf getting corrupted while serializing and serializing.

[Ilia Mirkin]

Perhaps you misunderstood... the two are actually identical. They're
just printed differently. I merely provided the steps to demonstrate
the fact that they were, in fact, identical.

There might be a way to force the text formatter to not escape
non-ascii characters (although I don't see that off-hand anywhere),
but what's printed by the text formatter shouldn't matter for
communications -- you should be using the binary format. The text
format is just there for debug purposes.

  -ilia

On Mon, Mar 9, 2015 at 5:33 PM, Nishant Verma  wrote:
> Thanks Ilia Mirkin,
> What I gave is just a part of message, the files generated are sometimes
> more than 4 MB in size. And this issue is spread across the fields. How to
> identify when and where apply the Unicode conversion will become issue.
>
> Regards,
> Nishant
>
> On Monday, March 9, 2015 at 4:36:53 PM UTC-4, Ilia Mirkin wrote:
>>
>> On Mon, Mar 9, 2015 at 4:19 PM, Nishant Verma  wrote:
>> > There is a multibyte character in one of the string field of the
>> > Protobuf
>> > messages. When I am serializing or de-serilizig characters are getting
>> > corrupted. Please advise solution.
>> >
>> >
>> > Corrupted message:
>> >  analyst {
>> >   analystNumber: 29003798
>> >   analystLastName: "Azconegui"
>> >   analystFirstName: "Mar\303\255a Valeri"
>>
>> Seems fine...
>>
>> >>> unicode('\303\255', "utf8")
>> u'\xed'
>>
>> i.e. U+00ED, which is
>>
>> http://www.fileformat.info/info/unicode/char/00ed/index.htm
>>
>> i.e. that í character.
>>
>> >
>> > Expected message:
>> >  analyst {
>> >   analystNumber: 29003798
>> >   analystLastName: "Azconegui"
>> >   analystFirstName: "María Valeri"
>> >
>> >
>> > Serializing:
>> > String issuerDataURL ="c:/fileSerilized.ser";
>> > Issuer.IssuerData dataToBeWritenToSer = issuerDataBuilder.build();
>> > FileOutputStream fileOut = new FileOutputStream(issuerDataURL);
>> >
>> > dataToBeWritenToSer.writeTo(fileOut);
>> >
>> >
>> > De-Serializing:
>> > String writeTo =  "C:\\DeSer\\20150308193244042.ser.txt";
>> > FileOutputStream fos = new FileOutputStream(writeTo);
>> > ObjectOutputStream oos = new ObjectOutputStream(fos);
>> > oos.writeObject(isd.toString());
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Newbie Question regarding multiple messages

[Ilia Mirkin]

Protobuf is a way to encode a message (data) to a byte stream. It does
not provide any sort of framing/type/etc information. You have to
supply that separately. For example a given endpoint may accept only
one type of message. Or you have another "standard" header type
message that describes what data comes next.

On Sat, Feb 28, 2015 at 4:30 PM,   wrote:
> Hi,
>
> I'm new to protocol buffers, and I've searched the docs without a clear
> answer to my question.
>
> I've defined several messages in a .proto file. I would like to use protocol
> buffers to serialize/deserialize these messages as commands between two
> processes. The .proto file was compiled with protoc, etc.
>
> The problem is I don't see how in the code the destination can figure out
> which message it is receiving.  All examples assume the destination knows
> which message it is receiving.
>
> I see the discussion of streaming multiple messages and self-describing
> messages, but neither seems to be what I need to do.
>
> This seems like a typical use case to me.  Can anyone point me to an example
> that shows how to determine the type of a received message or a link to an
> explanation?
>
> I've been using 2.6.1, not the recent 3 release.
>
> Thank you very much for any help!
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Multibye character in protobuf getting corrupted while serializing and serializing.

[Ilia Mirkin]

On Mon, Mar 9, 2015 at 4:19 PM, Nishant Verma  wrote:
> There is a multibyte character in one of the string field of the Protobuf
> messages. When I am serializing or de-serilizig characters are getting
> corrupted. Please advise solution.
>
>
> Corrupted message:
>  analyst {
>   analystNumber: 29003798
>   analystLastName: "Azconegui"
>   analystFirstName: "Mar\303\255a Valeri"

Seems fine...

>>> unicode('\303\255', "utf8")
u'\xed'

i.e. U+00ED, which is

http://www.fileformat.info/info/unicode/char/00ed/index.htm

i.e. that í character.

>
> Expected message:
>  analyst {
>   analystNumber: 29003798
>   analystLastName: "Azconegui"
>   analystFirstName: "María Valeri"
>
>
> Serializing:
> String issuerDataURL ="c:/fileSerilized.ser";
> Issuer.IssuerData dataToBeWritenToSer = issuerDataBuilder.build();
> FileOutputStream fileOut = new FileOutputStream(issuerDataURL);
>
> dataToBeWritenToSer.writeTo(fileOut);
>
>
> De-Serializing:
> String writeTo =  "C:\\DeSer\\20150308193244042.ser.txt";
> FileOutputStream fos = new FileOutputStream(writeTo);
> ObjectOutputStream oos = new ObjectOutputStream(fos);
> oos.writeObject(isd.toString());
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] same message name in diffent .proto file, how can i fix it?

[Ilia Mirkin]

Use "package". 
https://developers.google.com/protocol-buffers/docs/proto#packages

On Wed, Jan 14, 2015 at 2:27 AM, you zhou  wrote:
> in my project, file name as module name, like that
> file(module):battle.proto
> message SyncMapReq{
>
> }
> message SyncMapResp{
> message Map map = 1;
> repeated Player player = 2;
> }
> message Player {
> required int32 x = 1;
> required int32 y = 2;
> required int32 speed = 3;
> required int32 hp = 4;
> }
> file(module): base.proto
> message LoadPlayerReq{
> }
> message LoadPlayerResp{
> required Player player = 1;
> }
> message Player {
> required string name = 1;
> required int32 level = 2;
> required int32 gold = 3;
> required int32 gem = 4;
> }
> how can i make it work?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How To Distinguish signed interger and unsigned interger?

[Ilia Mirkin]

You can't tell from the binary stream. You have to know what data
you're decoding to decide exactly how to interpret the decoded varint
value.

On Tue, Dec 30, 2014 at 3:47 AM, Lai Champion  wrote:
>
>I read the protocol buffer documents,I know compiler use ZigZag to decode
> signed interger. I am confused that if  i receive a binary stream,I could
> extract the data type ,for example,it's a Varint Wire-Type,followed by the
> data number for example 96 01, so how to decide the number is a signed or an
> unsigned?
>
>can anyone help me clarify the problem?
>
>Best Regards!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Determining If a Message is of the Expected Message Format

[Ilia Mirkin]

protobuf messages are typeless. There is no difference between

message Foo {
 int field = 1;
}

and

message Bar {
  int otherfield = 1;
}

On Mon, Dec 22, 2014 at 1:32 PM, Ryan O'Meara
 wrote:
> Hi,
>
> I am using protobuf for a project, and I've run into an interesting case,
> which I've had no luck finding a standard solution to. For the record, I'm
> using the Java APIs to interact with my messages
>
> My project has several messages defined, a few of which have similar fields
> (for example, two of them have a required text field with index 1). The
> problem I'm running into is that if I erroneously pass the byte array form
> of one message to a builder of the other message, it will happily parse it
> without any exceptions or other issues. (Apparently
> InvalidProtoException is thrown only is the array is completely invalid
> as a protobuf message of any type, and does not indicate anything else)
>
> I have tried checking "isInitialized", but because they both have the same
> required fields at the same indexes, it doesn't have an error.
>
> I considered changing the indexes or something like that, but that depends
> on every person who works on this project knowing about and respecting that
> convention - which doesn't seem like a sustainable solution. I also thought
> of checking the unknown fields, but one of the reasons I chose protobuf was
> for its forward and backwards compatibility - and I believe that the unknown
> filed list will be (validly) non-empty if I add a field down the road and an
> old version of the project receives a new version of the message.
>
> The only other thing I can think to do is to add a field in my messages
> which indicates the message - is this my best option, or am I missing
> something in the standard libraries which handles this?
>
> Thanks!
>
> Example:
>
> message Message1 {
>
>   required string field1 = 1;
>
>   optional string field2 = 2;
> }
>
> message Message2 {
>
>   required string field1 = 1;
>
>   optional int32 field2 = 2;
>
> }
>
>
> public void parseMessage1(byte[] protobuf){
> try {
> Message1 message = Messag1e.parseFrom(protobuf);
>
> if(message.isInitialized()){
> //Do some stuff
> }else{
> //Never get here
> }
> } catch (InvalidProtocolBufferException e) {
> //Not thrown
> }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] packages and import

[Ilia Mirkin]

I think you want foo_primitives.bar ... going from memory though.

On Fri, Nov 7, 2014 at 7:19 PM, Michael Wagner  wrote:
> I would like to do the follwing:
>
> foo_primtives.proto:
>
> package foo_primitives;
>
> messaage bar {
> // some fields;
> };
>
> message baz {
>// some fields;
> }
>
> foo_composed_types.proto:
>
> import "foo_primtives.proto";
> package foo_composed;
>
> message foo_complicated {
>optional foo_primitves::bar bar = 1
>optional foo_primitives::baz baz = 2;
> };
>
> If I do thus protoc gives me an "Expected field name" error on the first
> colon in the package specifer.
>
> If I remove the "package foo_primitves;" from the "foo_primtives.proto" file
> and corresponding "foo_primitves::" from the fields in foo_composed.proto,
> then protoc is happy.
>
> Can I import packages? Am I fat-fingerig something?
>
> Thanks,
>
> Mike
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Parse a .proto file

[Ilia Mirkin]

On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota  wrote:
> Boolean extension =
> fieldDescriptor.getOptions().getExtension(Messages.isPii);

Shouldn't this use some sort of API that doesn't use the Messages class at all?

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Parse a .proto file

[Ilia Mirkin]

At no point are you specifying that you want to use the
"MessagePublish" descriptor, so you must still be using the API
incorrectly...

On Fri, Oct 31, 2014 at 5:10 PM, Pradeep Gollakota  wrote:
> Ok… awesome… I do have the .proto’s ahead of time, so I can have them
> compiled to the .desc files and store those.
>
> Here’s my .proto file:
>
> package com.lithum.pbnj;
>
> import "google/protobuf/descriptor.proto";
>
> option java_package = "com.lithium.pbnj";
>
> extend google.protobuf.FieldOptions {
> optional bool isPii = 50101;
> }
>
> message MessagePublish {
> required string uuid = 1;
> required int64 timestamp = 2;
> required int64 message_uid = 3;
> required string message_content = 4;
> required int64 message_author_uid = 5;
> optional string email = 6 [(isPii) = true];
> }
>
> I compiled this .proto file into a .desc file using the command you gave me.
> I’m now trying to parse a DynamicMessage from the .desc file. Here’s the
> code I have so far.
>
> DescriptorProtos.FileDescriptorSet descriptorSet =
> DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream("/messages.desc"));
> Descriptors.Descriptor desc =
> descriptorSet.getFile(0).getDescriptorForType();
>
> Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
> .setUuid(UUID.randomUUID().toString())
> .setTimestamp(System.currentTimeMillis())
> .setEmail("he...@example.com")
> .setMessageAuthorUid(1)
> .setMessageContent("hello world!")
> .setMessageUid(1)
> .build();
>
> DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
> event.toByteArray());
>
> The final line in the above code is throwing the following exception:
>
> Exception in thread "main"
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag.
> at
> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
> at
> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
> at
> com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
> at
> com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:180)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:419)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:171)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:412)
> at com.google.protobuf.DynamicMessage.parseFrom(DynamicMessage.java:119)
> at com.lithium.pbnj.PBnJ.main(PBnJ.java:36)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>
> On Friday, October 31, 2014 11:39:27 AM UTC-7, Oliver wrote:
>>
>> Basically, you can't do that in pure Java - the compiler is a C++
>> binary, there is no Java version.
>>
>> Still, working with the output of --descriptor_set_out is probably the
>> way to go here. If you have the .proto file ahead of time, you can
>> pregenerate the descriptor output at build time and store it instead
>> of the .proto file. If you don't have the .proto file ahead of time
>> (and you can't redesign - this is not a good design) then you could
>> run the compiler at runtime and read the output. Either way, now you
>> have a parsed version of the message format as a protobuf-encoded
>> message that you can read into your Java program and extract the
>> Descriptors you need.
>>
>> If you're looking at a selfdescribing message format, then I'd go with
>> using the parsed descriptors as your format description, not the text
>> .proto file.
>>
>> Oliver
>>
>>
>> On 31 October 2014 17:56, Pradeep Gollakota  wrote:
>> > Hi Oliver,
>> >
>> > Thanks for the response! I guess my question wasn't quite clear. In my
>> > java
>> > code I have a string which contains the content of a .proto file. Given
>> > this
>> > string, how can I create an instance of a Descriptor class s

Re: [protobuf] Move nested message.

[Ilia Mirkin]

On Mon, Oct 27, 2014 at 6:33 PM, Shuo Zhou  wrote:
> Hi,
>
> Suppose I got a message as follows:
>
> message Foo {
>   message Bar {
> optional string name = 1;
>   }
>   repeated Bar bar = 1;
> 
> }
>
> Then I find Bar is needed widely as a generic message, and want to move it
> outside of Foo as follows:
> message Bar {
>   optional string name = 1;
> }
> message Foo {
>   repeated Bar bar = 1;
> ...
> }
>
> Would this change cause serializing problem?

Nope. As long as the field numbers don't change, there's no difference.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] can protobuf cost less bytes?

[Ilia Mirkin]

I believe you're looking for zigzag encoding wrt negatives. If instead
of int32 you call it sint32, that should get a much better encoding
for small negative values.

On Sun, Oct 19, 2014 at 9:58 AM, Aijing Sun  wrote:
> 1.if a int value is -1,then through varint 128 serialze is 10 bytes.
> 2. a bool true is none 0,and false is 0,but it cost 1 byte .
>
> I hope -1 just cost 1 byte,and bool cost no byte.
> and it can cost less byte when serialize
> and I suggest change wire type:
> 0(000) --- positive varint
> 1(001) --- negative  varint
> 2(010)  true
> 3(011)  false
> 4(100)  bit32
> 5(101)  bit64
> 6(110)  Length-delimited
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Message encoding question

[Ilia Mirkin]

On Wed, Oct 15, 2014 at 6:05 AM, Tamás Somhegyi
 wrote:
> Hello,
>
> I am quite new to Protocol Buffers but I have read the tutorials. There is
> one part of the encoding which I don't understand.
> How the message types are encoded? As I see the tutorial mentions the field
> encoding but the field id is unique only in the message.

They're not. It's up to you to know what message you're decoding.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to define Structured Binary file

[Ilia Mirkin]

Not sure what is being asked, but simply having a

bytes foo = 1;

will cause the presence of the field, the length of the field, and the
contents of the field to be encoded into the resulting binary
representation (as   ).

However enforcing maximum lengths/other data validation is up to the
application layer, not the serialization layer. At least with the way
protobufs are designed.

On Mon, Oct 6, 2014 at 8:09 PM, Marc Gravell  wrote:

> You cannot. What you describe is not what protobuf offers.
> On 7 Oct 2014 00:55, "RPMASA"  wrote:
>
>> Hi,
>>
>> I have structured binary file specifying size of each element in bytes.
>> How can I mentioned them in ptoto file?
>>
>>
>>
>>No. Description Data Value Variable Type  Number of bytes  1 A1 0=unuse,
>> 1=use byte 1  2 A2 0=unuse, 1=use byte 1  3 A3 false=unuse, true=use byte
>> 1 2-2. Object[200] - this object can contains up to 200 times
>>No. Description Data Value Variable Type  Number of bytes  1 B1 1
>> to 200 int 4  2 B2   string 40  3 B3   string 52  4 B4 0=unuse, 1=use
>> byte 1  5 B5 MMDD string 16  6 B6   string 30  7 B7 0=unuse, 1=use
>> byte 1
>>
>>
>> In above example how can I define that fields B2 is string and can have
>> up to 40 bytes of data?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at http://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What format is DebugString using for bytes?

[Ilia Mirkin]

Octal. It has the same meaning as if you had put that string into a C file.

On Wed, Sep 17, 2014 at 3:07 PM, Nick Retallack  wrote:
> When I put some arbitrary bytes in a bytes field in a protocol buffer and
> then print it out with DebugString, it formats the bytes in a format I'm
> unfamiliar with.  It's full of three-digit escape sequences like "\364\007".
> What format is this?  364 is too big to be a byte, but nothing seems to
> exceed 3 digits.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: enquiry about bit format of data type bool

[Ilia Mirkin]

On Thu, Jul 24, 2014 at 7:38 PM, ching lu  wrote:
> Previous example is not good
>
> mssage {
>
> required bool x1 = 1;
> required bool x2 = 2;
> required bool x3 = 3;
> required bool x4 = 4;
> required bool x5 = 5;
> required bool x6 = 6;
> required bool x7 = 7;
> required bool x8 = 8;
> required bool x9 = 9;
> required bool x10 = 10;
> required bool x11 = 11;
> required bool x12 = 12;
> required bool x13 = 13;
> required bool x14 = 14;
> required bool x15 = 15;
> optional bool y = 16;
>
> }
>
> will it take 16 byte or 16 bit for the message? There is no documentation
> for the behaviour of bit field

AFAIK there's no boolean wire type, so they're just varints that
happen to only take on 0/1 values (and provide the appropriate
language bindings). So 16 bytes.

And that's because your field id's are < 16. Once you go over 16,
it'll be 2 bytes per field (i.e. the tag will take up a full byte, and
then the value will be another byte).

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Find the message type during processing of serialized message at run time!

[Ilia Mirkin]

The serialized message does not contain any type information. You
would either have to create a super-message for that, or have some
other sort of knowledge of what type to expect. (A problem with the
super-message approach is all the copying involved, so if that's a
concern, you can just manually encode stuff into a CodedOutputStream.)

On Fri, Jul 4, 2014 at 11:33 AM, Andrew Klitovchenko
 wrote:
> Hi,
>
> I am writing an app that uses google protobuf to decode the serialized
> messages which were coded using .proto definitions.
> If I have the .proto file and during run time I have different types of
> messages coming to my app. I wan to be able to identify which message it is
> according to .proto definitions.
> How can I do that? GetTypeName()? Or I cannot identify which message it is I
> am reading during run time?
>
> Thanks,
> Andrew
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Possible C++ serialization issue with repeated fields

[Ilia Mirkin]

On Wed, Jun 18, 2014 at 12:26 PM, Justin  wrote:
> My team and I noticed a potential bug in the serialization process, that
> seems unintended.
>
> If I defined a message structure as follows:
>
> message X {
>required Y y = 1;
> }
>
> message Y {
>repeated Things things = 1;
>repeated Stuff stuff = 2;
> }
>
> message Things {
>required string name = 1;
> }
>
> message Stuff {
>required string name = 1;
> }
>
>
> When I create an object of type X, serialize and deserialize it, using the
> following code (for example):
>
> X x_original;
> std::string x_content;
> x_original.SerializeToString(&x_content);

I believe this returns false, which indicates an error, since
x_original.y isn't set, and it's required.

>
> X x_new;
> x_new.ParseFromString(x_content);
>
>
> The ParseFromString call will throw an error :
>
>[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse
> message of type "testDomain.X" because it is missing required fields: y
>
>
> Our intended use case for this is, in our example, letting another
> application know that there are no Things and no Stuff available, so I
> believe this should work as written.

Perhaps y should be optional then? As the proto is written, you must supply a y.

>
> if I, however, change the code to :
>
> X x_original;
> Y y;
> x_original.mutable_y()->CopyFrom(y);
> std::string x_content;
> x_original.SerializeToString(&x_content);
>
> X x_new;
> x_new.ParseFromString(x_content);
>
>
> It works fine, as I believe would set_allocated_y() if I used a *y.
>
> Is this an intended 'feature' and should I be doing something different?

Yep, it's a feature -- if a field is required, it must be present in
order to serialize and deserialize without error.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protoc not generated add_...() method for repeated nested messages

[Ilia Mirkin]

On Wed, Jun 11, 2014 at 7:04 PM, CB  wrote:
>
>
> On Wednesday, June 11, 2014 7:01:42 PM UTC-4, Ilia Mirkin wrote:
>>
>> On Wed, Jun 11, 2014 at 6:54 PM, CB  wrote:
>> >
>> > I've got a proto file that looks like this:
>> >
>> > option optimize_for = SPEED;
>> >
>> > package test;
>> >
>> > message KeyedStrings
>> > {
>> > optional string key = 1;
>> > repeated string value = 2;
>> > }
>> >
>> > message Map
>> > {
>> > repeated KeyedStrings map = 1;
>> > repeated string test = 2;
>> > }
>> >
>> > My problem is that the generated C++ code does not include any methods
>> > that
>> > allow me to add or set the map values. The test field is fine as you can
>> > see
>> > the required methods in the attached files. What am I doing wrong?
>>
>> What method are you expecting to see but isn't there? add_map() seems
>> to be there...
>>
>>   -ilia
>
>
> Yes, but it takes no arguments. There are add_test() methods that take the
> string to add, for example. I expected an add_map() method that took the
> KeyedStrings object to add.

That's not how it works. It returns a KeyedStrings object for you to
modify. Take a look at

https://developers.google.com/protocol-buffers/docs/reference/cpp-generated#fields

Specifically the "Repeated Embedded Message Fields" section.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protoc not generated add_...() method for repeated nested messages

[Ilia Mirkin]

On Wed, Jun 11, 2014 at 6:54 PM, CB  wrote:
>
> I've got a proto file that looks like this:
>
> option optimize_for = SPEED;
>
> package test;
>
> message KeyedStrings
> {
> optional string key = 1;
> repeated string value = 2;
> }
>
> message Map
> {
> repeated KeyedStrings map = 1;
> repeated string test = 2;
> }
>
> My problem is that the generated C++ code does not include any methods that
> allow me to add or set the map values. The test field is fine as you can see
> the required methods in the attached files. What am I doing wrong?

What method are you expecting to see but isn't there? add_map() seems
to be there...

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Size of string Message not reducing even after serializing with Protocolbuffer

[Ilia Mirkin]

Can you elaborate a bit on the "before" scenario? (i.e. exactly what
are you comparing protobuf to?)

Protobuf adds overhead to data (in order to tell which field is which,
etc), and does nothing for reducing the sizes of strings. (Integers
are usually varint-encoded, so if they're small, they'll save space,
if they're large, they'll take up even more space than before. But you
can use fixed32/fixed64 to mitigate that if you expect large integers
to be sent.)

In your case, I would expect  
. Both tag 1 and the string length should be 1 byte each,
unless the string is longer than 127 bytes (in which case the string
length would take up 2 bytes). And the string data will be however
long the passed in string is... let's say it was 50 bytes, I would
expect the protobuf-serialized message to be 52 bytes.

Does that not line up with what you're seeing?

  -ilia


On Fri, May 16, 2014 at 7:21 AM, Ratheesh R  wrote:
>  Hello Experts,
>
>  I am new to this Protocol buffer techinque.I am trying to use PB in my
> project because my project is critical to the data transfer rate(Battery
> power).
> So I am trying to use PB to serialize data (in binary mode) to reduce the
> data size before transferring this data to the other application.
> But I was not successful in seeing the reduction in size even after
> serializing with API's SerializeToString() OR SerializeToArray().
>
> let's  say ,my string is like the below which I am sending on evey 100
> millisecond to other application,
>
>  datatype="8" rawdata="FFF1" default="65297" />
>
> Previously it was sending in text mode and now i am trying with ProtoBuff to
> serailize this to send,but I could not see any diffrence in size reduction
> of this string
> after serilaizng with PB.please see my code and let me know where i am doing
> the mistake,
>
> ---
> proto
> 
> message AMessage
>{
>   optional string str1=1;
>}
> -
> Code i have done to serailize the above string to reduce the size (binary
> format)
> -
>  AMessage A_msg;
>  A_msg.set_str1(m_txBuff); // m_txBuff -contains the above mentioned
> string
>  std::string encode_string;
>
> // tried with SerializeToString,but was not successful //
> A_msg.SerializeToString(&encode_string);
>
>// --tried  with SerializeToArray ,it was also not
> successful-//
> int size_bytes = A_msg.ByteSize();
> void *buffer_bytes = malloc(size_bytes);
> A_msg.SerializeToArray(buffer_bytes, size_bytes);
>
>//sending the data send
>  infomq_cmd_publish((unsigned
> char*)encode_string.c_str(),encode_string.size(),0);
>
> Anything i need to do extra for making this serailization in binary format
> for reducing the size on serialization ?
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Use different encoding while serializing strings

[Ilia Mirkin]

protobuf does not "use" \r or \n for any specific purposes. It does,
however, assume that it can use any byte it chooses. There is no way
to restrict that. You would have to create a protocol to encode
arbitrary bytes over your not-binary-safe-channel, which would have to
create a way to escape \r and \n (what about \0? can't imagine it'd be
too happy about that either).

  -ilia

On Thu, May 15, 2014 at 1:33 PM, Ganesh Sangle  wrote:
> Let me clarify the question.
> 1. I did not say that protobuf is affected by \r\n.
> 2. The problem is this:
> Take any message with repeated fields, and serialize it to text.
> The serialized message contains \r\n which are inserted by protobuf, which
> it is using as some kind of delimiter.
> That is throwing off my code which is itself using \r\n as a delimiter.
>
> So the question is : is there a way to configure protobuf to use something
> else as a delimiter and not \r\n ?
>
> Thanks,
> Ganesh
>
> On Wednesday, May 14, 2014 11:48:37 PM UTC-7, Marc Gravell wrote:
>>
>> protobuf is a binary-safe protocol, and is not impacted by contents such
>> as \r, \n or \t. In particular, text content is utf-8 encoded and
>> length-prefixed - it simply *does not care* what is inside the text. I
>> suspect any problem you are having relates to how you are transporting and
>> processing the payload, not to protobuf itself. Because protobuf does not
>> check for \r, \n or \t at any point.
>>
>> Marc
>>
>>
>> On 15 May 2014 00:56, Ganesh Sangle  wrote:
>>>
>>> Hi Guys,
>>> I am trying to use protobufs between two entities - one in python and
>>> another in c++.
>>> The advantage of using protobuf is i dont have to write
>>> serializing/deserializing code.
>>>
>>> However, there is a complication.
>>> The message that I create in python world, when i serialize it to be sent
>>> over to the other side, has '\r\n'.
>>> The code on the other side is already using \n as a delimiter and cutting
>>> out the strings.
>>> So what happens is that the message string gets split and when I try to
>>> re-assemble it it doent work.
>>>
>>> Since my code is a smaller part of a larger existing system, I wanted to
>>> know if there is a way to customize/configure protobuf so that it uses some
>>> other characters instead of '\n\t', or just use a different representation
>>> (all hex numbers or whatever) when converting it to string.
>>>
>>> Thanks,
>>> Ganesh
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>>
>>> Visit this group at http://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Regards,
>>
>> Marc
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-java 2.4.X and 2.5.0 are incompatible

[Ilia Mirkin]

On Wed, Apr 16, 2014 at 10:11 PM, Patrick Wendell  wrote:
> Established based on what conventions?

On the convention of how protobuf releases have been done... Would you
be happier if it was protobuf2 4.0 and protobuf2 5.0?

> It means that if you want to write a library that uses proto-bufs you can't
> inter-operate with other libraries that also use protobufs.

Not ones that use protobufs compiled against a different major version
of the library. These come out every year or two. AFAIK this has
always been the case. (You can avoid this problem by just building the
proto as part of the build process rather than shipping the generated
files. I'm fairly sure that the generated API's should be
backwards-compatible.)

Hopefully someone from Google can address your other concerns.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-java 2.4.X and 2.5.0 are incompatible

[Ilia Mirkin]

While I don't speak for Google, I believe it's fairly well-established
that 2.4 and 2.5 are considered to be "major" releases. Switching
between them requires regenerating the java files with protoc, as the
internal APIs used by the generated code tend to change. I believe
that in general the public API's remain the same, however that doesn't
let you have multiple protobuf versions without something like jarjar.
The minor releases (2.4.0 vs 2.4.1, etc) should be binary-compatible
AFAIK.

  -ilia

On Wed, Apr 16, 2014 at 4:24 PM, Patrick Wendell  wrote:
> Hi All,
>
> I work on Apache Spark which is an open source project. We have recently
> been dealing with a lot of pain due to the fact that the Java Protobuf
> libraries for 2.4.X and 2.5.0 are not binary compatible. This makes it
> really difficult for users to include two dependencies A and B that depend
> on different versions of protobuf-java.
>
> Are these incompatibilities an omission, or is this an intentional policy
> that protobuf is okay making API-breaking changes in minor versions? This
> violates typical semantic-versioning conventions and makes it pretty tough
> for downstream users.
>
> I don't see any references to library compatibility in the Java protobuf
> page or the FAQ - apologies if this is covered somewhere...
> https://developers.google.com/protocol-buffers/docs/javatutorial
> https://developers.google.com/protocol-buffers/docs/faq
>
> - Patrick
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to parse messages coming via streaming (TCP for example)

[Ilia Mirkin]

PB's don't include any protocol at all. Your RPC mechanism will need
to provide for this. It's common to do something like a  
type of thing; you could even use the CodedOutputStream (or its
equivalent, depending on the language you're using) to emit the length
as a varint, and then stick the pb into it. You do need to
pre-serialize the pb though, so if it's going to be huge, that can be
a problem. (One could come up with ways to mitigate that of course.)

On Mon, Mar 31, 2014 at 4:43 PM, Iñaki Baz Castillo  wrote:
> Hi,
>
> I want to send PB messages to a server via a persistent TCP connection. The
> idea is that the client can send lot of messages one behind the other over
> the same TCP connection. This means that data received in the server may be
> a complete PB message, a portion of a PB message, or more than a single PB
> message.
>
> Is the protocol ready for this case? is there a parsing API for this case?
> or should I add some kind of data delimiter and perform manual parsing on
> server side in order to parse single and entire PB received messages?
>
> Thanks a lot.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C++ ParseFromCodedStream takes too long

[Ilia Mirkin]

On Thu, Mar 20, 2014 at 8:40 AM,   wrote:
> The message will be changed. Score isn't optional. Every docid has a score.
>
> I could try it as you suggested. Without the message.
>
> But I'm a little bit confused.
>
> I tried it with Java and with C++. Java can parse the Response in less than
> a second. C++ takes 20-25 seconds. Both times it is the same Message
> ~2.600.000 bytes large and with ~2.100.000 repeates. How can this be?

Your sizes must be wrong. If score isn't optional, your current
message should be

(
) x2.1M

That means that each entry will take at least 9 bytes, more if the
docid > 127. I don't see how this can only take up 2.6MB...

> Shouldn't C++ be faster than Java?

Yes, it should be :) I'll let someone else answer here... You could
profile the C++ code and see where the time is going...

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C++ ParseFromCodedStream takes too long

[Ilia Mirkin]

On Thu, Mar 20, 2014 at 3:28 AM,   wrote:
> Hello,
>
> I have a message type like this:
>
> message SearchResponse {
> enum Status {
> OK = 200;
> BAD_REQUEST = 400;
> REQUEST_TIMEOUT = 408;
> INTERNAL_SERVER_ERROR = 500;
> }
> required Status status = 1;
>
> message Result {
> required int32 docid = 1;
> optional float score = 2;
> }
> repeated Result result = 2;
> }
>
> Now it it possible that the result repeats 5.000.000 times or more. At the
> moment I'm working with about 2.100.000 repeats and the ParseFromCodedStream
> function takes about 25 seconds.
>
> My Code looks like:
>
> google::protobuf::io::ArrayInputStream arrayIn(buffRecive, recived);
> google::protobuf::io::CodedInputStream codedIn(&arrayIn);
> google::protobuf::io::CodedInputStream::Limit msgLimit =
> codedIn.PushLimit(recived);
> srchResp.ParseFromCodedStream(&codedIn);
> codedIn.PopLimit(msgLimit);
>
> The parsing and everything works great, but the performance is the problem.
> The message with 2.100.000 repeats has around 2.600.000 bytes.
> Is there a way to improve the performance?

If that message structure can be changed, I'd suggest the following:

repeated int64 docid = 2 [packed=true];
repeated float score = 3 [packed=true];

Use the low bit of docid to indicate whether there's a score (hence
the int32 -> int64 change... if your docid's fit in 31 bits, no need
for that). So your encoding algo would be like

add_docid(docid << 1 | !!has_score)
if (has_score) add_score(score)

And your reading algo would be the inverse. Unfortunately this means
there's no easy way to seek to a particular doc id and get its score,
so some post-processing on the receiver end will be necessary.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.4.1 headers w/ 2.5.0 link-time library (?)

[Ilia Mirkin]

On Tue, Mar 4, 2014 at 11:52 PM, Jon Wooding  wrote:
> Hopefully this question isn't a waste of your time.
> I'm trying to build a program using OpenLighting Architecture and I'm
> receiving an error that OLA requires 2.5.0, but installed is 2.4.1. "Make
> sure that your headers are from the same version . . ."
> In terminal "protoc --version" shows 2.5.0, but if I uninstall (apt-get or
> manual) it reverts to 2.4.1.  I believe there is a 2.4.1 installed somewhere
> on this machine that is invisible to apt-get and I don't recall installing a
> 2.4.1...  I've got this set-up working on an identical machine so it is
> certainly something that I have done incorrectly.  Any ideas?

locate protoc

Should find all files named 'protoc' on your system.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protocol Buffers and Generics

[Ilia Mirkin]

On Sun, Mar 9, 2014 at 4:05 AM, Ori Popowski  wrote:
> Hello,
>
> I want to created a protobuf based infra for inter-service communication. It
> means that this infra should be able to pass any type of Java object. So
> we'll have a Java wrapper class with all meta-data about the passed object
> that is agreed between all clients of the infra, and this class will contain
> a generic type, say T, which represents the object itself.
>
> I experimented a bit and found out that it is not possible to define generic
> types in Protocol Buffers.
> Is there any solution to this?

Are you looking for 'bytes'? Something's going to have to serialize
that java object to bytes at some point, you can just stick it into
the protobuf then.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] If proto Foo and proto Bar have exactly the same fields, can I use Bar to parse a serialized Foo message?

[Ilia Mirkin]

On Tue, Mar 11, 2014 at 7:28 PM, Zihan Liu  wrote:
> If proto Foo and proto Bar have exactly the same fields, can I use Bar to
> parse a serialized Foo message?

Yes. The actual message name (or any of the field names for that
matter) are not encoded. The only things that matter are the field/tag
id's and types.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Serialization problem with repeated nested filed sending through a socket

[Ilia Mirkin]

On inspection, your code seems fine. (Why you bother to create a
mmChar instead of just passing mmStr.data(), mmStr.length() to send is
unclear to me, but I'm guessing you're trying to replicate the larger
codebase.) It's been long enough that I don't remember off-hand if
send() is guaranteed to send everything on a TCP socket (it definitely
isn't if you've set a E_NONBLOCK flag on the fd, or on a
non-SOCK_STREAM/SEQPACKET socket). Of course a signal to the process
would definitely make send return early though, so to be safe you need
a retry loop around it.

I would _really_ recommend looking at tcpdump output to make sure that
the full data is sent instead of speculating though. And also to make
the server end print the data received.

  -ilia

On Wed, Mar 12, 2014 at 10:20 AM, Ke Wang  wrote:
> OK, Thanks. Let me post my code here a little bit:
>
> message MatrixMsg
> {
> required string msgType = 1;
> optional int64 count = 2;
> optional string extraInfo = 3;
> message TaskMsg
> {
> required string taskId = 1;
> required string user = 2;
> required string dir = 3;
> required string cmd = 4;
> required int64 dataLength = 5;
> }
> repeated TaskMsg tasks = 4;
> }
>
> In my code, at some place, I did something like the following:
>
> MatrixMsg mm;
> mm.set_msgtype("client send task");
> mm.set_count(100);
>
> for (int i = 0; i < 100; i++)
> {
> MatrixMsg_TaskMsg *tm = mm.add_tasks();
> tm->set_taskid("someid");
> tm->set_user("kwang");
> tm->set_dir("somedir");
> tm->set_cmd("somecmd");
> tm->set_datalength(0);
> }
>
> string mmStr = mm.SerializeAsString();
> char *mmChar = new char[mmStr.length()];
>
> for (int i = 0; i < mmStr.length(); i+=)
> {
> mmChar[i] = mmStr[i];
> }
>
> /*
>  create some socket "sockfd" here
> */
>
> send(sockfd, mmChar, mmStr.length(), 0);
>
> Now, at the sever side, I still received the message that got truncated.
> Does that mean the server hasn't finished received the whole data in
> "mmChar", do I need to do a loop receive until I get all the messages?
> Thanks!
>
> Ke
>
> On Wednesday, March 12, 2014 8:56:32 AM UTC-5, Ilia Mirkin wrote:
>>
>> On Wed, Mar 12, 2014 at 9:48 AM, Ke Wang  wrote:
>> > Thanks, but when I print out the char* using string.data(), it got
>> > truncated
>> > too.
>>
>> Right... most print functions will stop when they see a null
>> character... you can't use printf/cout/etc -- you'd have to write a
>> custom print mechanism that e.g. printed out hex. For example
>>
>> void print_data(const std::string& str) {
>>   for (int i = 0; i < str.length(); i++) {
>> if (i && (i % 20) == 0) printf("\n");
>> printf("%02X ", str[i]);
>>   }
>>   if (i % 20) printf("\n");
>> }
>>
>> Or something along those lines. (I don't actually remember if str[i]
>> works, but if not, should be easy to do e.g. str.data()[i] or
>> something.)
>>
>> > Later, I need to transfer this char* through TCP socket, which
>> > apparently cannot be decoded right at the server side. Did I do
>> > something
>> > wrong with the definition of the message?
>>
>> Check what data is sent over the socket with e.g. tcpdump and make
>> sure that it's the data you expect. Make sure that if you're never
>> handling the string as a char*. It needs to always be a (char *,
>> length) pair (which is essentially what std::string is). Otherwise the
>> implicit termination of c-style strings (0 byte) will break things.
>>
>>   -ilia
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Serialization problem with repeated nested filed sending through a socket

[Ilia Mirkin]

On Wed, Mar 12, 2014 at 9:48 AM, Ke Wang  wrote:
> Thanks, but when I print out the char* using string.data(), it got truncated
> too.

Right... most print functions will stop when they see a null
character... you can't use printf/cout/etc -- you'd have to write a
custom print mechanism that e.g. printed out hex. For example

void print_data(const std::string& str) {
  for (int i = 0; i < str.length(); i++) {
if (i && (i % 20) == 0) printf("\n");
printf("%02X ", str[i]);
  }
  if (i % 20) printf("\n");
}

Or something along those lines. (I don't actually remember if str[i]
works, but if not, should be easy to do e.g. str.data()[i] or
something.)

> Later, I need to transfer this char* through TCP socket, which
> apparently cannot be decoded right at the server side. Did I do something
> wrong with the definition of the message?

Check what data is sent over the socket with e.g. tcpdump and make
sure that it's the data you expect. Make sure that if you're never
handling the string as a char*. It needs to always be a (char *,
length) pair (which is essentially what std::string is). Otherwise the
implicit termination of c-style strings (0 byte) will break things.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Serialization problem with repeated nested filed sending through a socket

[Ilia Mirkin]

On Tue, Mar 11, 2014 at 11:09 PM, Ke Wang  wrote:
> Hi all,
>
> I am using google protocol buffer to transmit complex data structures over
> socket. Here is the .proto file:
>
> message MatrixMsg
> {
> required string msgType = 1;
> optional int64 count = 2;
> optional string extraInfo = 3;
> message TaskMsg
> {
> required string taskId = 1;
> required string user = 2;
> required string dir = 3;
> required string cmd = 4;
> required int64 dataLength = 5;
> }
> repeated TaskMsg tasks = 4;
> }
>
> I serialized a MatrixMsg to string through SerializeAsString(), and the
> string length is 500. Now, I want to convert the string to char* in order to
> send it through socket. However, when I converted to char* through
> string.c_str(), the string got truncated. I printed out the string, and
> figured out there are white space in it. How do I get a char* that is
> exactly matches the string? Thanks!

If you have a std::string, you can get at the data with str.data(),
and the length is str.length(). There can be null characters in the
output, so things like strlen won't produce useful results. [And
there's some subtle difference between .data() and .c_str() but I
don't quite remember what it is, and it might only be a theoretical
difference and not an actual one given the gcc implementation. My
memory is a little hazy on the topic though.]

HTH,

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to extract enumeration value with its comments from a protobuf descriptor file ?

[Ilia Mirkin]

On Thu, Feb 27, 2014 at 11:04 AM,   wrote:
> Hello,
>
> I have a proto buf file with an enumeration with leading comments.
>
> I do this commands to obtains FileDescriptorSet :
> protoc.exe --cpp_out=proto File.proto --include_imports
> --include_source_info -oFile.desc
>
> I want extract in C++ all enumerations values and its comments from this
> file descriptor.
>
> I can obtain the comments with SourceCodeInfo, I can obtains the name of
> enum value with
> EnumValueDescriptorProto.
>
> But how obtains the comment associated with a value ?
>
> I know do it with a protobuf source file, but with file descriptor I have
> not found how ?

I'm not 100% sure what your input is, but if you can get a
FileDescriptor (which you can obtain from a FileDescriptorProto using
a DescriptorPool, or if it's compiled in, it's much simpler...
message->descriptor->file() should do it), then you can look at all of
its enums by using enum_type(i). This returns an EnumDescriptor, which
in turn lets you get at the EnumValueDescriptors, which in turn have a
name/number.

HTH,

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Serialize BufferedImage with Proto

[Ilia Mirkin]

What are the alternatives? One way or another, you have to create a
representation of the object... enough to be able to create a new one
on deserialization.

On Wed, Feb 26, 2014 at 10:34 PM, jCoder  wrote:
> I was afraid of that answer.
> I feel like sending the whole object to a byte[] would cost just as much if
> not more time then it currently takes.
>
> On Wednesday, February 26, 2014 10:22:20 PM UTC-5, Ilia Mirkin wrote:
>>
>> On Wed, Feb 26, 2014 at 10:13 PM, jCoder  wrote:
>> > I was able to change the BufferedImage to an ImageIcon which is
>> > Serializable.
>> > This along with using the FST Serialization class/jar I was able to
>> > improve
>> > the over all write time to about 10 seconds for a total write size of
>> > about
>> > 140 MB
>> > Which is still only 14MB/s write speed when I know I can write at
>> > 150+MB/s
>> >
>> > My question is can ProtoBuf handle serializing an ImageIcon, Color, or
>> > any
>> > Java objects besides basic Strings and Ints? and is it faster the FST?
>>
>> Protobuf does not handle serialization of Java objects. You can give
>> it a byte[] array (or an int/float/etc or a String). Hence my
>> suggestion for the benchmark. I suspect that the majority of the time
>> is going to computing the byte[] array representing your object, which
>> you'd have to do for protobuf anyways.
>>
>> >
>> > Also one of my custom object classes has a nested object class inside of
>> > it,
>> > will this slow down serialization?
>> >
>> >
>> > On Wednesday, February 26, 2014 9:10:24 PM UTC-5, Ilia Mirkin wrote:
>> >>
>> >> On Tue, Feb 25, 2014 at 3:03 PM, jCoder  wrote:
>> >> > I was wondering if anyone had any success implementing this with a
>> >> > HashMap
>> >> > that has a pointer to a BufferedImage.
>> >> > Example:
>> >> > Map thumb = new HashMap();
>> >> >
>> >> > I am currently using Serialization with a custom writeObject() and
>> >> > readObject() to turn the BufferedImage into a byte[] and back again.
>> >> >
>> >> > However this process takes roughly 25 second+ for approximately
>> >> > 18,000
>> >> > BufferedImages (size: 16pixels x 12pixels) previously loaded into
>> >> > memory
>> >> > to
>> >> > be serialized into a file (resulting size 9,408 KB).
>> >> > Please note this all happening on a SSD (so disk write speed should
>> >> > not
>> >> > be
>> >> > an issue).
>> >> >
>> >> > There has be a faster way to do this perhaps Protocol Buffers can
>> >> > help,
>> >> > I am
>> >> > just not sure the best way to handle a BufferedImage with it would
>> >> > be.
>> >> >
>> >> > Any help would be greatly appreciated.
>> >>
>> >> Try writing a benchmark that simply converts the BufferedImages to
>> >> byte[] and throws away the results. That's a lower-bound on your
>> >> overall serialization speed (without switching away from BufferedImage
>> >> to something else).
>> >>
>> >>   -ilia
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Serialize BufferedImage with Proto

[Ilia Mirkin]

On Wed, Feb 26, 2014 at 10:13 PM, jCoder  wrote:
> I was able to change the BufferedImage to an ImageIcon which is
> Serializable.
> This along with using the FST Serialization class/jar I was able to improve
> the over all write time to about 10 seconds for a total write size of about
> 140 MB
> Which is still only 14MB/s write speed when I know I can write at 150+MB/s
>
> My question is can ProtoBuf handle serializing an ImageIcon, Color, or any
> Java objects besides basic Strings and Ints? and is it faster the FST?

Protobuf does not handle serialization of Java objects. You can give
it a byte[] array (or an int/float/etc or a String). Hence my
suggestion for the benchmark. I suspect that the majority of the time
is going to computing the byte[] array representing your object, which
you'd have to do for protobuf anyways.

>
> Also one of my custom object classes has a nested object class inside of it,
> will this slow down serialization?
>
>
> On Wednesday, February 26, 2014 9:10:24 PM UTC-5, Ilia Mirkin wrote:
>>
>> On Tue, Feb 25, 2014 at 3:03 PM, jCoder  wrote:
>> > I was wondering if anyone had any success implementing this with a
>> > HashMap
>> > that has a pointer to a BufferedImage.
>> > Example:
>> > Map thumb = new HashMap();
>> >
>> > I am currently using Serialization with a custom writeObject() and
>> > readObject() to turn the BufferedImage into a byte[] and back again.
>> >
>> > However this process takes roughly 25 second+ for approximately 18,000
>> > BufferedImages (size: 16pixels x 12pixels) previously loaded into memory
>> > to
>> > be serialized into a file (resulting size 9,408 KB).
>> > Please note this all happening on a SSD (so disk write speed should not
>> > be
>> > an issue).
>> >
>> > There has be a faster way to do this perhaps Protocol Buffers can help,
>> > I am
>> > just not sure the best way to handle a BufferedImage with it would be.
>> >
>> > Any help would be greatly appreciated.
>>
>> Try writing a benchmark that simply converts the BufferedImages to
>> byte[] and throws away the results. That's a lower-bound on your
>> overall serialization speed (without switching away from BufferedImage
>> to something else).
>>
>>   -ilia
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Serialize BufferedImage with Proto

[Ilia Mirkin]

On Tue, Feb 25, 2014 at 3:03 PM, jCoder  wrote:
> I was wondering if anyone had any success implementing this with a HashMap
> that has a pointer to a BufferedImage.
> Example:
> Map thumb = new HashMap();
>
> I am currently using Serialization with a custom writeObject() and
> readObject() to turn the BufferedImage into a byte[] and back again.
>
> However this process takes roughly 25 second+ for approximately 18,000
> BufferedImages (size: 16pixels x 12pixels) previously loaded into memory to
> be serialized into a file (resulting size 9,408 KB).
> Please note this all happening on a SSD (so disk write speed should not be
> an issue).
>
> There has be a faster way to do this perhaps Protocol Buffers can help, I am
> just not sure the best way to handle a BufferedImage with it would be.
>
> Any help would be greatly appreciated.

Try writing a benchmark that simply converts the BufferedImages to
byte[] and throws away the results. That's a lower-bound on your
overall serialization speed (without switching away from BufferedImage
to something else).

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] ruby parse_from_file ?

[Ilia Mirkin]

On Sun, Feb 16, 2014 at 11:09 AM, Jay Wilson  wrote:
> I created some code that puts 10 GPB messages into a file. When I look at
> the file with a hex editor all of the messages are in the file.
>
> I now want to read the messages back in, processing them 1 at a time. The
> following produces only the last record in the input file. What am I
> missing? How do I read each message individually?

I think you forgot to create a record format. A common one is 
 (where the varint says how many bytes the next record is).

>
>
> Yes I know the below is an infinite loop.
>
> require 'protobuf/message/message'
> require 'protobuf/message/enum'
> require 'protobuf/message/service'
> require 'protobuf/message/extend'
>
> module Test
>   class Interface < ::Protobuf::Message
> defined_in __FILE__
> required :string, :name, 1
> optional :bool, :deleted, 2
>   end
>   class System < ::Protobuf::Message
> defined_in __FILE__
> required :string, :name, 1
> optional :bool, :deleted, 2
>   end
>   class Rec < ::Protobuf::Message
> defined_in __FILE__
> optional :uint32, :timestamp, 1
> optional :System, :system, 2
> repeated :Interface, :interface, 3
>   end
> end
>
>
> jrec = Test::Rec.new
> counter = 1
> file = File.open("test", "r")
> while (jrec.parse_from_file(file))
>puts "#{jrec.system.name}: #{counter}"
>counter = counter + 1
> end
> file.close
>
>
> if I do a jrec.display, the last message in the file is what I see.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Generate .pb with pb4php library?

[Ilia Mirkin]

A quick search for pb4php turns up

https://code.google.com/p/pb4php/

which in turn has a link to

http://coderpeek.com/2008-07-17-protocol-buffer-for-php

And the example on there is

"""
$book = new AddressBook();
$person = $book->add_person();
$person->set_name('Kordulla');
$person->set_surname('Nikolai');

$phone_number = $person->add_phone();
$phone_number->set_number('49');
$phone_number->set_type(Person_PhoneType::WORK);

$phone_number = $person->add_phone();
$phone_number->set_number('171');
$phone_number->set_type(Person_PhoneType::MOBILE);

// serialize
$string = $book->SerializeToString();
"""

I guess in order for that to work you need to do something like (from
the first url):

"""
require_once('../parser/pb_parser.php');
$test = new PBParser();
$test->parse('./performance.proto');
"""

which creates the classes specified in the proto at runtime,... Can
you clarify what you're confused by?

On Tue, Feb 11, 2014 at 5:51 AM, Joel Zimmerli  wrote:
> Hi,
>
> I need to provide data in Protocol buffers format to Google (transportation
> alert) but I cannot instal/configure a compiler on my website (limitation
> due to the hosting solution).
>
> Fortunately, I found "pb4php" and I'm wondering whether this solution is
> still supported?
>
> I downloaded protocolbuf_025.zip and can load a first sample
> "example/protoc.php" with a good result "string(18) "File parsing done!"".
>
> Now I would like to "generate" a file to send it to Google (generate a .pb
> file), but I cannot understand/see how to do that using the different PHP
> scripts available.
>
> Hope someone can help me on this issue.
> Regards
> Joel
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] How to Create new Unknown Fields on a Message?

[Ilia Mirkin]

message->GetReflection()->MutableUnknownFields() should get you
a UnknownFieldSet. I believe you can manipulate that.
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.unknown_field_set#UnknownFieldSet.


On Fri, Feb 7, 2014 at 2:52 AM, Gregory Hlavac  wrote:

> I'm rewriting a JSON <-> Protobuf bridge and I want to allow it to handle
> unknown fields in the JSON but for the life of me I can't figure out how to
> set unknown fields with C++.
>
>
> Another thing I've noticed is
>
> string *
> *mutable_length_delimited*()
> Is that an unknown string field? Because its named oddly.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Reading .pb file

[Ilia Mirkin]

You can use protoc --decode_raw to decode it. It will give you a
probable decoding of the message -- some specifics can't be known from
the encoding alone (e.g. string vs submessage, various types of ints,
etc are confusable).

You could implement this same functionality using the libraries
(protoc does it, so there must be a way!) but... it seems like it'd be
easier to create a proto file based on the sample for you to use and
work with that.

On Wed, Feb 5, 2014 at 9:49 PM, Vijay Miriyala  wrote:
> I am new to ProtoBuff file formats. I received a zip file which contains .pb
> file. I do not have .proto file. Is there a way to read the contents of the
> .pbp file in Java using the libraries?
>
> Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Finding encoded size of var int?

[Ilia Mirkin]

On Mon, Jan 6, 2014 at 2:07 PM,   wrote:
> Sorry if this has been covered before. I searched but couldn't find a
> complete answer (or at least what I thought was complete).
>
> When I write a varint to a coded output stream via
> coded_stream.WriteVarInt32([some value]) is it possible to just do a quick
> calculation to find the number of bytes that would be written to the stream
> in that scenario just based on the value of the integer passed in?
>
> Is there any additional overhead to indicate that it is a varint when
> encoded to the stream or is the varint size just the same calculation as
> dictated in the language docs (here
> https://developers.google.com/protocol-buffers/docs/encoding?hl=zh-CN#varints).
> Obviously one easy way to find the size out be to create a coded output
> stream, write the varint to it and then find the byte size difference. I'm
> just wondering if there is a better/faster way than having to construct and
> delete a coded output stream for a calculation.

Find the highest bit set on your integer, divide by 7 and round up
(i.e. + 6 / 7) -- that should be the number of bytes it takes to
encode the varint. On x86, there is a bsr instruction which computes
ilog2 (with gcc you could do e.g. 32 - __builtin_clz(var)).

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Handeling large amount of data

[Ilia Mirkin]

If data size is an issue for you (and compatibility with an existing
protocol isn't), consider adding [packed=true] on the x/y/z arrays. I
would imagine that it will make serialization faster, and should
definitely decrease data size, assuming those arrays are bigger than a
2-3 items ("tag length, item item item item" vs "tag item tag item tag
item").

On Tue, Dec 3, 2013 at 5:23 AM, Mathijs Jonker  wrote:
>
>
> Op dinsdag 3 december 2013 10:33:47 UTC+1 schreef Ilia Mirkin:
>>
>> You mentioned you added a reserve function. Are you talking about
>>
>> accelContainer.mutable_data()->Reserve(event.size)?
>
>
> Yes
>
>>
>> Unfortunately that will still construct the object when calling
>> add_data. You may consider getting rid of the sub-object and just
>> having repeated x/y/z. It's not as nice, but should be faster.
>
>
> This method does yield more performance. The data usage is smaller aswell.
> I'll stick to this method and try to optimize further.
>
>> Where does event.data come from? Perhaps it can be a RepeatedPtrField
>> that is populated with the accel data as it comes in? Then you can
>> just use ->Swap() to copy it into place.
>>
>> On Tue, Dec 3, 2013 at 4:24 AM, Mathijs Jonker  wrote:
>> > Is there some other way that I can use so the elements are added faster?
>> >
>> > Op maandag 2 december 2013 21:15:41 UTC+1 schreef Mathijs Jonker:
>> >>
>> >> Sorry I meant microseconds, my bad.
>> >> The total time spend however is 0.256 seconds.
>> >> Which I find quite long.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Handeling large amount of data

[Ilia Mirkin]

You mentioned you added a reserve function. Are you talking about

accelContainer.mutable_data()->Reserve(event.size)?

Unfortunately that will still construct the object when calling
add_data. You may consider getting rid of the sub-object and just
having repeated x/y/z. It's not as nice, but should be faster.

Where does event.data come from? Perhaps it can be a RepeatedPtrField
that is populated with the accel data as it comes in? Then you can
just use ->Swap() to copy it into place.

On Tue, Dec 3, 2013 at 4:24 AM, Mathijs Jonker  wrote:
> Is there some other way that I can use so the elements are added faster?
>
> Op maandag 2 december 2013 21:15:41 UTC+1 schreef Mathijs Jonker:
>>
>> Sorry I meant microseconds, my bad.
>> The total time spend however is 0.256 seconds.
>> Which I find quite long.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Re: Parsing sub message not working with certain message

[Ilia Mirkin]

Well, fundamentally, it should all work fine. So you're doing
something wrong. Your best bet for getting further help would be
supplying a full repro, i.e. the full proto spec + data being parsed.
The fact that it says 9 {} and not details {} means that it doesn't
seem to know about the details field at all. Try changing "test = 1"
to "testasdf = 1" and see if that is reflected in the output...



On Fri, Oct 25, 2013 at 3:28 PM, Jesse Carter  wrote:
> Please anyone? :s I really can't understand what's wrong...
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] can client works on 2.4.1 and server works on 2.5.0

[Ilia Mirkin]

Yes, protobuf wire format should remain constant across all versions.
If you have any indication that it doesn't, you should report a bug.
(However internally, there can be API changes between versions, so you
might need to adjust code.)

On Fri, Oct 11, 2013 at 4:53 AM,   wrote:
> Hi all, now I would like to upgrade to 2.5.0, just have question, can I just
> upgrade server side to 2.5.0, and keep client side with lower version(like,
> 2.4.1)?   Thanks very much for your help.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Repeated combined with Required

[Ilia Mirkin]

Protobuf is about serializing and deserializing data, not enforcing
restrictions on it. ("required" being the odd case there, and it is
often recommended that you avoid it.) Using repeated in the spec, and
then enforcing any additional restrictions in your code is the
preferred way to go. That way, different users will be compatible at
the wire level, but you are free to change requirements as necessary.

On Tue, Oct 8, 2013 at 1:52 AM, Dror Cohen  wrote:
> Hi,
> if I have a field that is required to show at least 1 time but can occur a
> few times - What's the expected decleration in the proto file?
> I can think of the following options:
> 1. Just using repeated isn't good enough - it has to appear at least once.
> 2. Use first option and force a check in my code - seems to misuse the whole
> versionning system.
> 3. Using one field with required and then a field with repeated - code looks
> bad.
>
> I'm inclined to the first option - only because in the future the field
> might become totally optional/removed (not that I can think of that
> happening but you can never know).
>
> Your thoughts/ Suggestions?
>
> Dror
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] enum to string example

[Ilia Mirkin]

You need to get a EnumDescriptor for the field in question, and then
retrieve the EnumValueDescriptor using FindValueByNumber. That will
have all the name information you need. To get the EnumDescriptor of
an enum Foo, you can use Foo_descriptor() which should be in the
generated code. I assume this is similar to what NameOfEnum does
(should be easy to check, of course, just look at the code).

On Mon, Oct 7, 2013 at 10:31 AM, amirk  wrote:
> Hi,
>
> I am new to protobuf, Can someone please post an example of enum to string
> conversion in C++?
> I see that there is generated code for it (see below example for enum called
> XXX), but when I try to call I get a segmentation fault.
>
> Thanks,
> Amir
>
> inline const ::std::xxx& XXX_XXXIdentifier_Name(XXX_XXXIdentifier value) {
>   return ::google::protobuf::internal::NameOfEnum(
> XXX_XXXIdentifier_descriptor(), value);
> }
> inline bool XXX_XXXIdentifier_Parse(
> const ::std::xxx& name, XXX_XXXIdentifier* value) {
>   return ::google::protobuf::internal::ParseNamedEnum(
> XXX_XXXIdentifier_descriptor(), name, value);
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] encoding with protobuf java classes

[Ilia Mirkin]

The distinction between "text" and "binary" is fairly small and
arbitrary. Can you elaborate a bit more as to what you're seeing and
what you're expecting to see? In the protobuf encoding, any string
values should show up unaltered inside of the stream (not sure why one
would expect them to be mangled in some way).

On Thu, Oct 3, 2013 at 5:47 PM, Alan Frye  wrote:
> I am trying to write a protobuf message out to a file using java and it
> appears that all of my string properties are being written as text instead
> of the bytes. I need to be able to write the data out as binary.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] How to encode nested Python Protobuf

[Ilia Mirkin]

I believe Chris's suggestion was to base64-encode the full message,
not the subproto, or you'll run into other problems down the line,
since it sounds like your transport can't handle arbitrary byte
sequences.

IOW, something like

response.response_proto = heartbeatResult.SerializeToString()
self.sendMessage(base64.b64encode(response.SerializeToString()))

Or even better, stick the b64encode into sendMessage itself (and
matching b64decode in the receive logic). But best still would be to
figure out why your transport doesn't seem to be handling arbitrary
byte sequences.

  -ilia


On Wed, Sep 25, 2013 at 1:24 PM,   wrote:
> Chris,
> Your suggestion of trying base64 worked on the nested
> protobuf. Thanks to you and Llia again for taking the time to
> share your thoughts.
>
> Here is the solution that ended up working with the websocket
> server that I'm using.
>
>
>
>def GetHeartbeat(self):
> print "GetHeartbeat called"
> heartbeatResult = rpc_pb2.HeartbeatResult()
> heartbeatResult.service = "ALERT_SERVICE"
> heartbeatResult.timestamp = self.getTimestamp()
>
> heartbeatResult.status_cd = rpc_pb2.OK
> heartbeatResult.status_summary = "OK"
>
> response = rpc_pb2.Response()
> response.service_name = ""
> response.method_name = "SendHeartbeatResult"
> response.client_id = "ALERT_SERVICE"
> response.status_cd = rpc_pb2.OK
> response.response_proto =
> base64.b64encode(heartbeatResult.SerializeToString())
> self.sendMessage(response.SerializeToString())
> print "GetHeartbeat finished"
>
>
>
>
>
>
> On Tuesday, August 27, 2013 12:12:30 PM UTC-7, Christopher Head wrote:
>>
>> On Mon, 26 Aug 2013 17:32:23 -0700 (PDT)
>> chri...@gmail.com wrote:
>>
>> > Thanks for taking the time to read/reply.
>> >
>> > GetHeartbeat2 is my original version which I included as well and the
>> > way I had thought it should probably work. This code is serializing
>> > to string
>> > on the inner buffer and then again on the outer.
>> >
>> >response.response_proto = heartbeatResult.SerializeToString()
>> > self.sendMessage(response.SerializeToString())
>> >
>> >
>> > The GetHeartbeat2 generates this error on the server ( I included
>> > more of the stack trace this time)
>> > Message: [org.java_websocket.exceptions.InvalidDataException:
>> > java.nio.charset.MalformedInputException: Input length = 1
>> > at
>> >
>> > org.java_websocket.util.Charsetfunctions.stringUtf8(Charsetfunctions.java:80)
>> > at
>> > org.java_websocket.WebSocketImpl.deliverMessage(WebSocketImpl.java:561)
>> > at
>> > org.java_websocket.WebSocketImpl.decodeFrames(WebSocketImpl.java:328)
>> > at org.java_websocket.WebSocketImpl.decode(WebSocketImpl.java:149) at
>> >
>> > org.java_websocket.server.WebSocketServer$WebSocketWorker.run(WebSocketServer.java:593)
>> > Caused by: java.nio.charset.MalformedInputException: Input length = 1
>> > at
>> > java.nio.charset.CoderResult.throwException(CoderResult.java:277) at
>> > java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:798) at
>> >
>> > org.java_websocket.util.Charsetfunctions.stringUtf8(Charsetfunctions.java:77)
>> >
>>
>> This stack trace suggests you’re still involving UTF-8 somewhere, this
>> time on the Java side (java.nio.charset, stringUtf8 method name, etc.).
>> Again, an encoded Protobuf message is a BYTE STRING. You should not be
>> doing UTF-8 things, or any other text-related things, to it. I have no
>> idea how Websockets work, but if they are not capable of transporting
>> byte strings directly (if they only carry text, for example), then you
>> will have to do further encoding on your Protobuf message—Base64 might
>> be suitable here.
>>
>> Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Error Setting Repeated Fields

[Ilia Mirkin]

A function is a member of a class. There is no set_field() function if
field is repeated. Take a look at
https://developers.google.com/protocol-buffers/docs/cpptutorial --
there's add_field() and mutable_field() depending on what you want to
do.

  -ilia

On Sat, Sep 7, 2013 at 1:38 PM, Austin Luu  wrote:
> Hi,
>
> I have a message class:
>
> message Foo {
> repeated Foo2 field = 1;
> }
> (Foo2 is another message)
>
> and in my cpp file:
>
> Foo* ci1;
> Foo* ci2;
> //call some function to to assign values to ci1->field(0)
> function( ci1 )
> //try to copy ci1-field(0) to ci2->field(0)
> ci2->set_field( 0, ci1->field(0) );
>
> But I get an error message:
> error: 'Foo' has no member named 'set_field'
> Why is it reading set_field as a member instead of the function set_ the
> member field?
> I'm new to protocol buffers so any help will be appreciated!
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] double encoding

[Ilia Mirkin]

It's just the standard IEEE 64-bit double encoding...

>>> struct.pack('d', 1.0)
'\x00\x00\x00\x00\x00\x00\xf0?'

[and of course ? == 0x3F]

On Mon, Sep 2, 2013 at 11:23 AM, Sébastien Muscat  wrote:
> Hi,
>
> I'm new with protocol buiffer and I try to understand the double encoding.
>
> I try to encode 1 and the result value is :
>
> 09 00 00 00 00 00 00 F0 3F
>
> I understand that 09 is tag and wire type and the next 8 bytes are the value
> 1 encoded.
>
> But how 1 is encoded into F0 3F ?
>
> thanks for your answers
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] How to encode nested Python Protobuf

[Ilia Mirkin]

On Mon, Aug 26, 2013 at 8:32 PM,   wrote:
> Thanks for taking the time to read/reply.
>
> GetHeartbeat2 is my original version which I included as well and the
> way I had thought it should probably work. This code is serializing to
> string
> on the inner buffer and then again on the outer.
>
>response.response_proto = heartbeatResult.SerializeToString()
> self.sendMessage(response.SerializeToString())
>
>
> The GetHeartbeat2 generates this error on the server ( I included more of
> the stack trace this time)
>
> Message: [org.java_websocket.exceptions.InvalidDataException:
> java.nio.charset.MalformedInputException: Input length = 1
> at
> org.java_websocket.util.Charsetfunctions.stringUtf8(Charsetfunctions.java:80)
> at
> org.java_websocket.WebSocketImpl.deliverMessage(WebSocketImpl.java:561)
> at org.java_websocket.WebSocketImpl.decodeFrames(WebSocketImpl.java:328)
> at org.java_websocket.WebSocketImpl.decode(WebSocketImpl.java:149)
> at
> org.java_websocket.server.WebSocketServer$WebSocketWorker.run(WebSocketServer.java:593)
> Caused by: java.nio.charset.MalformedInputException: Input length = 1
> at java.nio.charset.CoderResult.throwException(CoderResult.java:277)
> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:798)
> at
> org.java_websocket.util.Charsetfunctions.stringUtf8(Charsetfunctions.java:77)

I would assume that the error is in the sendMessage / receiving logic.
Try dumping out what you're sending and what you're receiving and make
sure they match up.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] How to encode nested Python Protobuf

[Ilia Mirkin]

On Sun, Aug 25, 2013 at 12:47 AM,   wrote:
> heartbeatResult = rpc_pb2.HeartbeatResult()
> heartbeatResult.service = "ALERT_SERVICE"
> heartbeatResult.timestamp = st
> heartbeatResult.status_cd = rpc_pb2.OK
> heartbeatResult.status_summary = "OK"
>
> response = rpc_pb2.Response()
> response.service_name = ""
> response.method_name = "SendHeartbeatResult"
> response.client_id = "ALERT_SERVICE"
> response.status_cd = rpc_pb2.OK
> response.response_proto = str(heartbeatResult).encode('utf-8')

I'll admit to not being _entirely_ familiar with the python API, but
shouldn't this be

response.response_proto = heartbeatResult.SerializeToString()

I would semi-assume that str(heartbeatResult) produces a text-encoded
version, but perhaps not.

When in doubt, dump the raw protobuf data bytes received and see
what's going on.

  -ilia

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] deserializing unknown protobuf message?

[Ilia Mirkin]

Something like protoc --decode_raw?

As for manipulating these in code, there's nothing too great available
if you don't have a descriptor at all. You could create a dummy
message (no fields) and then get them via unknown fields in the
message's reflection object, but they're not easy to manipulate. Note
that the encoded information is not always sufficient for correct
decoding. For example, strings, submessages, and perhaps packed arrays
are encoded in the same way. I also don't think that whether zigzag
encoding is used on varints is part of the serialized data. You really
really really want a descriptor. If not compiled in, then passed along
with the data.

  -ilia

On Mon, Aug 19, 2013 at 3:37 PM, Tom Ward  wrote:
> Hi all,
>
> I'm just getting to grips with protobuf and, having read through the
> documentation, I'm struggling to find what I'm after.
>
> Basically I'm trying to work out how to deserialize a protobuf message
> without using the generated headers, as we're likely to get messages that
> weren't generated at compile time. I've looked through the documentation,
> but I only seem to be able to find ones that use generated classes to
> deserialize, or that use a Descriptor from a generated class to create a
> DynamicMessage, which I can't seem to work out how to do when we don't have
> the proto.
>
> Here's a very simple example of what I mean, where Message* is set to some
> base type that allows deserialization, and then can use the reflection API
> (or some factory) to correctly deal with the message:
>
> #include 
>
> #include "google/protobuf/message.h"
>
> int main(int argc, char **argv)
> {
>   if( argc > 1)
>   {
> // This will fail to compile, as Message is pure abstract
> google::protobuf::Message* message = NULL; //new
> google::protobuf::Message();
> std::istringstream ss(std::string(argv[1],strlen(argv[1])));
>
> message->ParseFromIstream(&ss);
>
> // do something with the message
> // ...
>   }
>
>   return 0;
> }
>
>
> Apologies in advance if this has already been answered...
>
> Thanks
>
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Setting string fields

[Ilia Mirkin]

Clearly you're doing _something_ wrong... but what? There's more
surrounding code (if nothing else, you can't have a function called
"main-1"), and I suspect that code might be the issue. For example,
one place I've gotten caught before is if you have

foo &a = ...;
foo &b = ...;
a = b;

That copies b into a, rather than assigning a to a reference to b.

What happens if you change your function to take a pointer rather than
a reference?

  -ilia


On Mon, Aug 19, 2013 at 1:25 PM, Alvaro Aguilera
 wrote:
> The problem is that if I use main-1 the string value is lost when leaving
> the scope of the attribute_value() function.
> Am I doing something wrong?
>
>
>
>
> On Monday, August 19, 2013 7:17:56 PM UTC+2, Ilia Mirkin wrote:
>>
>> Perhaps I'm dense, but what's the problem? That gdb doesn't print the
>> exact type info that you expected? That could well be the difference
>> between -O2 and -O0...
>>
>> On Mon, Aug 19, 2013 at 9:55 AM, Alvaro Aguilera
>>  wrote:
>> > Hello,
>> >
>> > I'm having a strange problem when setting an string value in a protocol
>> > buffer. Keep in mind that my experience in C++ and PB is rather limited.
>> > The .proto file looks more or less like this:
>> >
>> > --
>> > package buffers;
>> >
>> > message AttributeValue {
>> >
>> > enum Type {
>> > INT64   = 0;
>> > UINT64  = 1;
>> > INT32   = 2;
>> > UINT32  = 3;
>> > FLOAT   = 4;
>> > DOUBLE  = 5;
>> > INVALID = 6;
>> > STRING  = 7;
>> > }
>> >
>> > required   Type type = 1;
>> > optional   int32 i32 = 2;
>> > optional   int64 i64 = 3;
>> > optional uint32 ui32 = 4;
>> > optional uint64 ui64 = 5;
>> > optional float f = 6;
>> > optionaldouble d = 7;
>> > optional  string str = 8;
>> >
>> > }
>> >
>> > --
>> >
>> > I also have a function to convert a class "VariableDatatype" into an
>> > AttributeValue buffer. It looks like this:
>> >
>> > void BufferConverter::attribute_value(const VariableDatatype &s_atv,
>> > buffers::AttributeValue &b_atv)
>> > {
>> > switch(s_atv.type()) {
>> > case VariableDatatype::Type::STRING:
>> > b_atv.set_type(buffers::AttributeValue::STRING);
>> > b_atv.set_str(s_atv.str());
>> > break;
>> > }
>> > }
>> >
>> >
>> > In the main function I try to create a protocol buffer containing the
>> > string
>> > stored in VariableDatatype by using the attribute_value() function in
>> > the
>> > way shown below:
>> >
>> > main-1()
>> > {
>> > buffers::AttributeValue *bs = new buffers::AttributeValue();
>> > c.attribute_value(ds, *bs);
>> >
>> > }
>> >
>> > This doesn't work.
>> > If I print the value of b_atv inside the attribute_value function using
>> > gdb
>> > I get something like:
>> >
>> > gdb print b_atv
>> > 26 = (buffers::AttributeValue &) @0x6496b0: {
>> > =
>> > { = {
>> >
>> >
>> > However, when I print *bs from inside main-1, I get something different:
>> >
>> > gdb print *bs
>> > $36 = { = { =
>> > {_vptr.MessageLite = 0x642650 
>> >
>> > I can work around the problem by using the following main-2:
>> >
>> > main-2()
>> > {
>> > buffers::AttributeValue *bs = new buffers::AttributeValue();
>> > buffers::AttributeValue &bss = *bs;
>> >
>> > c.attribute_value(ds, bss);
>> > }
>> >
>> > but that's not very elegant. Can someone point me to the right way to
>> > handle
>> > this?
>> >
>> > Thank you
>> > A.
>> >
>> >
>> >
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to protobuf+u...@googlegroups.com.
>> > To post to this group, send email to prot...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/protobuf.
>> > For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Setting string fields

[Ilia Mirkin]

Perhaps I'm dense, but what's the problem? That gdb doesn't print the
exact type info that you expected? That could well be the difference
between -O2 and -O0...

On Mon, Aug 19, 2013 at 9:55 AM, Alvaro Aguilera
 wrote:
> Hello,
>
> I'm having a strange problem when setting an string value in a protocol
> buffer. Keep in mind that my experience in C++ and PB is rather limited.
> The .proto file looks more or less like this:
>
> --
> package buffers;
>
> message AttributeValue {
>
> enum Type {
> INT64   = 0;
> UINT64  = 1;
> INT32   = 2;
> UINT32  = 3;
> FLOAT   = 4;
> DOUBLE  = 5;
> INVALID = 6;
> STRING  = 7;
> }
>
> required   Type type = 1;
> optional   int32 i32 = 2;
> optional   int64 i64 = 3;
> optional uint32 ui32 = 4;
> optional uint64 ui64 = 5;
> optional float f = 6;
> optionaldouble d = 7;
> optional  string str = 8;
>
> }
>
> --
>
> I also have a function to convert a class "VariableDatatype" into an
> AttributeValue buffer. It looks like this:
>
> void BufferConverter::attribute_value(const VariableDatatype &s_atv,
> buffers::AttributeValue &b_atv)
> {
> switch(s_atv.type()) {
> case VariableDatatype::Type::STRING:
> b_atv.set_type(buffers::AttributeValue::STRING);
> b_atv.set_str(s_atv.str());
> break;
> }
> }
>
>
> In the main function I try to create a protocol buffer containing the string
> stored in VariableDatatype by using the attribute_value() function in the
> way shown below:
>
> main-1()
> {
> buffers::AttributeValue *bs = new buffers::AttributeValue();
> c.attribute_value(ds, *bs);
>
> }
>
> This doesn't work.
> If I print the value of b_atv inside the attribute_value function using gdb
> I get something like:
>
> gdb print b_atv
> 26 = (buffers::AttributeValue &) @0x6496b0: { =
> { = {
>
>
> However, when I print *bs from inside main-1, I get something different:
>
> gdb print *bs
> $36 = { = { =
> {_vptr.MessageLite = 0x642650 
>
> I can work around the problem by using the following main-2:
>
> main-2()
> {
> buffers::AttributeValue *bs = new buffers::AttributeValue();
> buffers::AttributeValue &bss = *bs;
>
> c.attribute_value(ds, bss);
> }
>
> but that's not very elegant. Can someone point me to the right way to handle
> this?
>
> Thank you
> A.
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Re: Mapper for protocol buffer entities

[Ilia Mirkin]

Is your basic question "is there an easy way to convert between a type
I created and a vastly similar type created by protoc"? AFAIK the
answer is no, definitely not in the core protobuf library. However it
shouldn't be too difficult to create something that handles most cases
with reflection. (And if you run into a reference loop, throw an
error, since that's fundamentally non-encodable using protobuf.)
However note that Java reflection is slow, and protobuf is often used
in applications where this stuff needs to be fast. Furthermore your
internal models invariably diverge from the protobuf stuff, and so you
end up with a bunch of fromProto/toProto functions anyways...

On Fri, Aug 9, 2013 at 1:36 AM, prateek sharma  wrote:
> Is it something that I did not get properly w.r.t. mapper or is my question
> invalid?
>
> Thanks,
> Pratz
>
>
> On Thursday, August 8, 2013 6:20:51 PM UTC+5:30, prateek sharma wrote:
>>
>> Hi all,
>> I am new to buffers, using it with my java project.
>> I have one doubt about initializing proto buffer entities.
>>
>> Initializing this proto buffer entity is done manually in all the examples
>> given in tutorial, examples used setters to set the values directly to proto
>> buffer entities but what when I have a base entity which I want to convert
>> to proto buffer entity? Do I need to take values from the base entity and
>> initialize proto buffer entity or there are any mapper available to do this
>> task?
>>
>> For example:
>> In my project if I have a base entity vehicle which has parameters as
>> idint
>> name  String
>> typeint
>>
>> I wrote a .proto file vehicle.proto as
>> message vehicle_proto{
>>  required int32 id = 1; // Unique ID number for this person.
>>  required string name = 2;
>>  optional string type = 3;
>> }
>>
>> It created a file VehicleProto.java.
>>
>> Now I have one entity vehicle and one entity VehicleProto and want to map
>> vehicle to VehicleProto. How can I do it automatically?
>>
>> Thanks,
>> Pratz
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.