[protobuf] Re: Documentation about endianness support

[Marc Gravell]

protobuf ensures that the *value of primitives* will be preserved; so if 
you send the uint32 23433451, it will be decoded as  23433451 regardless of 
the CPU/process endianness of sender and receiver. The details are 
here: https://protobuf.dev/programming-guides/encoding/ - but they **don't 
matter** unless you're writing your own protocol-level tooling.

What the receiver does with that value, however, is up to the receiver. For 
example, if they interpret that as IPv4 bytes using shift/mask operators: 
it will be work the same anywhere. If they interpret that as IPv4 bytes 
using "punning" or any other in-place re-interpret cast: then expect 
problems. But that's not an issue in the protobuf side - it is an issue in 
the code that *consumed it*.

On Saturday, 21 October 2023 at 07:48:03 UTC+1 Kumar bhat wrote:

> What happens if a uint32_t is sent from little endian system and received 
> by bit endian system. For eg: An Ipv4 address. Can we send the IP address 
> in any byte order and be able to see the same value in receiver of a 
> different endian machine? If thats the case, I dont see any documentation 
> for the same. 

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/b5b87de0-6cd0-471a-b517-6d2e2f62f318n%40googlegroups.com.

Re: [protobuf] Parse .proto files

[Marc Gravell]

If .NET is an option, protobuf-net has this (in the protobuf-net.Reflection
package):

var schema = """
syntax = "proto3";

package helloworld;

// My cool new message
message MyMessage {
  string some_string = 1;
  repeated int64 some_numbers = 2;
}
""";
var set = new FileDescriptorSet();
set.Add("some.proto", source: new StringReader(schema));
set.Process();

foreach (var file in set.Files)
{
Console.WriteLine($"in {file.Name}");
foreach (var msg in file.MessageTypes)
{
Console.WriteLine(msg.Name);
foreach (var field in msg.Fields)
{
Console.WriteLine($"> {field.Name}");
}
}
Console.WriteLine();
}


This is entirely managed code (i.e. no "shell to protoc"). The API is
basically identical to the Google one - the idea is that the type system is
binary compatible (i.e. you can serialize it as protobuf and you'd get the
same thing that protoc would give, in binary mode)

On Wed, 4 Oct 2023 at 10:02, 'Ran Volkovich' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Hi,
> There is a way to transform .proto file like
> syntax = "proto3";
>
> package helloworld;
>
> // My cool new message.
> message MyMessage {
> string some_string = 1;
> repeated int64 some_numbers = 2;
> }
> to FileDescriptorProto class from com.google.protobuf jar (or any other
> proto lib that can represent the proto schema)
> without to use protoc to generate descriptor before?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/ad559f9f-8112-4bbe-b0b8-43f4b64d6d60n%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxxG9kVyzsCeSQtoMgo5TrtrVZ%3DhU_9mVC6SVc-SSTe1g%40mail.gmail.com.

Re: [protobuf] Is Resulting Binary from Changing Proto Definition Source (From Nested to Non-Nested) for Fields initialization Backward-Compatible?

[Marc Gravell]

As long as you aren't using `Any`: then at the binary payload data it won't
be visible. Any existing code that uses the generated types will need to be
updated, obviously.

On Thu, 20 Apr 2023, 07:00 'Felik' via Protocol Buffers, <
protobuf@googlegroups.com> wrote:

> Hello,
>
> I came to understand that the binary serialization only concern of field
> identifiying number (hence renaming field and message is backward
> compatible).
>
> I would like to confirm my understanding if following case is also
> backward compatible (I could not find any resource that discuss about this,
> or I may have used the incorrect search term for this).
>
> Let's say previously I have defined a nested proto definition as follow:
>
> message OuterProto {
>message InnerProto {
>   optional bool field_bool = 1;
>   optional int64 field_int = 2;
>}
>
>optional InnerProto field_composed = 1;
> }
>
> Now, I would like to "un-nest" the InnerProto definition by redefining
> another duplicate proto in top level as follow:
>
> // same exact field numbering & type
> message NewInnerAsTopLevelProto {
>   optional bool field_bool = 1;
>   optional int64 field_int = 2;
> }
>
> Would following modification to message OuterProto be backward-compatible
> ?
>
> message OuterProto {
>message InnerProto {// unused
>   optional bool field_bool = 1;
>   optional int64 field_int = 2;
>}
>
>optional NewInnerAsTopLevelProto field_composed = 1;
> }
>
> Any input & insight would be much appreciated. Thank you!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/40a99e57-b80e-4bff-bf41-0622046ec12fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzziS8UL%3DwdVtS5jpTHbzfu7ux%2BRR2UtGQ4wzmLJYwX8w%40mail.gmail.com.

Re: [protobuf] Compatibility error on adding sub message type in existing proto file

[Marc Gravell]

That is ... contextual, though; an unexpected sub-message can still,
depending on the implementation, be round-tripped and potentially even
inspected via reflection. It won't usually *break* code, just like adding
any new field. It really depends how strict you're being in you'd
definitions of compatibility.

On Sat, 4 Feb 2023, 18:57 Robert Yokota,  wrote:

> Hi Komal,
>
> Forward compatibility is broken as older consumers won't understand the
> new message.
>
> Regards,
> Robert
>
> On Fri, Feb 3, 2023 at 8:50 AM Komal Kamble 
> wrote:
>
>> Hello Robert,
>>
>> Could you please elaborate on reason of having check for message_removed?
>> Adding new message doesn’t mean compatibility is lost.
>>
>> Sent from my iPhone
>>
>> On 04-Dec-2022, at 9:41 AM, Robert Yokota  wrote:
>>
>> 
>> Hi,
>>
>> I believe this is coming from the Confluent compatibility checker.  (I
>> work for Confluent.)
>>
>> A forward compatibility check is just a backward compatibility check with
>> the arguments reversed.  Since removing a message is not backward
>> compatible, adding a message is not forward compatible.
>>
>> Hope that helps,
>> Robert
>>
>> On Sun, Dec 4, 2022 at 6:35 AM 'Adam Cozzette' via Protocol Buffers <
>> protobuf@googlegroups.com> wrote:
>>
>>> I don't think we maintain any code that returns that kind of error. Do
>>> you know where that error is coming from? Is it from some external protobuf
>>> compatibility checker?
>>>
>>> On Thu, Dec 1, 2022 at 4:21 AM Komal Kamble 
>>> wrote:
>>>

 Hello Adam Cozzette,
 We are getting Found incompatibility change error of type
 MESSAGE_REMOVED.


 On 02-Nov-2022, at 11:41 PM, Adam Cozzette 
 wrote:

 
 Adding more message types to the proto file is fine and has no
 compatibility issues. Can you say more about the error you're getting?

 On Wed, Nov 2, 2022 at 7:03 AM Komal Kamble 
 wrote:

> Hello ProtocolBufferSupport team,
>
> If I have a proto file with multiple sub-message types defined and If
> I want to add more sub-message types in the same file I am getting 
> BACKWARD
> and FORWARD compatibility errors.
>
> I am using com.github.os72:protoc-jar-maven-plugin:3.11.4 plugin in
> java module to compile proto files and kafka-protobuf-provider:6.1.6 for
> ProtobufSchema object.
>
> The query is getting compatibility errors is it correct behavior or
> not?
>
> I didn't find any documentation in protocol-buffer user guide
> regarding adding more sub-message types in the existing proto file.
>
> Could you please help me to understand the standard behavior of the
> protocol buffer?
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/8373c01f-8e61-4469-80e5-82a859035f2dn%40googlegroups.com
> 
> .
>
 --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/CADqAXr5pxtdS8Lcw-oXFNPdR3FDNi6_vngBMqjh4Pm%3Dh-QfEvA%40mail.gmail.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/CAG0mBZTsaTSgjWzL0aY8Y41cJ3P-VnZ%3DimmqjN8%3DROjuM-m-DA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxwY7GUGc-EsKzsNkLSnHmPN%2B5ygovENEOMscGjWfOuxg%40mail.gmail.com.

Re: [protobuf] What does singular mean in ProtoBuf >=3.15?

[Marc Gravell]

(oh, and not "map"; "map" is just a specialized example of the semantics
from "repeated", under the hood)

On Wed, 7 Dec 2022 at 10:11, Marc Gravell  wrote:

> An 'optional' field is still singular (optional having been re-added to
> proto3 relatively recently); singular just means: not 'repeated'
>
> With regards to "wrappers.proto": all of the inner values in the wrapper
> types are singular; as stated in the text part of wrappers.proto, it isn't
> intended to use "repeated" with wrappers types, as the field
> presence doesn't align perfectly in that case. But ultimately, that's
> mostly up to your usage.
>
> Marc
>
> On Mon, 5 Dec 2022 at 11:22, Michael Vodep 
> wrote:
>
>> Hi
>>
>> I have done now some research: singular seem to be fields, that have no
>> addition field rules (like optional, repeated, map). There is no presence
>> indication.
>>
>> BUT: The docu says:
>> "a well-formed message can have zero or one of this field (but not more
>> than one)"
>>
>> What does that mean? Are this wrapper messages with 1 field meant?
>> https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/wrappers.proto
>>
>> Thanks a lot
>> Michael
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/protobuf/7743f999-de53-4861-b82b-3779a400e900n%40googlegroups.com
>> <https://groups.google.com/d/msgid/protobuf/7743f999-de53-4861-b82b-3779a400e900n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>
>
> --
> Regards,
>
> Marc
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxbkxxQAak9ZkZuvpMfCfGj7jTWzok78mHv_H0Vz1yTGw%40mail.gmail.com.

Re: [protobuf] What does singular mean in ProtoBuf >=3.15?

[Marc Gravell]

An 'optional' field is still singular (optional having been re-added to
proto3 relatively recently); singular just means: not 'repeated'

With regards to "wrappers.proto": all of the inner values in the wrapper
types are singular; as stated in the text part of wrappers.proto, it isn't
intended to use "repeated" with wrappers types, as the field
presence doesn't align perfectly in that case. But ultimately, that's
mostly up to your usage.

Marc

On Mon, 5 Dec 2022 at 11:22, Michael Vodep 
wrote:

> Hi
>
> I have done now some research: singular seem to be fields, that have no
> addition field rules (like optional, repeated, map). There is no presence
> indication.
>
> BUT: The docu says:
> "a well-formed message can have zero or one of this field (but not more
> than one)"
>
> What does that mean? Are this wrapper messages with 1 field meant?
> https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/wrappers.proto
>
> Thanks a lot
> Michael
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/7743f999-de53-4861-b82b-3779a400e900n%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxuDvQ5d%2B55f3z1Y7623Pd9UGvOxB9WZAVK5b%2B1JLEvdA%40mail.gmail.com.

Re: [protobuf] Skip certain indices while encoding

[Marc Gravell]

For 1 options: a) don't assign a value to salary (unless it is "required"
in proto2, which doesn't appear to be the case), or b) create a new
EmployeeDataMinimal message that *doesn't have a salary*, and serialize that

(I can't comment on 2; not a C++ person)

On Mon, 28 Nov 2022 at 08:14, LakshmiKanth Gupta 
wrote:

> Hello All,
>
> message EmployeeData
> {
> string name =1;
> string address = 2;
> integer salary =3;
> }
>
> 1.
> When I transmit the EmployeeData., I would like to skip "salary" from
> encoding.
> Is that possible?
> 2.
> Is it possible to tag a void pointer with employeedata object in c++
> code?  Some auxillary information
>
> All Inputs are much appreciated.
>
> Thanks in advance,
>
> with best regards,
>Gupta
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/a380b0ec-287e-4d9b-8f5c-e9cb5718ef45n%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxqvsWpVQyeS2UY7rhOpORjP1SQTn58SyrgSgfKrb%3DcLg%40mail.gmail.com.

Re: [protobuf] ParseDelimited from unreliable comm channel

[Marc Gravell]

No, protobuf does not have any resync capability. You're going to need to
wrap that yourself.

On Mon, 10 Oct 2022, 18:19 David L,  wrote:

> When serializing a message with SerializeDelimitedToOstream, is the
> expectation that parsing the resulting serial stream will re-sync if some
> data is lost (eg, if the serialized stream is sent over a radio modem)?
> Some preliminary testing that I did showed that parsing data after some
> lost bytes failed continuously thereafter. Is that expected? I had hoped
> that a parsing from a delimited stream would have some re-sync logic so
> only the message with the missing bytes would be lost.
>
> Thanks,
>
>  David
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/f05befea-8b86-4742-bf44-4cefb2733bd8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwyXry1-VfBQgZzVqGTunpY58v89wsP_BUj%3DNO0GKYMww%40mail.gmail.com.

Re: [protobuf] using

[Marc Gravell]

No

On Sat, 3 Sep 2022, 09:48 berge,  wrote:

> Hi all!
>
> in C++(>=11) I can do:
> ```
> using TMyLittleType = int64_t;
>
> TMyLittleType xyz;
> ```
>
> Is there a trick I can use to do the same in proto3?
> like
> ```
> magictrick   TMyLittleType = sint64;
>
> message XYZ
> {
>  
>   TMyLittleType abc = 3;
>  
> }
> ```
>
> Thanx,
> Berge
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/7f385218-65a2-4aec-8a0b-16b905fc919an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyR3-fUZaefciT%3D6HjGy44uzUzR9dFBZ3M1s7Pp1kt9VQ%40mail.gmail.com.

Re: [protobuf] Protobuff

[Marc Gravell]

Is this really a gRPC question? protobuf is just a payload serialization
format

If it *is* gRPC, then: *how* is the API key expected to be transmitted? Is
it a header? a payload item? Honestly, linking to the API specification
would be a good idea. Otherwise this is very vague.

On Wed, 15 Jun 2022 at 07:58, Dinny Augustine  wrote:

> Hi,
>  I am new to protobuf. I want to get the response of an API key that uses
> protobuf .I am  using PHP language. How can I accomplish this.
>
> Thanks in advance
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/84cc1c93-d7f5-4ac6-8637-ff07478faa2dn%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyZ2zUXkKFmsxHE-197hA%3DUo0ps5hLgaEOdAmHGfJRu%2BQ%40mail.gmail.com.

Re: [protobuf] protocol buffers for inter-language IPC

[Marc Gravell]

Without a long-running process, what would this look like? what would cause
the remote / out-of-process execution?

On Thu, 5 May 2022 at 00:00, 'Bill Thorp' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Inter-language interop is hard, but protocol buffers have good code
> generation across a wide variety of languages.  Are there tools for
> generating stateless IPC calls / dispatchers, without the need for sockets
> or long running processes like gRPC?  I imaginec-interop code would be
> needed to pass byte arrays around, but I hope it'd be minimal and
> independent of the data serialization and RPC logic.  Does / could this
> exist?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/845317f9-d0d6-49f0-9a28-ab58b6e01fbfn%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyhdQH_pi3nr3fJKY5Q3MFxTvMv%2BzKcttjeFppp5FHjfg%40mail.gmail.com.

Re: [protobuf] If i have a 209 byte size data of about 50 parameters, if i use protobuf to serialize it,what would the size be like?

[Marc Gravell]

You say "209 byte size" - using what measure is it 209 bytes? And what is
the data? The payload size in protobuf often depends on the specific data,
for example:

- for any fields: whether it is the default / unset value, vs whether it is
a non-detault / explicitly set value, can change whether the field is
serialized *at all*
- different values can take different sizes; for string data, we're talking
about the UTF8 length; for integer data, we can usually think of the data
as "7 bits useful data per byte", so a small integer might take one or two
bytes, but a very large integer might take 5, 7, whatever bytes (negatives
take more explaining and nuance)
- the field header size depends on the magnitude of the field number; field
7 takes 1 byte, but field 42 takes 2 bytes for the header, etc

For 50 fields, assuming we number them 1-50 and all have explicit values
and are sent, we're talking about 85 bytes just for the field headers. If
they're all small-medium integer values, maybe another 100 bytes for the
values, so round up: maybe 200 bytes. If we presume that maybe half the
fields don't have explicit values, maybe half that to 100. If the data is
strings or sub-objects, it'll be bigger.

The best thing to do for a better answer is to test it using realistic data
for your scenario, and simply measure it.

On Sun, 30 Jan 2022, 10:10 Rishabh Unikrishanan, 
wrote:

> An estimation,approximation ,docs or any additional resources would be
> helpful.Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/575d19c7-0d5e-4b86-bdec-e6a5a21d4d6cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyMQc9QzRAvUR6DMVpZvC3P652A6k8w9t9S%2BJVHdY5iTg%40mail.gmail.com.

Re: [protobuf] [ACTION REQUIRED] Protobuf Java users, please update to our latest release

[Marc Gravell]

I notice that the advisory is scant on details at the moment; is there any
mechanism for non-Google protobuf library authors to request additional
details to see whether our own implementations may be vulnerable to the
attack? Thanks

On Thu, 6 Jan 2022 at 17:15, 'Derek Perez' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Hello everyone,
>
> If you are using protobuf-java, Kotlin, or our JRuby gem
> (google-protobuf), please update to our latest release, published yesterday.
> More information about this advisory can be found here:
>
> https://github.com/protocolbuffers/protobuf/security/advisories/GHSA-wrvw-hg22-4m67
>
> Thanks!
> - Derek on behalf of the Protobuf Team
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/CAJGs%2BiKxiTSGxh872Rh7sv1pzGENX53WaHfGQnSoRzJROoApSA%40mail.gmail.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAz5KHj224JJyC1H%2BxncoUSV72snGJSW9gQb5wi1pZ_ofw%40mail.gmail.com.

Re: [protobuf] Extra dot in descriptor data (protoc java)

[Marc Gravell]

IIRC, the leading dot means that the name is absolute rather than relative.
I'm not sure it represents an error.

On Wed, 22 Dec 2021, 19:00 'Venkat Duddu' via Protocol Buffers, <
protobuf@googlegroups.com> wrote:

> We are seeing extra dot for external referenced variables in
> descriptorData in the generated java class for a given Message.
>
> *Message Definition*
> syntax = "proto3";
> package com.chegg;
>
> import "google/protobuf/struct.proto";
>
> message OneGraphRequest {
> string operation_name = 1;
> google.protobuf.Struct variables = 2;
> string query = 3;
> }
>
> *DescriptorData (*java.lang.String[] descriptorData*)* from the generated
> Class for *variables* looks like this:
> "*.google.protobuf.Struct*"
>
> It is starting with extra dot (before google), how can we suppress (remove
> the extra dot at the begining) it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/338c0e38-6c7a-435d-945b-f2e23f8a6979n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxCr%2BfQ1HozrfhyXjBZVB55x_RkashLoNpPKh7%2BkspvNw%40mail.gmail.com.

Re: [protobuf] Protobuf for Enum

[Marc Gravell]

This is specific to protobuf-net. If you're using v3, the main usage here
is to provide explicit names for the enum and (via ProtoEnumAttribute) the
defined values; the names only matter if you are generating a .proto from a
code-first model (GetSchema(), GetProto(), etc)

In v2, you can *also* use ProtoEnumAttribute to provide explicit
serialization values different to the natural value, or you can tell it to
*always* use natural values (with no attempt to consider mapped values) via
EnumPassthru on ProtoContractAttribute, however: these features are
deprecated in v3 - enums are now *always* treated as direct pass-thru
values with no attempt to map values.

On Fri, 21 May 2021 at 07:56, Rashmi Vijay  wrote:

> What is the difference it makes if we decorate the enum with ProtoContract
> in C# enum while doing Protobuf Serialization. I do get the value when i do
> deserialization irrespective of that Attribute on Enum. Then what is the
> best Practice?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/1c8d3097-c2c5-4f6d-a80b-c89ff7f05272n%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwMSPs%3D455XXZfA0fEpXQQHuJfFYP6mKd-YcDOo%2B1zsDA%40mail.gmail.com.

Re: [protobuf] How to decode the data below,thanks

[Marc Gravell]

What language/framework?

On Wed, 5 May 2021, 09:58 Danny Lee,  wrote:

> Hello everyone,
>
> There is a proto file as below.
> And it is required not to use the .options file.
>
> How to decode the data below:
> MsgInfo.data.msg2_data.data.msg22_data.jobs.job_data.job_attr.age
>
> Any useful suggestions and code will be appreciated!
>
> Thanks!
>
> ===
> syntax = "proto3";
>
>
> message MsgInfo {
>   uint32 node_id = 1;
>
>   oneof data {
> SubMsg1 msg1_data = 14;
> SubMsg2 msg2_data = 15;
>   }
> }
>
> message SubMsg1 {
>   oneof data {
> SubMsg11 msg11_data = 1;
> SubMsg12 msg12_data = 2;
>   }
> }
>
> message SubMsg2 {
>   oneof data {
> SubMsg21 msg21_data = 1;
> SubMsg22 msg22_data = 2;
>   }
> }
>
>
>
> message SubMsg11 {
>   bytes name = 1;
> }
>
> message SubMsg12 {
>   string strname = 1;
> }
>
>
> message SubMsg21 {
>   bytes name = 1;
> }
>
> message SubMsg2234 {
>   string age = 1;
>   bytes version = 2;
>   string checksum = 3;
> }
>
> message SubMsg223 {
>   string job_id = 1;
>
>   oneof job_data {
> string job_node = 16;
> SubMsg2234 job_attr = 17;
>   }
> }
>
> message SubMsg22 {
>   repeated SubMsg223 jobs = 1;
> }
> ===
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/CAAjHcFAEe2nhtRODHrsAqm1NrF1ts8x-ko5AA5vKExKOq5hbpg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzDqRSu8nrXqaF1wqyq5LGKFXTe0aVo3SVB5wEhx0CDCw%40mail.gmail.com.

Re: [protobuf] protoc v3.6 and v3.15 compatibility for different languages

[Marc Gravell]

The actual serialization format hasn't changed since protobuf was released;
as long as *your schemas* haven't changed in that time (at least, not in
incompatible ways; adding fields etc is pretty safe as long as they aren't
"required" in proto2), you should be absolutely fine. It is expected and
normal that producers and consumers might be using different
implementations / languages / library versions / etc.

Marc

On Wed, 21 Apr 2021 at 09:23, Luigi Riefolo  wrote:

> Hello people,
>
> I have two services that are exchanging protobuf messages (not over gRPC),
> one is written in Ruby and uses v3.6 while the other one is written in Go
> v3.15.
>
> Recently I've come across a change that requires an update in the Ruby
> service, as it needs to support the new timestamp format. Unfortunately
> this will not be possible for a while.
>
> So my question is: given that these two services are using different
> protobuf versions and different languages, is there any chance that I might
> introduce a compatibility issue? Or anything related?
>
>
> Regards,
>
> Luigi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/f6be32d7-7f65-45fd-8574-add61e12c8abn%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxha5SgrGjozSXLvVOQg4zAtuLP8tFpmJqZm6bVUg_AcA%40mail.gmail.com.

Re: [protobuf] Protobuf Compiler in a C# Shared Library

[Marc Gravell]

The tooling package you need here is Grpc.Tools, which should work fine in
a class library.

On Sun, 18 Apr 2021 at 23:08, Chris Langlois  wrote:

> Thanks for the recommendations. I can add .proto files to the Shared
> projects. What I cannot do is add a build action for "protobuf compiler" to
> enable automatic compilation in Visual Studio. For some reason that doesn't
> seem to work within a shared project in particular.  I tried manually
> adding the PackageReference includes for protobuf and grpc that appear in
> the client and server projects to the shared project .projitems file and it
> doesn't seem to make any difference.
> Thanks
> On Sunday, April 18, 2021 at 12:56:38 PM UTC-6 marc.g...@gmail.com wrote:
>
>> What is stopping you from adding them to the shared project? What TFM is
>> it targeting? This should JustWorkTM. You could also look at the csproj
>> changes in the client and server, and try to apply the same changes in the
>> shared project - it might tell you why it isn't happy.
>>
>> On Sun, 18 Apr 2021, 19:43 Chris Langlois,  wrote:
>>
>>> I'm using Protobuf and Visual Studio C#. I have a Client project, Server
>>> project, and and shared project. I want my proto file and messages to live
>>> within the shared project. I'm installed the Nuget packages for protobuf,
>>> protobuf tools, grpc, grpc core, and grpc tools into both my client and
>>> server projects (you cant add them to a shared project). The build action
>>> for the proto file in my shared project cannot be changed to Protobuf
>>> compiler for automatic compilation like it can if I put the .proto in
>>> either the client or server project.
>>>
>>> Any thoughts on a better architecture or what I might be doing wrong
>>> here. This seems like it should be a common architecture but it seems to be
>>> a real struggle.
>>>
>>> Thanks!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/e9a8e8e4-e085-47aa-8f62-c34d65e9b11an%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/f9d19c61-fcaf-4bee-92e2-16793d9b42fdn%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxA95_%3D9Pb_cES-joivtY59N%3DxOaqS4Tbn6treekRqZ9g%40mail.gmail.com.

Re: [protobuf] Protobuf Compiler in a C# Shared Library

[Marc Gravell]

What is stopping you from adding them to the shared project? What TFM is it
targeting? This should JustWorkTM. You could also look at the csproj
changes in the client and server, and try to apply the same changes in the
shared project - it might tell you why it isn't happy.

On Sun, 18 Apr 2021, 19:43 Chris Langlois,  wrote:

> I'm using Protobuf and Visual Studio C#. I have a Client project, Server
> project, and and shared project. I want my proto file and messages to live
> within the shared project. I'm installed the Nuget packages for protobuf,
> protobuf tools, grpc, grpc core, and grpc tools into both my client and
> server projects (you cant add them to a shared project). The build action
> for the proto file in my shared project cannot be changed to Protobuf
> compiler for automatic compilation like it can if I put the .proto in
> either the client or server project.
>
> Any thoughts on a better architecture or what I might be doing wrong here.
> This seems like it should be a common architecture but it seems to be a
> real struggle.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/e9a8e8e4-e085-47aa-8f62-c34d65e9b11an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyAwRR-b4_2gmb2CkGuiQobyKRUQSi5pPACCRuNBmPQXw%40mail.gmail.com.

Re: [protobuf] Unable to get Client Service to work

[Marc Gravell]

I'm a little confused here...

Normally, assuming that this is using the Google implementation, you would
have, in your generated code:

- a generated concrete client proxy
- a generated abstract server stub

The client code would "new" the client proxy (the yellow code). Your server
code would declare a type that subclasses the abstract stub, and register
*that*. You wouldn't need to declare your own client/server core types.
Have you perhaps simply not generated the right client/server stubs?


On Sun, 21 Feb 2021, 16:01 Herbert Falk,  wrote:

> I have the following *.proto file:
>
> package CredentialExchange;
> option go_package = "
> gitlab.com/openfmb/psm/ops/protobuf/go-/CredentialExchange";
> option java_package = "pcitek.CredentialExchange";
> option java_multiple_files = true;
> option csharp_namespace = "pcitek.CredentialExchange";
>
> import "uml.proto";
>
> // The greeting service definition.
> service CredExchangeSend {
>   // Sends a Credential Exchange message
>   rpc Cred (Management) returns (Management) ;
> }
>
> // MISSING DOCUMENTATION!!!
> message sendCredentialsRequest
> {
> // MISSING DOCUMENTATION!!!
> bytes encrypted = 1 [(uml.option_required_field) = true,
> (uml.option_multiplicity_min) = 1];
> }
>
> // MISSING DOCUMENTATION!!!
> message Management
> {
> option (uml.option_openfmb_profile) = true;
> // Version of the redundancy protocol
> uint32 version = 1 [(uml.option_required_field) = true,
> (uml.option_multiplicity_min) = 1];
> // Is a number that can be used to correlate request/responses
> uint32 messageID = 2 [(uml.option_required_field) = true,
> (uml.option_multiplicity_min) = 1];
> // Date and Time at which the message was started to be encoded.
> string utcTimeOfMessage = 3 [(uml.option_required_field) = true,
> (uml.option_multiplicity_min) = 1];
> // MISSING DOCUMENTATION!!!
> sendCredentialsRequest sendCredentialsRequest = 4;
> }
>
> I am trying to stage a Client Service for CredExchangeSend.
>
> My current service declaration is:
>
> public  class CredentialExchangeService
> {
> public Task rxd(
> pcitek.CredentialExchange.Management request)
> {
> PCItek.Redundancy.CredentialExInt.decode(request);
> return (null);
> }
>
> }
>
> For the actual send, I need to pass in the actual Management class filled
> in.
>
> This is what I am trying to do:
>
>
> namespace pcitek.CredentialExchange
> {
> public class sendCredentials
> {
> public async Task Send(pcitek.CredentialExchange.Management info)
> {
>
> string address = Redundancy.getRedundancyPeerAddress();
> using var channel = GrpcChannel.ForAddress(address+":5001");
> //
>
> var client = new
> pcitek.CredentialExchange.sendCredentials(channel)
>
>
>
>
>
>
> }
> }
> }
>
> The yellow'd statement fails because sendCredentials is not known.  Please
> note I also have: mapped a receiver service
> to: 
> endpoints.MapGrpcService();
>
> Any help would be appreciated.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/d9500f87-994e-49a7-a30f-658f77652109n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwBfFBEWjoJMQ9gBuHVhe4E9j6z7Lg_Voxz5STsY5Pm3g%40mail.gmail.com.

Re: [protobuf] Feature Request: make protocol buffer AST/parser SDK

[Marc Gravell]

You can use protoc to output the compiled schema (one of the
command-line-options - something "file descriptor set"), and deserialize
the resultant binary file as a FileDescriptorSet instance, deserializing
via your choice of language via descriptor.proto - any use?

As an aside, I also have a separate implementation of a .proto parser,
implemented in C#/.NET (so it can be used in 100% managed .NET code). Not
sure if that is any use to you, but if so: let me know.

On Wed, 17 Feb 2021, 00:12 Gerardo Mora,  wrote:

> Hello everyone, I would like to create an application that analyzes a
> .proto file and dynamically creates GraphQL queries, I don't want to parse
> the .proto files because it will outdated and probably buggy at some point,
> but I was reading the source code and the only available thing is create a
> plugin that ouputs a file after the processing which is not idea.
>
> Is there an API/SDK available for it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/f78681b8-47f2-44ac-955a-73b828f1b08dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAz3nWGiqsHRQNaEBgWN5VvnXtvP-W6SsjekDHhzFt6Ctg%40mail.gmail.com.

Re: [protobuf] key/tag encoding when message types > 31

[Marc Gravell]

Field number 76 with which wire type? The lowest 3 bits are the wire-type,
then the field number is whacked on the end. Then split into groups of 7
bits for varint encoding.

On Fri, 8 Jan 2021, 14:05 'emaz...@cisco.com' via Protocol Buffers, <
protobuf@googlegroups.com> wrote:

> Hi Marc,
>
> OK thanks. 4 bits... Right. So for 76 it would look like 0xe004 in a
> wireshark trace ?
>
> Thanks,
> Ed
>
> On Friday, January 8, 2021 at 8:59:52 AM UTC-5 marc.g...@gmail.com wrote:
>
>> The composed wire-type and field number are treated as a varint. And
>> since the MSB is reserved for continuation, after the 3-bit wire type that
>> only leaves 4 bits of field number, not 5, for single-byte field headers.
>>
>> On Fri, 8 Jan 2021, 12:47 'emaz...@cisco.com' via Protocol Buffers, <
>> prot...@googlegroups.com> wrote:
>>
>>> Hello...
>>>
>>> I appreciate the clear documentation on encoding here:
>>> https://developers.google.com/protocol-buffers/docs/encoding
>>>
>>> However it doesn't discuss how the key/tag is encoded when the number of
>>> message types exceeds 5 bits (31).
>>>
>>> Each key in the streamed message is a varint with the value (field_number
>>> << 3) | wire_type – in other words, the last three bits of the number
>>> store the wire type.
>>>
>>> What happens if the message type is 76 for example? That is 0x4c. If the
>>> data encoding is 000 for a varint then this is just  0x0260. Is that how it
>>> is represented in the data stream? Or is it encoded like a varint which I
>>> *think* is 0xe004?
>>>
>>> Appreciate any information on this.
>>>
>>> Thanks,
>>>
>>> Ed Mazurek
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/0f270391-fe31-4d0d-a736-874c0520a147n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/ed75a2d3-655f-4961-a9b8-1ad20900fa9cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwZMTjKidxCk2N_%3DdXY9ErXWyuouy%3DXy0ejCNvmaAscAw%40mail.gmail.com.

Re: [protobuf] key/tag encoding when message types > 31

[Marc Gravell]

The composed wire-type and field number are treated as a varint. And since
the MSB is reserved for continuation, after the 3-bit wire type that only
leaves 4 bits of field number, not 5, for single-byte field headers.

On Fri, 8 Jan 2021, 12:47 'emaz...@cisco.com' via Protocol Buffers, <
protobuf@googlegroups.com> wrote:

> Hello...
>
> I appreciate the clear documentation on encoding here:
> https://developers.google.com/protocol-buffers/docs/encoding
>
> However it doesn't discuss how the key/tag is encoded when the number of
> message types exceeds 5 bits (31).
>
> Each key in the streamed message is a varint with the value (field_number
> << 3) | wire_type – in other words, the last three bits of the number
> store the wire type.
>
> What happens if the message type is 76 for example? That is 0x4c. If the
> data encoding is 000 for a varint then this is just  0x0260. Is that how it
> is represented in the data stream? Or is it encoded like a varint which I
> *think* is 0xe004?
>
> Appreciate any information on this.
>
> Thanks,
>
> Ed Mazurek
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/0f270391-fe31-4d0d-a736-874c0520a147n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyuC7tQ-nEWYSFREx9JMX9S6N0GD9C9gaa4ia7_05vsxw%40mail.gmail.com.

Re: [protobuf] Use of non-standard values for index

[Marc Gravell]

100 isn't "non-standard" as such, and shouldn't cause anything to fail.
What exactly are you seeing?

The valid range is 1-536870911, omitting 19000-1 (and any reserved
areas in your specific messages) smaller numbers are cheaper (fewer bytes)
to encode, so are usually preferred - but: that's it.

I'm guessing you're actually using some non-compliant code that is assuming
field-headers are single bytes? That is true for very low field numbers (4
bytes, so: 1-15), but field 100 will take two bytes for the header. Is that
the problem here?

On Thu, 7 Jan 2021, 22:00 peo stri tise safe, 
wrote:

> What is the purpose of setting starting index values for a GPB message
> definition to something other than 1?  I am seeing a value of 100 in a
> particular application and it is causing the encoding of the message to
> fail.  Just wondered why the use of values other than 1 for the first item
> in the message.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/957aae3b-32f8-4d9d-8f5f-732e26e08cedn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzHudaTfHndUOVbYKGV2finmioPLQxwcSHFVWs9NHgSNw%40mail.gmail.com.

Re: [protobuf] Adding metadata to an RPC

[Marc Gravell]

"custom options" exist, but frankly they don't do much unless you have
custom code generation that is going to emit something useful from them,
and that "something" itself does something useful at runtime - which may
itself be tricky between runtimes/platforms.

Does "add a comment" suffice?

On Wed, 14 Oct 2020 at 08:43, Guy Pardon  wrote:

> Hi,
>
> Being new to protocol buffers and gRPC, I am looking for a good way to add
> metadata "annotations" to the contract description.
>
> In particular, we are adding transaction context propagation across gRPC
> calls. We would like to mark some RPCs (or services) as "transactional" -
> meaning that they expect / support a transaction context header in the
> incoming request.
>
> What is the recommended way of doing this? I did not seem to find this in
> the documentation.
>
> Thanks,
> Guy
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/fdbe28a3-56de-4958-b138-ca4c2e6fb896n%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAy7C%3DGXrewhSjB1-W%2BiFXV7Xrf6JBzFZmyqbKyrjX7byQ%40mail.gmail.com.

Re: [protobuf] Re: [protobuf-net] Unknow wire-type 7 when serializing List data structure

[Marc Gravell]

Hi; it is pretty hard to comment without code. Do you have a minimal repro 
that shows what you're seeing? The fact that you mention Encoding suggests 
that you're treating protobuf data as though it were text, which is never a 
good idea.

Also, since this is library specific, you may wish to move this discussion 
to https://github.com/protobuf-net/protobuf-net instead.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/1e2ca6dc-0d90-4247-98f7-66cf17ec67d9o%40googlegroups.com.

Re: [protobuf] How to Assign encoding number to the field inside message

[Marc Gravell]

See
https://developers.google.com/protocol-buffers/docs/overview#assigning_field_numbers

Basically:

- positive integers
- unique within each message (can reuse between different messages)
- lower is cheaper
- avoid some reserved ranges

On Tue, 30 Jun 2020, 20:12 rohit nv,  wrote:

>
> I am new to the gRPC and Protobuf. Wanted to know the significance of
> encoding
> number infront of field. In tutorial I saw about the usage of no between
> 1-15 for frequently used fields as it require 1 byte of encoding and 16 to
> remaining carries 2 bytes of encoding. May I know when to to use what.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/78789856-f4cf-444b-a7a7-14ff3f260595n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyi_3djk0irmdL2qnVpg1j4z2FVLBKR2kQPDsptgQOrmQ%40mail.gmail.com.

Re: [protobuf] C# Official Supported .NET Version

[Marc Gravell]

The google protobuf package (
https://www.nuget.org/packages/Google.Protobuf/) supports .NET 4.5 upwards;

protobuf-net (an independent / unaffiliated implementation of the
serializer) supports .NET 2.0 and upwards for v2.*, but: from v3.* onwards
that will change to .NET 4.6.1

Frankly, you are going to have your work cut out for you long term if you
target .NET 4.0 - since that ended life in 2016. I am going to strongly
advise you to look into moving to a more modern platform; ultimately you're
currently asking open source library maintainer to pay for your technical
debt - I've discussed this in more detail here:
https://blog.marcgravell.com/2020/01/net-core-net-5-exodus-of-net-framework.html

On Wed, 29 Apr 2020 at 10:37, Magnus Woodgate 
wrote:

> Hi,
>
> I was looking at the support for Protobuf for C# and there was no clear
> definition for which version of .NET is officially supported.
>
> Is there a table of Protobuf releases to .NET releases support or similar?
>
> Or which version of Protobuf would support .NET 4.0?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/45f9aebd-cc4c-4958-a1f4-f3c2998f892c%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAz3vW%3DsD0bEv4DcZgWhimOHiDNvB4r-J2AhutZU8S_B9A%40mail.gmail.com.

Re: [protobuf] How to control the order of field while converting a protobuf message to string using TextFormat in java?

[Marc Gravell]

I would imagine that most implementations are still going to use field
number order. Is there a specific reason you need to control it? Most JSON
tools won't care, and if you want to control the JSON, frankly you
shouldn't be using an opinionated serializer like protobuf - you should be
using a suitable highly configurable JSON-dedicated serializer for your
platform.

On Sun, 22 Mar 2020, 14:10 tuk,  wrote:

> Cross-posting from stackoverflow
> 
>
>  have a protobuf message which when converted to string using
> TextFormat.printToString()
>
> deploymentDef {
>   id: "PX3C1ED"
>   default: true
>   type: ONPREM
>   limits {
> clusterSize: 3
> limits {
>   numVMs: 18000
>   numVMsWithFlows: 18000
>   activeFlows: 600
>   totalFlows: 2400
>   flowPlanning: 400
>   numDevices: 40
> }
>   }
>   isEnterprise: false
>   brickSize: XLARGE
>   clusterSize: 3
>   description: "Default Role, Non-Enterprise, App-Discovery and Vf services 
> stopped"}
>
> The proto definition looks like below
>
> message DeploymentDef {
> optional string id = 1;
> optional bool default = 2;
> optional DeploymentType type = 3;
> optional PlatformClusterLimits limits = 4;
> repeated Role roles = 5;
> optional bool isEnterprise = 6;
> optional Configs overrides = 7;
> optional BrickSize brickSize = 8;
> optional int32 clusterSize = 9;
> optional string description = 10;}
>
> Is it possible to display description as the first field while converting
> the proto message to string using TextFormat.printToString() ?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/a3e94ffc-3723-4b70-b647-15452f4f5731%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxzh%2BNa5HZ1-6UBsGMPUiY-Hc_Dc9_DTcg3wJsNkQ_o1Q%40mail.gmail.com.

Re: [protobuf] com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has be

[Marc Gravell]

The most likely thing is that it was truncated during your transmission,
meaning: a bug in your code that serializes and/or sends the data. Can we
see any of that code? Note that you can use "protoc" (decode-raw) or
https://protogen.marcgravell.com/decode to validate a payload for integrity.

On Mon, 3 Feb 2020, 14:51 Chetan M,  wrote:

> I have a proto definition that has too many fields.I ended up just
> populating a couple of fields for test and sending them out but the
> consumer complains when parsing with the above message.Why is it so?I
> wouldn't be sending down messages with just 2-3 fields in production env
> but for testing I don't want to end up populating the 50 odd fields that
> are there on the protobuf definition.My instance has no required fields so
> it passes the validity test during send.Could you please help me here ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/42b73d13-5628-4c2b-a0d5-e1b2d3df50b2%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwkkTzhr_9RB-6w1K1uAL2-h-amhbuzbJa29Fbzr-AvLQ%40mail.gmail.com.

Re: [protobuf] Enum field encoding question

[Marc Gravell]

Hi; default values *are not sent*, especially in proto3 where zero is
default and default is zero. Likewise, the root object in a message is not
wrapped in any way - only fields *on* the root object.

This means that the binary encoding of a CommandRequest with Code.RESET is:
zero bytes, which is a perfectly valid size in protobuf.

Sometimes, people find it convenient / helpful to use a DEFAULT / NONE /
UNKNOWN = 0, with their *actual* enum values above that. This makes the
default experience easier to understand, and avoid problems.

Marc

On Thu, 16 Jan 2020 at 08:08, Burak Kirazli  wrote:

> Hello,
>
> I am new to protobuf.
>
> I have a request message like that
>
> message CommandRequest
> {
> enum Code
> {
> RESET = 0;
> ERASE_APP = 2;
> ERASE_OTA = 3;
> }
>
> Code code = 1;
> }
>
> When I am trying to encode a request with zero value, i see that encoded
> size is zero. What will be exchanged?
>
> //Qt project
>
> CommandRequest req;
> req.set_code(CommandRequest_Code::CommandRequest_Code_RESET);
> qDebug() << req.ByteSizeLong() << endl;
>
> //Output of this code is:
> 0
>
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/376341b9-b429-4abe-b9ea-aeb2834c6f58%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAx2iLESr0Q18dF3ph-6xosCHEPet468qF-X7dyojGVT-g%40mail.gmail.com.

Re: [protobuf] Re: "Architectural" question - Java -> proto -> C#

[Marc Gravell]

The context here seems to be: "I have .proto file/files, and (for some
reason) I need to make additional tweaks to them in java/c#", right?

In the case of C#, "partial classes" may be your friend, especially if all
you are trying to do is add annotations/attributes. The tooling may also
generate "partial methods", but I haven't checked recently. No idea if
something similar exists in java.

On Sat, 4 Jan 2020, 03:16 Brandon Park,  wrote:

> I'm trying to solve this exact same problem and was wondering how to use
> compiled proto code with JPA and if there have been any developments on
> this since 10 years ago.  Thanks
>
> On Monday, September 13, 2010 at 12:09:00 PM UTC-7, roberto_sc wrote:
>>
>> Hi
>>
>> I have a basic question about how to organize my project.
>> I have a client  running C# code, a server running Java code and I
>> intend to use protocol buffers to exchange data.
>> I thought I could use the .proto file to describe the classes of my
>> datamodel and generate .java and .cs and then use these generated src
>> as my datamodel. But protoc generate code for message exchange and
>> cannot be edited, for example, I cannot generate java code for my
>> Person class and add the annotations to persist using JPA.
>>
>> So, the question is, do I have to mantain 3 files - .java, .cs
>> and .proto - that represent the same thing?
>>
>> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/addf93b3-53d2-4b6b-aaeb-933aa0e7a9a4%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzoQmZzdvB9kxOrkdbriiT7t7iX4i0RksVPakQu3G3YwQ%40mail.gmail.com.

Re: [protobuf] Can the client parse the interface and message structures of the server?

[Marc Gravell]

That would prpbably be a great question for James NK over on the relevant
GitHub repo. IIRC the sample server includes reflection, though:
https://github.com/grpc/grpc-dotnet/

On Sat, 14 Dec 2019, 12:22 erik stroeken,  wrote:

> Thanks Nadav Samet
> I've been trying to implement the concepts described in
> https://github.com/grpc/grpc/blob/master/doc/server-reflection.md and for
> C#:
> https://github.com/grpc/grpc/blob/master/doc/csharp/server_reflection.md in
> the implementation of ASP.NET Core. You have to somehow configure Kestrel
> to enable Reflection first. I just don't succeed and have to give up. Does
> somebody know how to enable serverside reflection in ASP.NET Core with
> Kestrel? Or even better has a complete working example of serverside
> reflection in C# ASP.NET Core?
> Thanks, Erik
>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/18d51f1a-8cb4-481e-b9ed-3f3f454d91d8%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwecDj7QsGhr1T-3YtpL7U_oz1V6jUCjPxhV5LHQGmwXQ%40mail.gmail.com.

Re: [protobuf] How to get all dependency .proto files of a given .proto file hierarchically?

[Marc Gravell]

Untested, but can't you just use protoc with the descriptor-set mode to
build a FileDescriptorSet (IIRC) which will be the processed output of
protoc's efforts? Then parse it using the details from descriptor.proto,
and see what descriptors are in the FileDescriptorSet?

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzHeYjXi5JRDW5UcCy%3D4ncgfDdN0X-%3Dm-gCjA6oxAxD4A%40mail.gmail.com.

Re: [protobuf] Protocol Buffer Wrappers returning single character

[Marc Gravell]

Can you show what you mean by wrappers here? If you dump the payload and
inspect it (protoc, or there's a tool at
https://protogen.marcgravell.com/decode), is the data in the payload? A
minimal example, in some way, would really help

On Tue, 5 Nov 2019, 20:55 SW89,  wrote:

> Good Afternoon,
>
> My team is working to implement Protobuf into our angular/C# web
> application however we have run into a snag regarding the implementation of
> wrapper types.
>
> We needed to support the notion of null values and came across the
> implementation of wrappers to facilitate this with protobuf. We have
> implemented all our non required fields using wrappers and what we have
> seen is that values encapsulated within the Wrappers are only returning a
> single character when inspected server side.
>
> For Example we have a form input in which we enter Name = "Test Name" and
> then when inspecting the server side values all Name is holding is "T".
>
> Has anyone encountered this issue and could shed some light on how to fix
> it.
>
> Thanks,
>
> Sam
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/45b6bfbe-e20b-427d-8bab-26197cfe4293%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzaA7e%3D5YXEvjUsWEXNRBYTD6sb5j4OTuEtdCY5%2Btg7PQ%40mail.gmail.com.

Re: [protobuf] How to correctly understand the term "wire format ordering" and "on the wire" in document "Language Guide (proto3)"?

[Marc Gravell]

It just means "when serialized" i.e. when written into binary or JSON for
storage / transmission; the "wire format ordering" one is emphasizing that
maps have no defined order whether in-memory or serialized, and you should
not rely on the order in which elements appear when serialized - on disk /
socket / whatever they can appear in *any order they want*, which has
nothing whatsoever to do with the order they were added. The "on the wire"
is used multiple times, but again, always means "when written to binary or
JSON".

On Tue, 5 Nov 2019 at 03:02, ken cao  wrote:

> I really really can't understand clearly these terms.
> thankyou very much.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/2ba6c259-94d7-491f-9cb7-36a840e7bda0%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyCvqJMjzaf6oyAjaJd_N-kqgr4dphhLL%3D89uhR4_MU%3DA%40mail.gmail.com.

Re: [protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

OK; that helps a bit; now, I've tried that with:

using Google.Protobuf;
using SCL.PC;
using System;
using System.Xml.Serialization;

static class P
{
static void Main()
{
var obj = new ClientIdentifier
{
Context = ClientIdentifier.Types.Context.Cfid,
Value = ByteString.CopyFrom(00, 01, 02, 03)
};
var ser = new XmlSerializer(obj.GetType());
ser.Serialize(Console.Out, obj);
}
}

---

I **cannot** reproduce the error you're seeing, but I *do* get another
error, as shown below.

This tells me, ultimately, that your model *isn't going to work with
XmlSerializer*. And that's fine - the code is only *tested* against the
specific API it is designed for - Google's protobuf library in this case.

If XmlSerializer on some down-level version of .NET has *even more*
problems, that's... unexpected, but not amazingly surprising.

If you want to use multiple serializers, usually you should have a
serialization DTO model *per serializer* - one that is designed for use
with that serializer.

However, there may be another option *if you want*. Google's generated code
that is contributing to the pain here. Looking at your generated code, it
looks like your schema is something like:

syntax = "proto3";
message ClientIdentifier {
  Context context = 1;
  bytes value = 2;
}
enum Context {
  Cfid = 0;
  Csn = 1;
}

Now, I can run that through protobuf-net's tooling (
https://protogen.marcgravell.com/#g50e914251fc5d16b94cdc695c8ebac20) which
gives me the simpler code (attached); if I then run that with the tweaked
code:

static class P
{
static void Main()
{
var obj = new ClientIdentifier
{
Context = Context.Cfid,
Value = new byte[] { 00, 01, 02, 03 }
};
var ser = new XmlSerializer(obj.GetType());
ser.Serialize(Console.Out, obj);
}
}

it now all works fine in my version of .NET, and I'm *guessing* that it'll
work in yours too. This is using a **different library**, specifically
protobuf-net rather than Google.ProtoBuf, but: the key point here is that
they both implement the same specification so should be interchangeable.

Up to you!

---

XmlSerializer exception.

Unhandled exception. System.InvalidOperationException: There was an error
reflecting type 'SCL.PC.ClientIdentifier'.
 ---> System.InvalidOperationException: To be XML serializable, types which
inherit from IEnumerable must have an implementation of Add(System.Object)
at all levels of their inheritance hierarchy. Google.Protobuf.ByteString
does not implement Add(System.Object).
   at System.Xml.Serialization.TypeScope.GetEnumeratorElementType(Type
type, TypeFlags& flags)
   at System.Xml.Serialization.TypeScope.ImportTypeDesc(Type type,
MemberInfo memberInfo, Boolean directReference)
   at System.Xml.Serialization.TypeScope.GetTypeDesc(Type type, MemberInfo
source, Boolean directReference, Boolean throwOnError)
   at System.Xml.Serialization.StructModel.GetPropertyModel(PropertyInfo
propertyInfo)
   at System.Xml.Serialization.StructModel.GetFieldModel(MemberInfo
memberInfo)
   at
System.Xml.Serialization.XmlReflectionImporter.InitializeStructMembers(StructMapping
mapping, StructModel model, Boolean openModel, String typeName,
RecursionLimiter limiter)
   at
System.Xml.Serialization.XmlReflectionImporter.ImportStructLikeMapping(StructModel
model, String ns, Boolean openModel, XmlAttributes a, RecursionLimiter
limiter)
   at
System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel
model, String ns, ImportContext context, String dataType, XmlAttributes a,
Boolean repeats, Boolean openModel, RecursionLimiter limiter)
   --- End of inner exception stack trace ---
   at
System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel
model, String ns, ImportContext context, String dataType, XmlAttributes a,
Boolean repeats, Boolean openModel, RecursionLimiter limiter)
   at
System.Xml.Serialization.XmlReflectionImporter.ImportElement(TypeModel
model, XmlRootAttribute root, String defaultNamespace, RecursionLimiter
limiter)
   at System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(Type
type, XmlRootAttribute root, String defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type, String
defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type)
   at P.Main() in C:\Users\marc\source\repos\ConsoleApp14\Program.cs:line 16


On Wed, 21 Aug 2019 at 10:28, Arun  wrote:

> I did the same but it throwing the error.. I dont know what I am making
> mistake here . All my application are 3.5 .net version,
>
> Please find  actual CS file which I am trying to serialize.
>
> On Wed, 21 Aug 2019 at 14:06, Marc Gravell  wrote:
>
>> K, I took the tiny bit of code you posted, and added something that uses
>> it with XmlSerializer, which basically meant implementing the interface
>> (with NIEs) and addin

Re: [protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

K, I took the tiny bit of code you posted, and added something that uses it
with XmlSerializer, which basically meant implementing the interface (with
NIEs) and adding:

using System;
using System.Xml.Serialization;
using pb = global::Google.Protobuf;
using pbr = global::Google.Protobuf.Reflection;

static class Program
{
static void Main()
{
var ser = new XmlSerializer(typeof(ClientIdentifier));
var obj = new ClientIdentifier();
ser.Serialize(Console.Out, obj);
}
}

---

And here's the thing: it works fine! So: if anyone is going to be able to
help you here, you're going to need to meet us half way, by *showing us the
code that doesn't work*.

On Wed, 21 Aug 2019 at 07:05, Arun  wrote:

>
> I am not able to initialize XmlSerializer class itself. It is failing
> saying "static Types cannot be serialized "
>
> On Wed, 21 Aug 2019 at 00:20, Marc Gravell  wrote:
>
>> Can you show any of your XmlSerializer code here, so can understand it? I
>> wouldn't expect the presence of a static type to break XmlSerializer, I
>> admit! But also: protobuf code doesn't usually guarantee anything other
>> than that it should work with the corresponding protobuf library.
>>
>> On Tue, 20 Aug 2019, 06:32 arun kumar,  wrote:
>>
>>> @Marc Gravell,
>>>
>>> I auto-generated ".cs" files from each ".proto" file. . Whereever a
>>> message declared inside another message in proto, auto-generated cs file is
>>> generated as " Nested Types " and Types class is generated as static. (I
>>> guess , its how the generator defined)
>>>
>>> And, I am trying to convert my xml data into CS object using
>>> XMLSerializer , it complains
>>>
>>> There was an error reflecting type 'SCL.PC.ClientIdentifier'. --->
>>>> System.InvalidOperationException: SCL.PC.ClientIdentifier+Types cannot be
>>>> serialized. Static types cannot be used as parameters or return types.
>>>
>>>
>>> All I am trying to do is ,
>>>
>>> Converting XML data --> CS Object (IMessage) --> proto buf Serialize
>>>> into stream
>>>
>>>
>>>
>>> Is there any way, to convert the xml data to IMessageObject.. I can see
>>> MessageParser has ParseJson object.  Likewise, anything available for xml ?
>>>
>>>
>>>var serializer = new XmlSerializer(typeof(SCL.PC.ClientIdentifier));
>>>
>>> Auto-Generated CS File :
>>>
>>> public sealed partial class ClientIdentifier :
>>> pb::IMessage {
>>> private static readonly pb::MessageParser _parser
>>> = new pb::MessageParser(() => new ClientIdentifier());
>>> private pb::UnknownFieldSet _unknownFields;
>>> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
>>> public static pb::MessageParser Parser { get {
>>> return _parser; } }
>>>
>>> ---
>>>
>>> #region Nested types
>>> /// Container for nested types declared in the
>>> ClientIdentifier message type.
>>> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
>>> public static partial class Types {
>>>   public enum Context {
>>>     /// 
>>> /// The client's identifier consists of a functional identifier
>>> (FID) in big-endian format.  Valid values range from 0x to
>>> 0x.
>>> /// 
>>> [pbr::OriginalName("CFID")] Cfid = 0,
>>> /// 
>>> /// The client's identifier consists of an ASCII-encoded serial
>>> number.
>>> /// 
>>> [pbr::OriginalName("CSN")] Csn = 1,
>>>   }
>>>
>>> }
>>> #endregion
>>>
>>>   }
>>>
>>>   #endregion
>>>
>>>
>>> On Monday, 19 August 2019 19:56:45 UTC+5:30, Marc Gravell wrote:
>>>>
>>>> And can we see some code that actually demonstrates this problem? That
>>>> would really help here.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com
>>> <https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>
> --
> -- Arun
>
>

-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwkzCH0uD8JT6McpSND1SEZtw98VObP9iNCta8-jo_P9g%40mail.gmail.com.

Re: [protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

Can you show any of your XmlSerializer code here, so can understand it? I
wouldn't expect the presence of a static type to break XmlSerializer, I
admit! But also: protobuf code doesn't usually guarantee anything other
than that it should work with the corresponding protobuf library.

On Tue, 20 Aug 2019, 06:32 arun kumar,  wrote:

> @Marc Gravell,
>
> I auto-generated ".cs" files from each ".proto" file. . Whereever a
> message declared inside another message in proto, auto-generated cs file is
> generated as " Nested Types " and Types class is generated as static. (I
> guess , its how the generator defined)
>
> And, I am trying to convert my xml data into CS object using XMLSerializer
> , it complains
>
> There was an error reflecting type 'SCL.PC.ClientIdentifier'. --->
>> System.InvalidOperationException: SCL.PC.ClientIdentifier+Types cannot be
>> serialized. Static types cannot be used as parameters or return types.
>
>
> All I am trying to do is ,
>
> Converting XML data --> CS Object (IMessage) --> proto buf Serialize into
>> stream
>
>
>
> Is there any way, to convert the xml data to IMessageObject.. I can see
> MessageParser has ParseJson object.  Likewise, anything available for xml ?
>
>
>var serializer = new XmlSerializer(typeof(SCL.PC.ClientIdentifier));
>
> Auto-Generated CS File :
>
> public sealed partial class ClientIdentifier :
> pb::IMessage {
> private static readonly pb::MessageParser _parser =
> new pb::MessageParser(() => new ClientIdentifier());
> private pb::UnknownFieldSet _unknownFields;
> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
> public static pb::MessageParser Parser { get {
> return _parser; } }
>
> ---
>
> #region Nested types
> /// Container for nested types declared in the
> ClientIdentifier message type.
> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
> public static partial class Types {
>   public enum Context {
> /// 
> /// The client's identifier consists of a functional identifier
> (FID) in big-endian format.  Valid values range from 0x to
> 0x.
> /// 
> [pbr::OriginalName("CFID")] Cfid = 0,
> /// 
> /// The client's identifier consists of an ASCII-encoded serial
> number.
> /// 
> [pbr::OriginalName("CSN")] Csn = 1,
>   }
>
> }
> #endregion
>
>   }
>
>   #endregion
>
>
> On Monday, 19 August 2019 19:56:45 UTC+5:30, Marc Gravell wrote:
>>
>> And can we see some code that actually demonstrates this problem? That
>> would really help here.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAycOVwTeWhrVWJ1e2KPBjVsHeNyrfqfsp6CUadxjhmqWQ%40mail.gmail.com.

[protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

And can we see some code that actually demonstrates this problem? That 
would really help here.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/576e5687-9319-4d3d-bd01-97fbdc79e763%40googlegroups.com.

Re: [protobuf] What are the encoding formats supported by protobuf?

[Marc Gravell]

Protobuf binary or protobuf-flavored JSON.

On Thu, 18 Jul 2019, 08:35 Pratibha Pruno Xavier, 
wrote:

> Hi all,
>
>  What are the encoding formats supported by protobuf 3?
>
> THanks,
> Pratibha
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/817d2ba0-872c-4f59-a284-1538cd054a5a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxWGwPWA1FbP%2B5nzc4mtvtz714T47Fva_m0oAMTDheZxw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf encoding with FIX prtocol

[Marc Gravell]

That definition is simply adding metadata to the field definitions; they
won't directly impact protobuf if you're using the default tools, but if
you're using custom tools, or a layer on top of protobuf that knows how to
inspect metadata, it can do additional things with the information. The
metadata isn't part of *payload* data; it exists independent of any
individual messages.

If you want more info, perhaps try clarifying the question so we understand
which bit it unclear.

On Thu, 11 Jul 2019 at 12:49, abdelrahman hamdy <
hamdyabdelrahman...@gmail.com> wrote:

> can anyone help me understanding this block of code and haw file have this
> value in the same line, example in FIX protocol tage = value so deliver
> companyID  tage=value is 128 = IBM for example.
> i need help please .
>  string deliver_to_comp_id = 3   [(fix.tag)=128, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>
> message StandardHeader {
>string begin_string = 1 [(fix.tag)=8, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>sfixed64 body_length = 2[(fix.tag)=9, (fix.type)=
> DATATYPE_LENGTH, (fix.field_added)=VERSION_FIX_4_0];
>string deliver_to_comp_id = 3   [(fix.tag)=128, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string deliver_to_sub_id = 4[(fix.tag)=129, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>sfixed64 msg_seq_num = 5[(fix.tag)=34, (fix.type)=
> DATATYPE_SEQ_NUM, (fix.field_added)=VERSION_FIX_4_0];
>MsgTypeEnum msg_type = 6[(fix.tag)=35, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string on_behalf_of_comp_id = 7 [(fix.tag)=115, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string on_behalf_of_sub_id = 8  [(fix.tag)=116, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>fix.Timestamp orig_sending_time = 9 [(fix.tag)=122, (fix.type)=
> DATATYPE_UTC_TIMESTAMP, (fix.field_added)=VERSION_FIX_4_0];
>bool poss_dup_flag = 10 [(fix.tag)=43, (fix.type)=
> DATATYPE_BOOLEAN, (fix.field_added)=VERSION_FIX_4_0];
>bool poss_resend = 11   [(fix.tag)=97, (fix.type)=
> DATATYPE_BOOLEAN, (fix.field_added)=VERSION_FIX_4_0];
>string secure_data = 12 [(fix.tag)=91, (fix.type)=
> DATATYPE_DATA, (fix.field_added)=VERSION_FIX_4_0];
>sfixed64 secure_data_len = 13   [(fix.tag)=90, (fix.type)=
> DATATYPE_LENGTH, (fix.field_added)=VERSION_FIX_4_0];
>string sender_comp_id = 14  [(fix.tag)=49, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string sender_sub_id = 15   [(fix.tag)=50, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>fix.Timestamp sending_time = 16 [(fix.tag)=52, (fix.type)=
> DATATYPE_UTC_TIMESTAMP, (fix.field_added)=VERSION_FIX_4_0];
>string target_comp_id = 17  [(fix.tag)=56, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string target_sub_id = 18   [(fix.tag)=57, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string deliver_to_location_id = 19  [(fix.tag)=145, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>string on_behalf_of_location_id = 20[(fix.tag)=144,
> (fix.type)=DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>string sender_location_id = 21  [(fix.tag)=142, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>string target_location_id = 22  [(fix.tag)=143, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>sfixed64 last_msg_seq_num_processed = 23[(fix.tag)=369,
> (fix.type)=DATATYPE_SEQ_NUM, (fix.field_added)=VERSION_FIX_4_2];
>string message_encoding = 24[(fix.tag)=347, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_2];
>string xml_data = 25[(fix.tag)=213, (fix.type)=
> DATATYPE_DATA, (fix.field_added)=VERSION_FIX_4_2];
>sfixed64 xml_data_len = 26  [(fix.tag)=212, (fix.type)=
> DATATYPE_LENGTH, (fix.field_added)=VERSION_FIX_4_2];
>ApplVerIDEnum appl_ver_id = 27  [(fix.tag)=1128, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_4];
>string cstm_appl_ver_id = 28[(fix.tag)=1129, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_4];
>repeated HopGrp hop_grp = 29
>  [(fix.field_added)=VERSION_FIX_4_4];
>sfixed64 appl_ext_id = 30   [(fix.tag)=1156, (fix.type)=
> DATATYPE_INT, (fix.field_added)=VERSION_FIX_5_0];
> }
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at 

Re: [protobuf] scalar types in service RPCs and partial messages

[Marc Gravell]

The "why" is because the marshaller assumes the root is a message; however,
you should look at "wrappers.proto" - there are well-known wrappers for
single values of most common types, including string. Also, prefer
"empty.proto" for empty, not your own.

On Sat, 29 Jun 2019 at 13:25, Ananya Bhat  wrote:

> Hi,
> I am new to protobuf.
> I have question related to specifying partial messages as RPC parameters
> and using scalar types in RPCs for returning simple strings
> Why message wrappers are required?
> Please clarify my doubts to decide better message specification
> I have definition something like below
>
> Option A
>
> message Student
> {
> message Empty
> {
> }
>
> //Name of the Student
> optional string name = 1;
> // Year of Passing
> optional string YOP  = 2;
> message Certificate
> {
> // Certificate name
> optional string name= 1;
> // Absolute location of the file on the disk
> // e.g. C:\university\2019\
> optional string path= 2;
> //strudent Identification code
> optional string ID  = 3;
> // section e.g. "A", "B"
> optional string section = 4;
> // Certificate type
> optional string type = 5;
> }
> optional Certificate cert = 3;
> }
> service StudentService
> {
> rpc getStudentName(Strudent.Empty) returns (Student);
> rpc getCertificate(Student.Certificate) returns (Student.Certificate);
> }
>
> alternatively
>
> Option B
>
> message Student
> {
>
> //Name of the Student
> optional string name = 1;
>
> message FilePath
> {
> optional string path = 1;
> }
> message Certificate
> {
> optional string   name = 1;
> optional Filepath path = 2;
> //strudent Identification code
> optional string ID  = 3;
> // section e.g. "A", "B"
> optional string section = 4;
> // Certificate type
> optional string type = 5;
>
> }
> optional Certificate cert = 2;
> }
> service StudentService
> {
> rpc getStudentName(Strudent.Empty) returns (Student);
> rpc getCertificate(Student.Certificate.FilePath) returns
> (Student.Certificate);
> }
>
> In Option A, file path can not be specified to get the certificate. Full
> message needs to be passed.
> In Option B, there is a extra type introduced to simplify
> However it adds nested definition for path
> as it has to be cert.path.path for getters and setters
>
> Please clarify the following
> Why scalar types can not be used as IN parameter or OUT parameters of
> Service RPCs?
> How partial messages can be explicitly specified in service RPCs without
> using nesting?
> What are the advantages and disadvantages of using message wrappers?
> Regards
> Ananya
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/544ad006-75ea-44fd-8e60-6d4db311f1ac%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxUPpnoYJXWQ%3DBH-6DWWeDmJ-%2BHEJUTS9bQb0J6_YNBmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C# - Can I compile Proto 2 with compiler Proto 3

[Marc Gravell]

Unless it has been updated recently, then no - the "protoc"/Google C#
implementation is (was?) proto3-only.

On Sun, 2 Jun 2019, 17:23 Derek Perez,  wrote:

> Yes it should be fine, just set syntax = "proto2" in your file, the
> compiler understands both I believe.
>
> On Sun, Jun 2, 2019 at 8:59 AM Marc Gravell 
> wrote:
>
>> If it helps, protobuf-net's schema parser can do this. It is a slightly
>> different API to the Google implementation, though.
>>
>> You can test it online at https://protogen.marcgravell.com/ - or the
>> command-line tool is available as a standalone utility via various
>> mechanisms (the standalone command-line tool is more versatile than the
>> online version, especially when processing multiple files).
>>
>> On Sun, 2 Jun 2019, 14:44 Marina Karmely, 
>> wrote:
>>
>>> HI
>>>
>>> Is there some way to compile ptoto 2 with a compiler of proto 3 in VS
>>> 2019 for C#?
>>> For example it can be done easily in Java(Eclipse and C++)
>>>
>>> But with the latest version of protoc 3.8.0 when I try to compile proto
>>> 2 with compiler of proto 3 I have got the following exception* "C# code
>>> generation only supports proto3 syntax"..*
>>>
>>> *Do you have any advise?*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/0bc3b2c5-f604-432e-9ad9-c679e40f1455%40googlegroups.com
>>> <https://groups.google.com/d/msgid/protobuf/0bc3b2c5-f604-432e-9ad9-c679e40f1455%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/protobuf/CAF95VAyE8hVG8p5qh98ETM91vaaigDbVihXwNX8R1NNb%3DZ3E_A%40mail.gmail.com
>> <https://groups.google.com/d/msgid/protobuf/CAF95VAyE8hVG8p5qh98ETM91vaaigDbVihXwNX8R1NNb%3DZ3E_A%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzUK9LHm6bikE%2BNwax-R%3DzbznP7cGC70_6mEG8bqWdkuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C# - Can I compile Proto 2 with compiler Proto 3

[Marc Gravell]

If it helps, protobuf-net's schema parser can do this. It is a slightly
different API to the Google implementation, though.

You can test it online at https://protogen.marcgravell.com/ - or the
command-line tool is available as a standalone utility via various
mechanisms (the standalone command-line tool is more versatile than the
online version, especially when processing multiple files).

On Sun, 2 Jun 2019, 14:44 Marina Karmely,  wrote:

> HI
>
> Is there some way to compile ptoto 2 with a compiler of proto 3 in VS 2019
> for C#?
> For example it can be done easily in Java(Eclipse and C++)
>
> But with the latest version of protoc 3.8.0 when I try to compile proto 2
> with compiler of proto 3 I have got the following exception* "C# code
> generation only supports proto3 syntax"..*
>
> *Do you have any advise?*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/0bc3b2c5-f604-432e-9ad9-c679e40f1455%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyE8hVG8p5qh98ETM91vaaigDbVihXwNX8R1NNb%3DZ3E_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Can protocol buffer support partial reading?

[Marc Gravell]

What exactly happens? Protocol buffers are designed to be forwards
compatible, so adding fields etc is usually fine and expected. Depending on
the implementation unexpected fields may be ignored or stored as extension
fields. So: is that what you did? And: what happened?

Note: changing the data type (or meaning) of an existing field is *not*
forwards compatible.

On Mon, 27 May 2019, 12:26 ill W,  wrote:

> I am using protocol buffer in C++ and I notice that if I use an old
> protocol buffer generated code and parse a new protocol buffer file, it
> will crash.
>
> Is there any way to partial parse the file like just ignore the
> unrecognized structure in my new file and get the supported total file?
>
> For example, If I support new layer in caffe model, can I parse the caffe
> model without compiling my code and get all the supported layer in my old
> code?
>
> I have read the protobuf api and find ParsePartialFromIstream function,
> but it still don't work.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/f65a1540-da98-4813-b012-afad4f665841%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxZjR8g0TLg0n4bqzohYTkPeXEAQnbBKGqY_YjmxipGrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] windows cmb protoc error

[Marc Gravell]

that just looks like `protoc` isn't in your path, and to be honest I
wouldn't really expect it to be; you'll need to either add it to your
system path list, or just use it from where-ever it installed

On Wed, 3 Apr 2019 at 16:20, abdelrahman hamdy <
hamdyabdelrahman...@gmail.com> wrote:

> I installed protobuf successfully but when I tried to install protocol I
> didn't get it always pops up an error that its invalid syntax, any help?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-net desrialization taking too long

[Marc Gravell]

>  Where can I send you mail? My e-mail is ...

I'll ping you

On Wed, 27 Mar 2019 at 15:12, Shweta Sharma  wrote:

> Thanks again Marc. Yes Deserialization code is as you listed below. I was
> using protobuf-net 2.4.0.0 but then moved to older version currently being
> used across our code base, 2.0.0.668, to see any impact. Where can I send
> you mail? My e-mail is shwetad...@gmail.com. I can get these objects
> serialized to a binary file and send over to you.
>
> public T Deserialize(byte[] serializedObject)
> {
> using (var ms = new MemoryStream(serializedObject))
> {
> return (T)Serializer.Deserialize(ms, null, typeof(T));
> }
> }
>
> Thanks much!
> Shweta
>
> On Wednesday, March 27, 2019 at 7:34:58 AM UTC-7, Marc Gravell wrote:
>>
>> > using StackExchange.Redis MGET
>>
>> Yeah, there's really no way for me to dodge this, is there? ;p
>>
>> Minor note: your parallel code currently doesn't actually allow any
>> meaningful parallelism - you *might* want to move the "lock" so that you
>> only "lock" around the "Add". You're also currently doing a "sync over
>> async" here - I'd probably use:
>>
>> var results = db.StringGet(rKeys);
>>
>> but that is unrelated to the real question here. The
>> _serializer.Deserialize(RedisValue) API you're using isn't something that
>> is directly exposed on protobuf-net, so I'd be grateful if I could see that
>> code. I'm *guessing* it is something like:
>>
>> using (var ms = new MemoryStream((byte[])value))
>> {
>> return Serializer.Deserialize(ms);
>> }
>>
>> ?
>>
>> If that is right, then ... it gets tricky; to give a really meaningful
>> analysis, I'd kinda have to see something akin to realistic data. Is there
>> anything I can do to setup an indicative minimal repro here? Or is there
>> anything you can share with me that I can work with to try and repro what
>> you're seeing? My preference here is a randomized data creation script like
>> my example, but if the data doesn't contain anything privileged, an RDB
>> file (from redis) might work, too.
>>
>> As I say: I can a range of in-progress changes that might help *anyway*,
>> but it would be great if I could actually test things to see how it
>> performs. If this allows me to confirm that the proposed changes make a
>> good improvement, I can always just ship those! But there may be other
>> things I can do here too. The key thing is again, to emphasize: me being
>> able to reproduce what you're seeing - because right now, I'm getting ~1ms
>> times.
>>
>> Marc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-net desrialization taking too long

[Marc Gravell]

> using StackExchange.Redis MGET

Yeah, there's really no way for me to dodge this, is there? ;p

Minor note: your parallel code currently doesn't actually allow any
meaningful parallelism - you *might* want to move the "lock" so that you
only "lock" around the "Add". You're also currently doing a "sync over
async" here - I'd probably use:

var results = db.StringGet(rKeys);

but that is unrelated to the real question here. The
_serializer.Deserialize(RedisValue) API you're using isn't something that
is directly exposed on protobuf-net, so I'd be grateful if I could see that
code. I'm *guessing* it is something like:

using (var ms = new MemoryStream((byte[])value))
{
return Serializer.Deserialize(ms);
}

?

If that is right, then ... it gets tricky; to give a really meaningful
analysis, I'd kinda have to see something akin to realistic data. Is there
anything I can do to setup an indicative minimal repro here? Or is there
anything you can share with me that I can work with to try and repro what
you're seeing? My preference here is a randomized data creation script like
my example, but if the data doesn't contain anything privileged, an RDB
file (from redis) might work, too.

As I say: I can a range of in-progress changes that might help *anyway*,
but it would be great if I could actually test things to see how it
performs. If this allows me to confirm that the proposed changes make a
good improvement, I can always just ship those! But there may be other
things I can do here too. The key thing is again, to emphasize: me being
able to reproduce what you're seeing - because right now, I'm getting ~1ms
times.

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-net desrialization taking too long

[Marc Gravell]

Hi; I'd love to help you on this - I'm the protobuf-net author, and I also
know more than a little about redis; it *might* be a little off-piste for
this group though. Running some tests locally with 5000 instances, I get
times like 1ms, but it might be that I'm misunderstanding your object
model. It would be great to perhaps see a realistic payload that I can play
with to see what's going on here.

Question: how fast is "fast enough" here, and how big are your payloads?

I have some incomplete changes in the pipe which at a stroke double the
throughput, but without knowing the answers to these questions, it is hard
to know whether that is sufficient

Depending on your target scenario, if the problem here turns out to be
allocations (of either the SubObj or the array), I have recently been doing
a lot of work in "arena allocators" which might be directly relevant - I
have a future plan to allow protobuf-net to consume / use the arena
allocator I've been working on, which would eliminate both of those
completely.

Example output from my test (the first serialize/deserialize is always more
expensive) - which is around 68k in payload size:

Serialize: 47ms
Serialize: 1ms
Serialize: 1ms
Serialize: 1ms
Serialize: 1ms
Deserialize: 6ms
Deserialize: 1ms
Deserialize: 1ms
Deserialize: 1ms
Deserialize: 1ms

My test code:
https://gist.github.com/mgravell/7afc08f432661f60138f6798efe2b15b

If you want to email me directly (especially if your data contains "real"
things that you don't want to send to the entire group), please feel free
to do so.

On Tue, 26 Mar 2019 at 18:32, Shweta Sharma  wrote:

> I am prototyping an application to store data in redis using
> StackExhange.Redis client. I am using Protobuf for serializing my objects
> and storing them as key/value pair in redis. When reading 20K keys using
> pipeline (splitting keys in batches of 5000), I observe deserialization
> times in 60-300 ms for a batch of say 5000 objects. Serialization code is
> using protobuf-net library version. Deserialization code is using
> protobuf-net 2.4.0.0.
>
> Is there anyway I can speed up desrialization time since this will impact
> my api performance.
>
> The object I am trying to desrialize is not too complex.
>
> [ProtoContract]
> public struct MyObject
> {
> [ProtoMember(1)]
> public int Key { get; set; }
> [ProtoMember(2)]
> public byte Attr1 { get; set; }
> [ProtoMember(3)]
> public bool IsUsed { get; set; }
> [ProtoMember(4)]
> public byte Attr2 { get; set; }
> [ProtoMember(5)]
> public SubObj[] SubObjects { get; set; }
>
> public static MyObject Null
> {
> get
> {
> return default(MyObject);
> }
> }
>
> public bool IsNull
> {
> get { return Attr1 == default(byte) && SubObjects == null; }
> }
> }
> [ProtoContract]
> public class SubObj
> {
> [ProtoMember(1)]
> public int StartOffset { get; set; }
> [ProtoMember(2)]
> public int EndOffset { get; set; }
> [ProtoMember(3)]
> public byte Attr1 { get; set; }
> [ProtoMember(4)]
> public byte Attr2 { get; set; }
> [ProtoMember(5)]
> public bool IsUsed { get; set; }
>
> public static SubObj Null
> {
> get
> {
> return default(SubObj);
> }
> }
>
> public bool IsNull
> {
> get { return Attr1 == default(byte); }
> }
> }
>
> Here're some sample timings using StopWatch:
>
> Time to desrialize 5000 keys from redis:134
> Time to desrialize 5000 keys from redis:147
> Time to desrialize 5000 keys from redis:160
> Time to desrialize 5000 keys from redis:65
> Time to desrialize 5000 keys from redis:67
> Time to desrialize 5000 keys from redis:242
> Time to desrialize 5000 keys from redis:371
> Time to desrialize 5000 keys from redis:190
>
> Thanks,
> Shweta
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] which protobuf type should be used for byte arrays

[Marc Gravell]

"bytes" is correct. The problem here is usually "treating binary data as
nul-terminated strings", which is almost certainly something *outside* of
the protobuf code, but in your code. Basically, you can't ever treat
protobuf data as nul-terminated strings.

On Wed, 23 Jan 2019, 00:52  What is the correct type to use when encoding byte arrays that contain 0s?
> If I use "bytes" then arrays that contain 0s are truncated.
>
> This email and its contents are confidential. If you are not the intended
> recipient, please do not disclose or use the information within this email
> or its attachments. If you have received this email in error, please report
> the error to the sender by return email and delete this communication from
> your records.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Decode google-protocol-buffer binary data using hex?

[Marc Gravell]

Presumably: convert the hex to bytes, and run it though the normal API for
your library/platform?

On Fri, 4 Jan 2019, 22:35  I have the java schema and the .proto schema for the data.
>
> The data is binary represented as hex. Is it possible to decode the data
> with the schemas I have? if so how?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Regarding serializing multiple .proto files

[Marc Gravell]

without knowledge of the specific library/framework/language that you're
using, I doubt anyone can answer this.

On Tue, 18 Dec 2018 at 05:11, Ishaan Aggarwal 
wrote:

> I have multiple .proto files in my project. I came to know that there can
> only be a single serializer and de-serializer file in the project (please
> confirm this understanding). Is there a way to serdes them?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Getting IOException while deserializing a proto which contains a repeated JKL and JKL has a string type set ( works for repeated JKL and JKL has the same with int type set )

[Marc Gravell]

there's nothing wrong with those message definitions; you haven't told us
which platform and/or library you're using, but: my hunch would be that
you're transporting the data incorrectly in some way, meaning: corrupting
it in *your* code - or the code you're using to compress/decompress is
slightly wrong. Unfortunately we can't see any of that. But: speaking
generally: there's no reason it shouldn't work, so: we'd need more details
to provide useful help.

On Thu, 13 Dec 2018 at 18:52, Shibha Malik  wrote:

>
>
> On Thursday, December 13, 2018 at 10:32:52 AM UTC-8, Shibha Malik wrote:
>>
>> Hi Team,
>>
>> i am getting IOException while deserializing a proto which contains a
>> repeated Data and Data has a string type set ( works for repeated Data and
>> Data has the same with int type set )
>>
>> I have a proto which has the following structure :
>>
>> message XYZData
>> {
>>   ABCId abc_id = 1;
>>   DEF def= 2;
>>   google.protobuf.Timestamp ghi= 3;
>>   repeated JKL jkl = 4;
>>   MNO mno = 5;
>> }
>>
>> where JKL is
>> message JKL
>> {
>>   Culprit culprit = 1;
>>   PQR pqr = 2;
>> }
>> where Culprit is
>> message Culprit
>> {
>>   oneof culprit
>>   {
>> int32 int_culprit = 1;
>> string string_culprit = 2;
>>   }
>> }
>>
>> When i use the int_culprit and use GZIP compression. it serializes and
>> deserializes fine.
>> BUT when i use the string_culprit and use GZIP compression. it serializes
>> fine and deserialization gives me IOException.
>>
>>
>> PLEASE help us on this as this is blocking a major deliverable for us.
>>
>> Thanks,
>> Shibha
>>
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] define Set> message ,what should I do?

[Marc Gravell]

"set" isn't a protobuf concept - the closest that exists is "repeated";
likewise "object" - ideally using some defined message type. Putting those
together, that gives you:

syntax = "proto3";

message SomeRoot {
  repeated SomeLeaf items = 1;
}
message SomeLeaf {
  map values = 1;
}
message SomeVal {
  // ...
}

On Fri, 23 Nov 2018 at 08:31,  wrote:

> I want define a message. Example: Set> please help me.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Defining wrapper types in proto file

[Marc Gravell]

Are you talking about Java's Long here? If so, frankly I'd say: in your DTO
layer, use whatever type protoc wants to use for your data, and just use:

syntax = "proto3";
message TestMessage{
string attribute1 = 1;
map attribute2 = 2;
int64 attribute3 = 3;
int32 attribute4 = 4;
int64 attribute5 = 5;
int32 attribute6 = 6;
repeated int64 attribute7 = 7;
}

However; there *is* also wrappers.proto (
https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto)
which allows something similar, but that
uses com.google.protobuf.Int64Value, not Java.Lang.Long; but if you want to
try it:

syntax = "proto3";
import "google/protobuf/wrappers.proto";
message TestMessage{
string attribute1 = 1;
map attribute2 = 2;
int64 attribute3 = 3;
int32 attribute4 = 4;
.google.protobuf.Int64Value attribute5 = 5;
.google.protobuf.Int32Value attribute6 = 6;
repeated .google.protobuf.Int64Value attribute7 = 7;
}



On Wed, 12 Sep 2018 at 09:44, qplc  wrote:

> Hi Marc,
>
> Thanks for you reply. Here, I'm trying to create a user defined message
> given follow.
>
> Class TestMessage{
>   String attribute1;
>   Map attribute2;
>   long attribute3;
>   int attribute4;
>   Long attribute5;
>   Integer attribute6;
>   Set attribute7;
> }
>
> I want to define above message format in .proto file. Hence, I've created
> following message.
> message TestMessage{
>   string attribute1 = 1;
>   map attribute2 = 2;
>   int64 attribute3 = 3;
> }
>
> But, I'm not sure how do I define attribute5, attribute6 and attribute7 in
> proto file. Please help me on this.
>
>
> On Wednesday, September 12, 2018 at 1:30:14 PM UTC+5:30, qplc wrote:
>>
>> Hi,
>>
>> How do I define wrapper Long value in .proto file. Can anyone help in
>> this?
>>
>>
>> Thanks,
>> qplc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Defining wrapper types in proto file

[Marc Gravell]

That's pretty vague. Could you clarify what you're trying to do?

On Wed, 12 Sep 2018, 09:00 qplc,  wrote:

> Hi,
>
> How do I define wrapper Long value in .proto file. Can anyone help in this?
>
>
> Thanks,
> qplc
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf version 3.6 a compile libprotobuf?

[Marc Gravell]

You mention both 3.5 and 3.6 - which did you mean?

As for both compiling and links: it kinda depends what platform and/or
language you are targeting. So: what are you targeting? You'll usually have
to run the protoc output through your own build tools.

On Thu, 6 Sep 2018, 07:47 Natalia Duality,  wrote:

> Hello,
>
> is it possible to create static link  with protobuf version 3.5???
>
> And second question,
>
>
> I found protoc.exe to generate files, but hot to compile dll?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Inheriting protobuf messages

[Marc Gravell]

I'm going to share some thoughts here simply for discussion purposes - I
don't expect them to be directly applicable.

FWIW, protobuf-net has spoofed inheritance for many many years. I'm able to
do this because protobuf-net only needs to target .NET, which has good
inheritance support.
I don't really expect the main "protobuf" project will add inheritance in
any timescale, because it might need to target platforms that don't have
inheritance support.
It would also require ".proto" language considerations.

That said... if anyone at Google does ever want to reignite the discussion
around inheritance, I'm all ears :)

---

The way it protobuf-net does this can be *essentially* represented by
"oneof" - so going back to the original example from (wow, a long time ago):

message MSG1
 {
   required string account = 1;
   required string symbol = 2;
 }
message MSG2 extends MSG1
 {
   required int32  id  = 2;
 }

the way protobuf-net does this is essentially:

 message MSG1
 {
   required string account = 1;
   required string symbol = 2;
   oneof _subtype { // keep in mind that this is actually "zero-or-one-of"
  MSG2 msg2 = 3; // this field-number needs to be unique in MSG1
}
 }
 message MSG2
 {
   required int32  id  = 2; // no restrictions on this field-number
 }

but protobuf-net does some tricks so that when deserializing a MSG1, it
might end up *actually* creating an instance of MSG2 instead (where MSG2 :
MSG1).
For LSP purposes, everything is *always* serialized from the outside in, so
if you had a list/array of MSG2, they would still write the MSG1 fields
first - essentially
it would be like a list/array of MSG1, but where everything *happens to be*
an MSG2. You're right to say that the base type needs to know about the
derived types,
or at least have lib support to help it there, with some kind of deferred
registration.

But: I wonder whether the same approach - just without the actual
inheritance in the generated types - might still be useful to you.

Concretely, from a code-first angle:
https://gist.github.com/mgravell/a33755800e823ee77ba01183be7084df

Or maybe it is completely unhelpful and you're worse off for reading
this... who knows!

Marc




On Mon, 3 Sep 2018 at 11:34, SangemV  wrote:

> I see two problems with this approach. It is neither cut & paste nor
> personal preference issue as I see. The real issues I see are:
>
> 1) The base message (MSG in the example)  and the extended message (MSG2
> in the example ) can belong to different package owned by different
> group/org. The package defining base message (MSG) (call it Pkg1) does not
> event know the existing of the package defining extended message (MSG2)
> (call it Pkg2). How can the base message foresee all the extended message
> from it? Even if Pkg1 and Pkg2 are owned by the same group/org, making Pkg1
> aware of Pkg2 is not a good idea as it can potentially create cyclic
> dependencies.
>
> 2) Polymorphic Lists: I have a list of MSG types which can potentially
> have both MSG and MSG2 types. How is this modeled in proto3 using the
> proposed solution?
>
> These are real issues which I am facing right now. Any suggestions to
> handle the above problems would be of great help.
>
>> ​
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Change Timestamp Format

[Marc Gravell]

It is a little unclear what you mean, since the javascript Date constructor
itself takes the year first, but that is nothing to do with protobuf or
JSON, and nothing to do with the format.


If you mean the JSON, then:

In the JSON format, timestamp is always RFC 3339. In the binary format,
timestamp is just epoch integers (or two).

If you need different control over the JSON format, you might want a
general purpose JSON library, not s strongly opinionated tool like
protobuf. Alternatively: store a string.


On Sun, 19 Aug 2018, 03:01 Eduardo Demar,  wrote:

> Hi, I just need change default format of timestamp to dd/mm/
>
> I am using this code, on https://script.google.com:
>
>   input = {
> "timestamp": new Date(DD/MM/)
>   };
>
> I really appreciate this help!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Is protobuffers right for me?

[Marc Gravell]

>  I didn't see anything about lists though

See: "repeated"; this is pretty much a synonym for "list".

>  Some items may reference other incoming packets.

No, there is no concept of an object id, whether for in-message or
out-of-message referencing.

On Sun, 29 Jul 2018 at 22:38, M P  wrote:

> I will be sending* hundreds of results of data in the form of a list as
> part of a protocol* I'm designing. Protobuffers looks ideal to help speed
> development across multiple services. I didn't see anything about lists
> though. *Is it possible to do? *
>
> Some items may reference other incoming packets. *Is this also possible?*
> I know I can just assign IDs and do lookups the traditional way but I want
> to know if protobuffers has a built-in solution for this as well.
>
> Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

The first step, then, is to find how to get your ipv4 in a 4 byte
std::string rather than a 15 byte std::string - I can't advise on that, but
it should absolutely be possible. You don't need it as ASCII - you just
want the bytes.

As for will it be longer? No, especially when compared to uint32; a 4-byte
"bytes" will take 5 bytes to encode (1 byte for the length prefix, 4 bytes
payload); the uint32 with value composed of 4 bytes with value 111 will be
varint-encoded, so will be 5 bytes (since the penultimate high bit is set);
in fact, anything with the most-significant byte having value 16 or above
will require 5 bytes to encode as uint32. Now, you *could* encode an ipv4
using "fixed32", which will always take 4 bytes, but... if you do that, you
can't conveniently store ipv6 in the same field. So: since you mention
needing to store both ipv4 and ipv6, "repeated bytes" is your simplest
option. And as above: it isn't any more expensive than "uint32" (for most
IPs).

On Wed, 18 Jul 2018 at 10:58, sanjana gupta 
wrote:

> I would like to understand from the protobuf generated API point of view:
>
> If I have,
> repeated bytes localEndpoint = 1;
>
> The generated file profile.pb.h shows me the following APIs to add+set
> values in it :
>
>   // repeated bytes localEndpoint = 1;
> *  inline void add_localendpoint(const ::std::string& value);*
> *  inline void add_localendpoint(const char* value);*
> *  inline void add_localendpoint(const void* value, size_t size);*
>
> If I use any of them, perhaps my boost::asio::ip::address_v4
> (111.111.111.111) needs to be converted to a std::string and then only I
> can use add_localendpoint() to set a value.
> Am I correct?
>
> If the conversion to string is done, isn't the size of protobuf message
> going to be more compared to the case of using
> repeated uint32 localEndpoint = 1;
> ?
>
> Please let me know if I have confused you :) Thanks again!
>
> On Wednesday, July 18, 2018 at 3:08:52 PM UTC+5:30, Marc Gravell wrote:
>>
>> I would be amazed if there isn't a way of getting the raw underlying
>> bytes from the IP address, rather than the text (ASCII). The address
>> 111.111.111.111 is literally the 4 bytes with decimal values 111, 111, 111
>> and 111. No ASCII required.
>>
>> On Wed, 18 Jul 2018, 10:28 sanjana gupta,  wrote:
>>
>>> Hey Marc,
>>>
>>> I am sorry if I am repeating my words. Please enlighten me on this thing
>>> :
>>>
>>> "bytes" requires me to give a std::string (c++) value as input. The
>>> problem with string for me is that a usual 4-byte ipv4 address like
>>> 111.111.111.111 becomes 15-bytes string "111.111.111.111"
>>>
>>> Having said that, I feel discouraged to use bytes over uint32 (or int
>>> type for that matter) as it already increases the *size of the data to
>>> be encoded*. As a result, we can say that :
>>> size of (encoded message using "bytes" type)  >  size of (encoded
>>> message using "int" type)
>>>
>>> Is my understanding correct? Thanks!
>>>
>>>
>>> On Wednesday, July 18, 2018 at 2:40:01 PM UTC+5:30, Marc Gravell wrote:
>>>>
>>>> At that point I'd probably use "repeated bytes", then. It'll cost you
>>>> an extra byte on v4 addresses, but it is simple.
>>>>
>>>> On Wed, 18 Jul 2018, 09:36 sanjana gupta, 
>>>> wrote:
>>>>
>>>>> Thanks Marc for your reply! By mistake I wrote 8-bytes for ipv4. I
>>>>> have corrected it as 4-bytes in my question. Thanks for that :)
>>>>>
>>>>> One thing I want to ask you is that can I use "oneof" if my field
>>>>> "localEndpoint" is repeated?
>>>>> I mean to say that my one "profile" message can have any among :
>>>>> -> 2 ipv4 addresses
>>>>> -> OR 2 ipv6 addresses
>>>>> -> OR 1 ipv4 + 1 ipv6 addresses
>>>>> in the field "localEndpoint".
>>>>>
>>>>> So in that case, is "oneof" usage correct? I think that a "oneof"
>>>>> cannot be repeated. Please correct me if I am wrong.
>>>>>
>>>>> On Wednesday, July 18, 2018 at 12:54:39 PM UTC+5:30, Marc Gravell
>>>>> wrote:
>>>>>>
>>>>>> You should be able to encode ipv4 in 4 bytes, making fixed32 ideal,
>>>>>> since you can avoid the length prefix. For ipv6, you're going to need 16
>>>>>> bytes, so "bytes" is

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

I would be amazed if there isn't a way of getting the raw underlying bytes
from the IP address, rather than the text (ASCII). The address
111.111.111.111 is literally the 4 bytes with decimal values 111, 111, 111
and 111. No ASCII required.

On Wed, 18 Jul 2018, 10:28 sanjana gupta, 
wrote:

> Hey Marc,
>
> I am sorry if I am repeating my words. Please enlighten me on this thing :
>
> "bytes" requires me to give a std::string (c++) value as input. The
> problem with string for me is that a usual 4-byte ipv4 address like
> 111.111.111.111 becomes 15-bytes string "111.111.111.111"
>
> Having said that, I feel discouraged to use bytes over uint32 (or int type
> for that matter) as it already increases the *size of the data to be
> encoded*. As a result, we can say that :
> size of (encoded message using "bytes" type)  >  size of (encoded message
> using "int" type)
>
> Is my understanding correct? Thanks!
>
>
> On Wednesday, July 18, 2018 at 2:40:01 PM UTC+5:30, Marc Gravell wrote:
>>
>> At that point I'd probably use "repeated bytes", then. It'll cost you an
>> extra byte on v4 addresses, but it is simple.
>>
>> On Wed, 18 Jul 2018, 09:36 sanjana gupta,  wrote:
>>
>>> Thanks Marc for your reply! By mistake I wrote 8-bytes for ipv4. I have
>>> corrected it as 4-bytes in my question. Thanks for that :)
>>>
>>> One thing I want to ask you is that can I use "oneof" if my field
>>> "localEndpoint" is repeated?
>>> I mean to say that my one "profile" message can have any among :
>>> -> 2 ipv4 addresses
>>> -> OR 2 ipv6 addresses
>>> -> OR 1 ipv4 + 1 ipv6 addresses
>>> in the field "localEndpoint".
>>>
>>> So in that case, is "oneof" usage correct? I think that a "oneof" cannot
>>> be repeated. Please correct me if I am wrong.
>>>
>>> On Wednesday, July 18, 2018 at 12:54:39 PM UTC+5:30, Marc Gravell wrote:
>>>>
>>>> You should be able to encode ipv4 in 4 bytes, making fixed32 ideal,
>>>> since you can avoid the length prefix. For ipv6, you're going to need 16
>>>> bytes, so "bytes" is probably your best bet, since it will only require a
>>>> single header. You can then create a union of those:
>>>>
>>>> oneof ip_addr {
>>>> fixed32 v4 = 1;
>>>> bytes v6 = 2;
>>>> }
>>>>
>>>> That seems pretty optimal to me.
>>>>
>>>> Marc
>>>>
>>>> On Wed, 18 Jul 2018, 08:16 sanjana gupta, 
>>>> wrote:
>>>>
>>>>> I read that protobuf has a type called "*bytes*" which can store
>>>>> arbitrary number of bytes and is the equivalent of "C++ string".
>>>>>
>>>>> The reason why I don't prefer to use "bytes" is that it expects input
>>>>> as a C++ string i.e., boost IP will need to be converted to a string.
>>>>>
>>>>>
>>>>> Now my concern lies here : I want to perform serialize and send the
>>>>> encoded protobuf message over TCP socket. I want to ensure that the 
>>>>> *encoded
>>>>> message size is as small as possible*.
>>>>>
>>>>>
>>>>> Currently, I am using the below .proto file :
>>>>>
>>>>> syntax = "proto2";
>>>>>
>>>>> message profile
>>>>>
>>>>> {
>>>>>
>>>>> repeated *uint32* localEndpoint = 1;
>>>>>
>>>>> repeated *uint32* remoteEndpoint = 2;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> In order to save boost IP in the protobuf message, I am first
>>>>> converting boost IP into byte-format array by using
>>>>> "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array
>>>>> size is 8. Then I am converting 1st 4 bytes from the resultant byte-array
>>>>> into one uint32_t number and then storing in "localEndpoint" field of the
>>>>> protobuf message. Likewise, I repeat for the next 4 bytes. I am taking 4
>>>>> bytes at a time so as to utilize full 32 bits of the uint32.
>>>>>
>>>>>
>>>>> Hence for a v4 address, 2 occurrence of "localEndpoint" field is used.
>>>>> Similarly, for a v6 ad

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

At that point I'd probably use "repeated bytes", then. It'll cost you an
extra byte on v4 addresses, but it is simple.

On Wed, 18 Jul 2018, 09:36 sanjana gupta, 
wrote:

> Thanks Marc for your reply! By mistake I wrote 8-bytes for ipv4. I have
> corrected it as 4-bytes in my question. Thanks for that :)
>
> One thing I want to ask you is that can I use "oneof" if my field
> "localEndpoint" is repeated?
> I mean to say that my one "profile" message can have any among :
> -> 2 ipv4 addresses
> -> OR 2 ipv6 addresses
> -> OR 1 ipv4 + 1 ipv6 addresses
> in the field "localEndpoint".
>
> So in that case, is "oneof" usage correct? I think that a "oneof" cannot
> be repeated. Please correct me if I am wrong.
>
> On Wednesday, July 18, 2018 at 12:54:39 PM UTC+5:30, Marc Gravell wrote:
>>
>> You should be able to encode ipv4 in 4 bytes, making fixed32 ideal, since
>> you can avoid the length prefix. For ipv6, you're going to need 16 bytes,
>> so "bytes" is probably your best bet, since it will only require a single
>> header. You can then create a union of those:
>>
>> oneof ip_addr {
>> fixed32 v4 = 1;
>> bytes v6 = 2;
>> }
>>
>> That seems pretty optimal to me.
>>
>> Marc
>>
>> On Wed, 18 Jul 2018, 08:16 sanjana gupta,  wrote:
>>
>>> I read that protobuf has a type called "*bytes*" which can store
>>> arbitrary number of bytes and is the equivalent of "C++ string".
>>>
>>> The reason why I don't prefer to use "bytes" is that it expects input as
>>> a C++ string i.e., boost IP will need to be converted to a string.
>>>
>>>
>>> Now my concern lies here : I want to perform serialize and send the
>>> encoded protobuf message over TCP socket. I want to ensure that the *encoded
>>> message size is as small as possible*.
>>>
>>>
>>> Currently, I am using the below .proto file :
>>>
>>> syntax = "proto2";
>>>
>>> message profile
>>>
>>> {
>>>
>>> repeated *uint32* localEndpoint = 1;
>>>
>>> repeated *uint32* remoteEndpoint = 2;
>>>
>>> }
>>>
>>>
>>> In order to save boost IP in the protobuf message, I am first converting
>>> boost IP into byte-format array by using
>>> "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array
>>> size is 8. Then I am converting 1st 4 bytes from the resultant byte-array
>>> into one uint32_t number and then storing in "localEndpoint" field of the
>>> protobuf message. Likewise, I repeat for the next 4 bytes. I am taking 4
>>> bytes at a time so as to utilize full 32 bits of the uint32.
>>>
>>>
>>> Hence for a v4 address, 2 occurrence of "localEndpoint" field is used.
>>> Similarly, for a v6 address, 4 occurrence of "localEndpoint" field is used.
>>>
>>>
>>> Please allow me to highlight that if I had used "bytes" here, my input
>>> string itself would have been of size 15 bytes for a v4 ip like
>>> 111.111.111.111
>>>
>>>
>>> Using uint32 instead of "bytes" does save me some encoded-data-size but
>>> I am looking for a more efficient protobuf type requiring lesser number of
>>> bytes.
>>>
>>>
>>> Sorry for a long description but I wanted to explain my query in
>>> details. Please help me.. Thanks a lot in advance :)
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

You should be able to encode ipv4 in 4 bytes, making fixed32 ideal, since
you can avoid the length prefix. For ipv6, you're going to need 16 bytes,
so "bytes" is probably your best bet, since it will only require a single
header. You can then create a union of those:

oneof ip_addr {
fixed32 v4 = 1;
bytes v6 = 2;
}

That seems pretty optimal to me.

Marc

On Wed, 18 Jul 2018, 08:16 sanjana gupta, 
wrote:

> I read that protobuf has a type called "*bytes*" which can store
> arbitrary number of bytes and is the equivalent of "C++ string".
>
> The reason why I don't prefer to use "bytes" is that it expects input as a
> C++ string i.e., boost IP will need to be converted to a string.
>
>
> Now my concern lies here : I want to perform serialize and send the
> encoded protobuf message over TCP socket. I want to ensure that the *encoded
> message size is as small as possible*.
>
>
> Currently, I am using the below .proto file :
>
> syntax = "proto2";
>
> message profile
>
> {
>
> repeated *uint32* localEndpoint = 1;
>
> repeated *uint32* remoteEndpoint = 2;
>
> }
>
>
> In order to save boost IP in the protobuf message, I am first converting
> boost IP into byte-format array by using
> "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array
> size is 8. Then I am converting 1st 4 bytes from the resultant byte-array
> into one uint32_t number and then storing in "localEndpoint" field of the
> protobuf message. Likewise, I repeat for the next 4 bytes. I am taking 4
> bytes at a time so as to utilize full 32 bits of the uint32.
>
>
> Hence for a v4 address, 2 occurrence of "localEndpoint" field is used.
> Similarly, for a v6 address, 4 occurrence of "localEndpoint" field is used.
>
>
> Please allow me to highlight that if I had used "bytes" here, my input
> string itself would have been of size 15 bytes for a v4 ip like
> 111.111.111.111
>
>
> Using uint32 instead of "bytes" does save me some encoded-data-size but I
> am looking for a more efficient protobuf type requiring lesser number of
> bytes.
>
>
> Sorry for a long description but I wanted to explain my query in details.
> Please help me.. Thanks a lot in advance :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Decoding message protobuff

[Marc Gravell]

Btw you might find https://protogen.marcgravell.com/decode useful - it
explains how each byte is interpreted. It doesn't give the details (so: it
doesn't explain how varint works), but it at least makes it explicit which
bytes have contributed to which outcomes.

On Tue, 17 Jul 2018, 08:22 Marc Gravell,  wrote:

> AA 06 is indeed 101, that's "varint" encoding. The MSB of each byte is
> continuation, 7 bits payload, and adjust for endianness. The idea of
> decode-raw is that it is enough to start guessing at how to reverse
> engineer a message, for example you can see values that are clearly string
> IP-addresses. However, the other fields are much harder to interpret
> without context. If you do have the original .proto schema, it would help
> tremendously.
>
> On Tue, 17 Jul 2018, 05:17 ,  wrote:
>
>> How to decode protobuff embedded message. Please give any pseudo code. I
>> see aa 06 gets decoded as 101. Please let me know how?
>>
>> Protobuff data:
>>
>> echo -n "0a 0b 31 39 32 2e 31 30 2e 31 31 2e 31 10 01 aa 06 88 01 e2 a4
>> 01 83 01 1a 80 01 0a 1e 0a 06 70 6f 72 74 5f 32 10 80 b7 e3 88 08 18 02 32
>> 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70 6f 72 74 5f 33 10 80
>> b7 e3 88 08 18 03 32 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70
>> 6f 72 74 5f 34 10 80 b7 e3 88 08 18 04 32 0c 08 00 10 00 28 00 30 00 38 00
>> 40 00 0a 1e 0a 06 70 6f 72 74 5f 35 10 80 b7 e3 88 08 18 05 32 0c 08 00 10
>> 00 28 00 30 00 38 00 40 00" | xxd -r -p | protoc --decode_raw
>> 1: "192.10.11.1"
>> 2: 1
>> 101 {
>>   2636 {
>> 3 {
>>   1 {
>> 1: "port_2"
>> 2: 2165889920
>> 3: 2
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>>   1 {
>> 1: "port_3"
>> 2: 2165889920
>> 3: 3
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>>   1 {
>> 1: "port_4"
>> 2: 2165889920
>> 3: 4
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>>   1 {
>> 1: "port_5"
>> 2: 2165889920
>> 3: 5
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>> }
>>   }
>> }
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Decoding message protobuff

[Marc Gravell]

AA 06 is indeed 101, that's "varint" encoding. The MSB of each byte is
continuation, 7 bits payload, and adjust for endianness. The idea of
decode-raw is that it is enough to start guessing at how to reverse
engineer a message, for example you can see values that are clearly string
IP-addresses. However, the other fields are much harder to interpret
without context. If you do have the original .proto schema, it would help
tremendously.

On Tue, 17 Jul 2018, 05:17 ,  wrote:

> How to decode protobuff embedded message. Please give any pseudo code. I
> see aa 06 gets decoded as 101. Please let me know how?
>
> Protobuff data:
>
> echo -n "0a 0b 31 39 32 2e 31 30 2e 31 31 2e 31 10 01 aa 06 88 01 e2 a4
> 01 83 01 1a 80 01 0a 1e 0a 06 70 6f 72 74 5f 32 10 80 b7 e3 88 08 18 02 32
> 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70 6f 72 74 5f 33 10 80
> b7 e3 88 08 18 03 32 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70
> 6f 72 74 5f 34 10 80 b7 e3 88 08 18 04 32 0c 08 00 10 00 28 00 30 00 38 00
> 40 00 0a 1e 0a 06 70 6f 72 74 5f 35 10 80 b7 e3 88 08 18 05 32 0c 08 00 10
> 00 28 00 30 00 38 00 40 00" | xxd -r -p | protoc --decode_raw
> 1: "192.10.11.1"
> 2: 1
> 101 {
>   2636 {
> 3 {
>   1 {
> 1: "port_2"
> 2: 2165889920
> 3: 2
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
>   1 {
> 1: "port_3"
> 2: 2165889920
> 3: 3
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
>   1 {
> 1: "port_4"
> 2: 2165889920
> 3: 4
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
>   1 {
> 1: "port_5"
> 2: 2165889920
> 3: 5
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
> }
>   }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is wrong with my proto file here? Is it not possible to have 2 repeated fields of different types?

[Marc Gravell]

also, field zero (type) is invalid; you are welcome to use this as an
online check tool - but protoc also outputs similar messages:

https://protogen.marcgravell.com#gc4759103e204eec4ae1a11ce6089a4bf

On Thu, 12 Jul 2018 at 23:01, Marc Gravell  wrote:

> datavalues_srarray  and  datavalues_prarray   need to have different
> field numbers
>
> On Thu, 12 Jul 2018 at 22:50, Himabindu Dittakavi 
> wrote:
>
>> Hi All,
>>
>> New to protobuf.
>> Here is my proto file. If i use the code highlighted in yellow it works
>> but when I add the one in yellow - there is no .cs file generated...what is
>> wrong with the code below? TIA. Also, is there a proto file validator by
>> any chance?
>>
>> syntax = "proto3";
>> package tutorial;
>> option csharp_namespace = "Google.Protobuf.Examples.SS";
>>
>> message SS {
>>   int32 id = 1;  // Unique ID number for this message or ss.
>>   enum GeneratedDataType {
>> SR = 0;
>> PR = 1;
>> SE = 2;
>>   }
>>message DataValueSr {
>> GeneratedDataType type = 0;
>> string value = 100;
>>   }
>>message DataValuePr {
>> GeneratedDataType type = 1;
>> string value = 100;
>>   }
>>
>> repeated DataValueSr datavalues_srarray = 1000;
>> repeated DataValuePr datavalues_prarray = 1000;
>>  }
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Regards,
>
> Marc
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is wrong with my proto file here? Is it not possible to have 2 repeated fields of different types?

[Marc Gravell]

 datavalues_srarray  and  datavalues_prarray   need to have different field
numbers

On Thu, 12 Jul 2018 at 22:50, Himabindu Dittakavi 
wrote:

> Hi All,
>
> New to protobuf.
> Here is my proto file. If i use the code highlighted in yellow it works
> but when I add the one in yellow - there is no .cs file generated...what is
> wrong with the code below? TIA. Also, is there a proto file validator by
> any chance?
>
> syntax = "proto3";
> package tutorial;
> option csharp_namespace = "Google.Protobuf.Examples.SS";
>
> message SS {
>   int32 id = 1;  // Unique ID number for this message or ss.
>   enum GeneratedDataType {
> SR = 0;
> PR = 1;
> SE = 2;
>   }
>message DataValueSr {
> GeneratedDataType type = 0;
> string value = 100;
>   }
>message DataValuePr {
> GeneratedDataType type = 1;
> string value = 100;
>   }
>
> repeated DataValueSr datavalues_srarray = 1000;
> repeated DataValuePr datavalues_prarray = 1000;
>  }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the relation between length of data block and file length in varint?

[Marc Gravell]

I added a much longer version of this here:
https://stackoverflow.com/questions/50500626/what-is-the-relation-between-length-of-data-block-and-file-length-in-varint/50503626#50503626

On Thu, 24 May 2018 at 08:33, Marc Gravell <marc.grav...@gmail.com> wrote:

> I wonder if you're decoding it incorrectly. Maybe if you could post the
> bytes that you're decoding to reach this contradictory result, we can
> advise you.
>
> On Thu, 24 May 2018 at 04:59, Ruman Ahmed Rizvi <
> ruman.ahmed.rizvi@gmail.com> wrote:
>
>> When working with protocol buffers and encoding and decoding of these
>> there is one issue occurred. In proto stream when decoding the file size
>> and data block length in varint, the data block length is bigger than the
>> file size? I am getting data block length greater than file length.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Regards,
>
> Marc
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the relation between length of data block and file length in varint?

[Marc Gravell]

I wonder if you're decoding it incorrectly. Maybe if you could post the
bytes that you're decoding to reach this contradictory result, we can
advise you.

On Thu, 24 May 2018 at 04:59, Ruman Ahmed Rizvi <
ruman.ahmed.rizvi@gmail.com> wrote:

> When working with protocol buffers and encoding and decoding of these
> there is one issue occurred. In proto stream when decoding the file size
> and data block length in varint, the data block length is bigger than the
> file size? I am getting data block length greater than file length.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] first time using protobuf INTERESTING SCENARIO

[Marc Gravell]

It would probably help if you could be specific about what is happening
now. Have you got as far as running "protoc", or is the problem getting
"protoc" ready? If you have "protoc", what command are you using? Are there
error messages? If so: what do they say? Do you have an example .proto file?

Note: if the barrier is getting "protoc" to work at all, then as a
workaround you might be able to use https://protogen.marcgravell.com -
change the drop-down to "(protoc) C++", paste in your .proto, and click
"generate". Then copy the generated code from the bottom panel.


On Mon, 21 May 2018, 07:20 matt egler,  wrote:

> Greetings! I am new to google proto buf and I have over a million
> phonenumbers and I am designing a phone system for my company that requires
> the use of protobuf in order to complete. I am having trouble compiling a
> .proto file in a qubes linux evnvironment but I am using Eclipse IDE in a
> fedora 26 qube virtual machine... any help on how to compile a .proto file
> in terminal on essentialy fedora -26 linux would be greatly appreciated!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf3 InvalidProtocolBufferException with some strings

[Marc Gravell]

(mainly for the list) see also the stackoverflow question:
https://stackoverflow.com/q/50387660/23354

On Thu, 17 May 2018 at 10:42, Alexey Vishnyakov 
wrote:

> Hello
>
> We using protobuf v.3 to transfer messages from C# client to Java server
> over HTTP.
>
> The message proto looks like this:
>
> message CLIENT_MESSAGE {
> string message = 1;
> }
>
> Both client and server uses UTF-8 character encoding for strings.
>
> Everything is fine whe we are using short string values like "abc", but
> when we trying to transfer string with 198 chars in it, we catchig an
> Exception:
>
>
>com.google.protobuf.InvalidProtocolBufferException:
> While parsing a protocol message, the input ended unexpectedly in the
> middle of a field. This could mean either that the input has been
> truncated or that an embedded message misreported its own length.
>
>
> We tried to compare even byte array containing protobuf data, and didn't
> found a solution.
> For "aaa" string byte array starts with this bytes:
>
> 10 3 97 97 97
>
>
> Where 10 is protobuf field number, and 3 is string length, 69 65 67 is
> "aaa".
>
> For string
>
>
>> "aa"
>
>
> which contains 198 characters in it, byte array starts with this:
>
> 10 198 1 97 97 97
>
>
> Where 10 is protobuf field number, and 198 is string length, and 1 seems
> to be like string identifier, or what?
>
>
> And why protobuf cannot parse this message?
>
> Already spent almost a day on looking for solution for this problem, any
> help appreciated.
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

Oddly neither of those is correct! The actual hex should be:

08-03-1A-06-08-01-10-63-18-01

which is the first version minus some trailing data. The second version is
not valid at all - it isn't just missing data: it is an incomplete and
invalid payload. Unfortunately I'm not familiar enough with the C++ API to
advise on whether you're using the wrong method vs this being a bug.


On Wed, 11 Apr 2018, 01:12 Tony Tony, <xxteknolus...@gmail.com> wrote:

> So I upgraded to 3.5.0, same issue.
>
> I guess I should give some more information: I'm basically using protobuf
> to serialize/deserialize string data I'm sending through a games socket
> implementation via hooks/code caves in the games network engine. I
> currently can validate that the bytes after serialization are sent over
> completely to the client. After looking at the decoded output that you guys
> recommended, I do see some differences.
>
> - When I send type=1, count=99, index=1, the server shows the below bytes
> after serialization and the client shows these bytes after receiving the
> data. The decoding tool shows the correct content.
>
> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD
>
> - When I send type=1, count=0, index=1, the server shows the below bytes
> after serialization and the client shows these bytes after receiving the
> data. The decoding tool basically shows no content, which is obvious from
> the missing data.
>
> 08 03 1A 06 08
>
> The biggest difference is that it seems the during serialization on the
> server, this combo of data seems to break everything. I guess I'm not sure
> why this combo of data would break during serialization. I posted some
> pseudo code of where the problem occurs for that combo of data below. It's
> just strange that only that combo of data above results in this behavior. I
> am able to send large strings just fine, plenty of other data types with no
> issues (well none that I've found yet..)
>
> Packet pak;
> pak.set_type(Packet_Type_set_info);
>
> set_info *setter = pak.mutable_set_info();
> setter->set_count(99);
> setter->set_type(0);
> setter->set_index(1);
>
> char* SendBuf = new char[pak.ByteSize()];
> pak.SerializeToArray(SendBuf, pak.ByteSize());
> //TODO: the bytes seem to be wrong right after this point for the second
> combo of data
> //we send before we delete the sendBuf
>
> delete[] SendBuf;
>
>
> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>>
>> I think Marc is right that it would be best to isolate the problem by
>> manually checking that the bytes are valid.
>>
>> Tony, how are you framing the messages? Protocol buffers do not describe
>> their own size and so you have to know what size to expect before you parse
>> a proto. The typical approach is to frame the serialized messages by
>> prefixing them with their size. It's easy to get this wrong, though, so
>> that could potentially be the reason that the messages aren't showing up as
>> you expect.
>>
>> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell <marc.g...@gmail.com> wrote:
>>
>>> Well, if it was me: the first thing I'd do is isolate whether it is
>>> serialize or deserialize that is failing - by taking your currently
>>> serialized data as a flat file or byte dump, and checking whether the
>>> values are right or wrong.
>>>
>>> Depending on the volume and layout of the data, you might be able to use
>>> protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
>>> welcome to try using https://protogen.marcgravell.com/decode
>>>
>>> That would allow you to focus your checks.
>>>
>>> Marc
>>>
>>> On 10 April 2018 at 14:20, Tony Tony <xxtekn...@gmail.com> wrote:
>>>
>>>> c++/windows
>>>>
>>>> I'll upgrade protobuf and see if issue goes away. Just wanted to make
>>>> sure I'm not missing anything. Is there any additional troubleshooting I
>>>> can look into to troubleshoot further if the recent release reproduce the
>>>> issue?
>>>>
>>>> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>>>>
>>>>> First thought: what language / platform is this? the C++ generated
>>>>> code is very different to the Java generated code, for example
>>>>>
>>>>> Second thought: 2.6 is pretty old; it is very possible that a bug
>>>>> existed and has been fixed since then (Aug 2014) - does it still happen
>>>>> with more recent releases?
>>>>>
>>>>> The scenario you present seems fine an

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

Well, if it was me: the first thing I'd do is isolate whether it is
serialize or deserialize that is failing - by taking your currently
serialized data as a flat file or byte dump, and checking whether the
values are right or wrong.

Depending on the volume and layout of the data, you might be able to use
protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
welcome to try using https://protogen.marcgravell.com/decode

That would allow you to focus your checks.

Marc

On 10 April 2018 at 14:20, Tony Tony <xxteknolus...@gmail.com> wrote:

> c++/windows
>
> I'll upgrade protobuf and see if issue goes away. Just wanted to make sure
> I'm not missing anything. Is there any additional troubleshooting I can
> look into to troubleshoot further if the recent release reproduce the issue?
>
> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>
>> First thought: what language / platform is this? the C++ generated code
>> is very different to the Java generated code, for example
>>
>> Second thought: 2.6 is pretty old; it is very possible that a bug existed
>> and has been fixed since then (Aug 2014) - does it still happen with more
>> recent releases?
>>
>> The scenario you present seems fine and reasonable.
>>
>> Marc
>>
>> On 10 April 2018 at 03:46, Tony Tony <xxtekn...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>>>
>>> Every time I send a certain set of values over my socket, it ends up
>>> with missing data. Although I've compared the string received after
>>> serialization on the server and on receiving the string on the client and
>>> they are exactly the same, yet the values are all -1.
>>>
>>> I've attached a snippet of the proto file below.
>>>
>>> At this point these are below combos that always fail
>>>
>>> type=99, count= 0, index = 1 (all values are -1 after deserialization)
>>> type=99, count=99, index= 1 (works)
>>> type=9, count=99, index= 1 (works)
>>>
>>> - Basically anytime count is 0, all the values seem to be -1. However if
>>> I make type 0, the same behavior does not occur. Am I missing something or
>>> there is some underlying bug in my code? Again I've checked multiple times
>>> the correct raw string data is being sent (with correct size). I use the
>>> parseFromString method to parse the receiving string data.
>>>
>>> - I've sent plenty of other types of data just fine.
>>>
>>> - There are other messages defined in the same protobuf packet
>>> definition file, can that make a difference?
>>>
>>> message packet {
>>> required int32 type = 1 [default = -1];
>>> required int32 count = 2 [default = -1];
>>> required int32 index = 3 [default = -1];
>>> }
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Regards,
>>
>> Marc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

First thought: what language / platform is this? the C++ generated code is
very different to the Java generated code, for example

Second thought: 2.6 is pretty old; it is very possible that a bug existed
and has been fixed since then (Aug 2014) - does it still happen with more
recent releases?

The scenario you present seems fine and reasonable.

Marc

On 10 April 2018 at 03:46, Tony Tony  wrote:

> Hi,
>
> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>
> Every time I send a certain set of values over my socket, it ends up with
> missing data. Although I've compared the string received after
> serialization on the server and on receiving the string on the client and
> they are exactly the same, yet the values are all -1.
>
> I've attached a snippet of the proto file below.
>
> At this point these are below combos that always fail
>
> type=99, count= 0, index = 1 (all values are -1 after deserialization)
> type=99, count=99, index= 1 (works)
> type=9, count=99, index= 1 (works)
>
> - Basically anytime count is 0, all the values seem to be -1. However if I
> make type 0, the same behavior does not occur. Am I missing something or
> there is some underlying bug in my code? Again I've checked multiple times
> the correct raw string data is being sent (with correct size). I use the
> parseFromString method to parse the receiving string data.
>
> - I've sent plenty of other types of data just fine.
>
> - There are other messages defined in the same protobuf packet definition
> file, can that make a difference?
>
> message packet {
> required int32 type = 1 [default = -1];
> required int32 count = 2 [default = -1];
> required int32 index = 3 [default = -1];
> }
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Migrating legacy code to use Protobuff

[Marc Gravell]

Great! Give that a go, and see if it gets you what you need. As a footnote:
I advocate *not* using data tables as a data transfer tool, except in very
specific circumstances (such as ad-hoc query systems where the structure is
very flexible). But that is a huge thing to change.

On 14 Feb 2018 7:07 am, "Som Shankar Bhattacharyya" <
bhattacharyya...@gmail.com> wrote:

> I see no remting format set. Looks like it used the default xml format.
>
> On Wed, Feb 14, 2018 at 12:11 AM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>> Protobuf doesn't touch security, so we can ignore that one.
>>
>> Modelling datasets/DataTable is awkward. It isn't really a natural fit,
>> but it can be manually forced. However, the first thing I'd say is: have
>> you set the "RemotingFormat" on the dataset to **binary** before using your
>> existing serialization? Assuming you are using BinaryFormatter, setting the
>> "RemotingFormat" can have a *huge* efficiency benefit, and might allow you
>> to get most of what you wanted without doing much work.
>>
>> On 13 Feb 2018 8:08 am, "Som Shankar Bhattacharyya" <
>> bhattacharyya...@gmail.com> wrote:
>>
>>> So i work in a legacy Visual Basic project. It is a client server
>>> application. The client application sends up some table data to the server
>>> that persists to a file system. A separate windows service then reads this
>>> data and saves to the server database.
>>> Now the data that is sent up from the client is a datatable in a raw
>>> .NET serialized format. The service that reads this information has the
>>> model on its path and is hence able to reconstruct the objects.
>>>
>>> Now i want to get rid of this flow and use a more secure as well as
>>> better performant way of doing the same.
>>> Can someone give me  an idea of how to approach this ?
>>>
>>> How do i model a dataset in the protobuff language ?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Migrating legacy code to use Protobuff

[Marc Gravell]

Protobuf doesn't touch security, so we can ignore that one.

Modelling datasets/DataTable is awkward. It isn't really a natural fit, but
it can be manually forced. However, the first thing I'd say is: have you
set the "RemotingFormat" on the dataset to **binary** before using your
existing serialization? Assuming you are using BinaryFormatter, setting the
"RemotingFormat" can have a *huge* efficiency benefit, and might allow you
to get most of what you wanted without doing much work.

On 13 Feb 2018 8:08 am, "Som Shankar Bhattacharyya" <
bhattacharyya...@gmail.com> wrote:

> So i work in a legacy Visual Basic project. It is a client server
> application. The client application sends up some table data to the server
> that persists to a file system. A separate windows service then reads this
> data and saves to the server database.
> Now the data that is sent up from the client is a datatable in a raw .NET
> serialized format. The service that reads this information has the model on
> its path and is hence able to reconstruct the objects.
>
> Now i want to get rid of this flow and use a more secure as well as better
> performant way of doing the same.
> Can someone give me  an idea of how to approach this ?
>
> How do i model a dataset in the protobuff language ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] what is TypeModel.cs and why is it missing ?

[Marc Gravell]

This is protobuf-net, so I suppose I should chime in :)

Can you please give a precise copy of the error message? And how did you
reference the library? Did you just add the NuGet package as normal? Or did
you do something more exotic? .NET libraries are almost usually distributed
as compiled assemblies - not as source code, so there is no reason for it
to be looking for TypeModel.cs - a source file. To understand the problem,
I'd need to see the actual error message.

No, this isn't expected.



On 31 Jan 2018 1:51 am, "Erick Treetops"  wrote:

Writing my first GTFS Realtime app in .net..  I had some issues getting out
of our proxy but now I'm stuck because the protobuf is looking for a file
called TypeModel.cs  and can't find it.  I've searched and haven't found
anyone else with this issue.  The error occurs when I try and serialize the
request (last line).
Can anyone point me in the right direction


WebRequest req = HttpWebRequest.Create("https://gtfsrt.api.translink.com.au/
Feed/SEQ");

IWebProxy proxy = req.Proxy;

proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;

FeedMessage feed = Serializer.Deserialize(req.GetResponse().
GetResponseStream());



Erick Treetops

-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Precompiled Version of probuf for VS2017

[Marc Gravell]

Protoc is available for multiple OSes here:
https://github.com/google/protobuf/releases/tag/v3.5.1

Note sure about pre-compiled libs; for Java, they're on mvnrepository; for
C# they're on NuGet, etc.

On 26 January 2018 at 15:22, 'Frank Willen' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Hi,
>
> I want to use the protocol buffers for a mixed solution (C+, C#, both in
> VS2017). Where can I download precompiled libs, headers and the protoc.exe?
> Or is it neccessary to build the probuffers for everyone?
>
> Thanks for your support?
>
> Grettings
> Frank
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Tags and values

[Marc Gravell]

Yes, tags (field numbers) can be non-contiguous.

Yes, hex is accepted by protoc

No, repeated fields cannot be required

On 22 January 2018 at 23:08, Ashwin Kini  wrote:

> Hi all,
>
> While defining messages can the *tags *be non continous? The
> documentation never mentions the same
>
> message abc {
> optional uint32 xyz = *100*;
> optional uint32 def = *200*;
> optional unit32 abc = *0xDEAD01*;
>
> }
>
> The documentation says it needs to be unique. Agreed. Can they be non
> continous as explained in example?  . Also does it take HEX values as far
> as it falls in the range mentioned in the doc?
>
> One more question is, can repeated fields be made required? I did a lot of
> digging, seems its not possible. Can anybody confirm? I understand required
> should be completely avoided.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: How to speedup serialization and deserialization process?

[Marc Gravell]

I did a test locally using protobuf-net (since that is what I'm most
familiar with); to get an output of about 3,784,000 I used a count of
172000 items in the inner array - does that sound about right? Then I
tested it in a loop as per this gist: https://gist.github.com/mgravell/
10a21970531485008731d700b89ec732

The timings I get there are about 25ms to serialize, 40ms to deserialize -
although my machine is quite fast, so this may mean you're pretty much
getting "about right" numbers. What sort of numbers are you looking for
here? Note that in .NET the first run will always be slightly slower due to
JIT.

Additionally, I know nothing about the zmq overheads - are you including
the zmq cost in your numbers?

If you want to squeeze the last few drops of performance, you usually can -
for example, by looking at whether zmq allows you to pass a stream or span
or an *oversized* array - again, I'm not as familiar with the Google C#
version as I am with protobuf-net, but *if* (and it is a huge "if") the
"ToByteArray()" is essentially writing to a MemoryStream then calling
ToArray, you can probably avoid an extra blit and some allocs by providing
your own re-used memory-stream and using GetBuffer to access the oversized
array, remembering to limit to just the first .Length bytes. But again a
lot of this depends on specifics of zmq and the Google C# version. It is
almost certainly diminishing returns.

So: what numbers are you *looking to get*? what would be "acceptable"? And
how complex is your data? is what you've shown the *only* data you need to
transfer? if so, it might not be a bad candidate for fully manual explicit
serialization not involving a library - just a payload of:

Height [int32 fixed 4 bytes]
Width [int32 fixed 4 bytes]
Time [int64 fixed 8 bytes]
ElementCount [int32 fixed 4 bytes]
then for each element: 16 bytes consisting of
X [float fixed 4 bytes]
Y [float fixed 4 bytes]
Z [float fixed 4 bytes]
RGB [int32 fixed 4 bytes]

this would require manual coding, but would typically outperform other
options - but would be more brittle and would require you to be reasonably
good at IO code.

Personally, I'd probably leave it alone...

Marc



On 22 December 2017 at 13:03, Ravi  wrote:

> Any suggestions, please?
>
>
> On Wednesday, December 20, 2017 at 10:00:31 PM UTC+9, Ravi wrote:
>>
>> I have defined the Protocol Buffers message file as follows:
>>
>> syntax = "proto3";
>> package Tutorial;
>> import "google/protobuf/timestamp.proto";
>> option csharp_namespace = "Tutorial";
>>
>> message PointCloud {
>>   int32 width  = 1;
>>   int32 height = 2;
>>
>>   message Point {
>> float x = 1;
>> float y = 2;
>> float z = 3;
>> fixed32 rgb = 4;
>>   }
>>   repeated Point points = 3;
>>   google.protobuf.Timestamp timestamp = 4;
>> }
>>
>> This is how I am preparing the data and serializing it in C#:
>> using Google.Protobuf;
>> using Tutorial;
>> using ZeroMQ;
>>
>> PointCloud pointCloud = new PointCloud();
>> pointCloud.Height = Height
>> pointCloud.Width = Width;
>> pointCloud.Timestamp = Timestamp.FromDateTime(DateTime.UtcNow);
>>
>> for (var index = 0; index < POINTS3D_COUNT; index++) {
>>   PointCloud.Types.Point point = new PointCloud.Types.Point {
>> X = points3D[index].X,
>> Y = points3D[index].Y,
>> Z = points3D[index].Z,
>> Rgb = (uint)((red << 16) | (green << 8) | blue)
>>   };
>>
>>   pointCloud.Points.Add(point);
>> }
>>
>> zmqPublisher.Send(new ZFrame(pointCloud.ToByteArray()));
>>
>> This is how I deserialize the data in C++:
>> while (receive) {
>>   zmq::message_t msg;
>>   int rc = zmq_socket.recv();
>>   if (rc) {
>> Tutorial::PointCloud point_cloud;
>> point_cloud.ParseFromArray(msg.data(), msg.size());
>>   }point_cloud.ParseFromArray(msg.data(), msg.size())
>> }
>>
>> I am able to get the data back properly. However, the serialization and
>> deserialization process seems slow.
>>
>>- I used *System.Diagnostics.Stopwatch *in C# and noticed that
>>*pointCloud.ToByteArray()* is taking 100ms approximately.
>>- Similarly I used *std::chrono::steady_clock::now()* in C++ and
>>noticed that *point_cloud.ParseFromArray(msg.data(), msg.size())* is
>>taking 96ms approximately.
>>- Just for information, the length of the byte array is roughly
>>3,784,000.
>>
>>
>> *I want to know how to speed up serialization and deserialization
>> process?*
>>
>> -
>> Thanks
>> Ravi
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the 

Re: [protobuf] Re: validating protobuf messages

[Marc Gravell]

a zero tag is never valid in any protobuf data, although it wouldn't be
unheard of for folks to use a zero tag as a sentinel value to demark
multiple root messages. Protoc has some facilities to check the insides of
a message that might help you figure out how likely it is to be a match,
but it won't make a zero tag work.

On 10 Dec 2017 6:35 p.m., "Annu Bansal"  wrote:

> Hey Jeroen!
> Dis you get any method to check if a .pb file is generated from givem
> .proto file? Actually, I'm getting error while deserializing a .pb file,
> "The protocol message contained invalid tag(zero)." I was wondering if the
> two files are compatible.
> Thanks in advance.
>
> On Saturday, November 17, 2012 at 5:38:16 AM UTC+5:30, Jeroen Ooms wrote:
>>
>> I am using the RProtoBuf package, which interfaces to the c++ protobuf
>> library. I was wondering if there is any way of 'validating' a pb
>> message, in the sense that you can check if it actually is a valid
>> message for a given pb description. Currently, RProtoBuf is very
>> permissive and will basically accept anything. E.g the code below will
>> not throw an error, even though the file obviously is not a valid
>> tutorial.Person message:
>>
>> library(RProtoBuf)
>> read(tutorial.Person, "/etc/passwd")
>>
>> In practice, it is quite easy to introduce errors by accidentally reading
>> a message using the wrong description. It would be very helpful if there
>> was a way of preventing this kind of problems by some sort of validation.
>>
>> Is there support for this in the c++ library, or does protobuf basically
>> assume that the user/application has some other way of validating a message?
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

oh, there is a root. if i had to guess it has something like:


message TheRootMessageType {
required string name = 1;
repeated SomeNoun items = 100;
}



On 10 Dec 2017 4:42 p.m., "Jim Baldwin" <j...@baldw.in> wrote:

> In this case, it looks like there isn’t a root.  Rather, this format is a
> series of top-level parameters.  So, I have to give it the Parameter I’m
> looking for.  The problem I have with this is the order of parameters
> _might_ matter, and I lose that by only looking for one.
>
> On Dec 10, 2017, at 8:35 AM, Marc Gravell <marc.grav...@gmail.com> wrote:
>
> as I ubderstand it, --decode *will do that*. it doesn't decode *just* the
> root : but, it needs to know the root message type in order to correctly
> interpret the data
>
> On 10 Dec 2017 4:32 p.m., "Jim Baldwin" <jmbl...@gmail.com> wrote:
>
>> OK, this helps.  I need to figure out what the "root" message is.  It
>> seems like an omission in the whole PB thing that you can't specify the
>> .proto and do a --decode_everything.
>>
>> On Sunday, December 10, 2017 at 8:23:12 AM UTC-8, Marc Gravell wrote:
>>>
>>> You can and it does. The problem is that the wire format by itself
>>> doesn't tell it **what message type** the root object is. So you need to
>>> tell it in the additional parameter to --decode
>>>
>>> On 10 Dec 2017 3:14 p.m., "Jim Baldwin" <jmb...@gmail.com> wrote:
>>>
>>> It's not really just a sequence; it's a hierarchy, isn't it?  Why can't
>>> I use --decode  or something like that?
>>>
>>>
>>> On Saturday, December 9, 2017 at 4:20:15 PM UTC-8, Ilia Mirkin wrote:
>>>
>>>> On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin <jmb...@gmail.com> wrote:
>>>> > I have a protobuf file, and a .proto file that describes the schema.
>>>> >
>>>> > The .proto describes dozens of different messages that may be in the
>>>> > protobuf file.
>>>> >
>>>> > I would like to know what messages can be found in the file.  I do a
>>>> protoc
>>>> > --decode_raw and get something out, but I don't see how to use that to
>>>>
>>>> > figure out how to extract messages from the file.
>>>> >
>>>> > I assume there's something I don't get about protobufs, but it seems
>>>> to me I
>>>> > should be able to take a protobuf data file and corresponding .proto
>>>> and
>>>> > turn it into a file that lets me see what the message hierarchy is in
>>>> the
>>>> > file.  JSON would be a great way to do that.
>>>> >
>>>> > What am I missing?
>>>>
>>>> An encoded protobuf is just a sequence of (tag, value) pairs. If you
>>>> don't know which proto it is, decode_raw is the best you can do. If
>>>> you do know which proto it is, you can use --decode instead and pass
>>>> it a proto name to use for the decoding.
>>>>
>>>> Cheers,
>>>>
>>>>   -ilia
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

as I ubderstand it, --decode *will do that*. it doesn't decode *just* the
root : but, it needs to know the root message type in order to correctly
interpret the data

On 10 Dec 2017 4:32 p.m., "Jim Baldwin" <jmbl...@gmail.com> wrote:

> OK, this helps.  I need to figure out what the "root" message is.  It
> seems like an omission in the whole PB thing that you can't specify the
> .proto and do a --decode_everything.
>
> On Sunday, December 10, 2017 at 8:23:12 AM UTC-8, Marc Gravell wrote:
>>
>> You can and it does. The problem is that the wire format by itself
>> doesn't tell it **what message type** the root object is. So you need to
>> tell it in the additional parameter to --decode
>>
>> On 10 Dec 2017 3:14 p.m., "Jim Baldwin" <jmb...@gmail.com> wrote:
>>
>> It's not really just a sequence; it's a hierarchy, isn't it?  Why can't I
>> use --decode  or something like that?
>>
>>
>> On Saturday, December 9, 2017 at 4:20:15 PM UTC-8, Ilia Mirkin wrote:
>>
>>> On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin <jmb...@gmail.com> wrote:
>>> > I have a protobuf file, and a .proto file that describes the schema.
>>> >
>>> > The .proto describes dozens of different messages that may be in the
>>> > protobuf file.
>>> >
>>> > I would like to know what messages can be found in the file.  I do a
>>> protoc
>>> > --decode_raw and get something out, but I don't see how to use that to
>>> > figure out how to extract messages from the file.
>>> >
>>> > I assume there's something I don't get about protobufs, but it seems
>>> to me I
>>> > should be able to take a protobuf data file and corresponding .proto
>>> and
>>> > turn it into a file that lets me see what the message hierarchy is in
>>> the
>>> > file.  JSON would be a great way to do that.
>>> >
>>> > What am I missing?
>>>
>>> An encoded protobuf is just a sequence of (tag, value) pairs. If you
>>> don't know which proto it is, decode_raw is the best you can do. If
>>> you do know which proto it is, you can use --decode instead and pass
>>> it a proto name to use for the decoding.
>>>
>>> Cheers,
>>>
>>>   -ilia
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+u...@googlegroups.com.
>> To post to this group, send email to prot...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

They are field numbers. They don't mean anything by themselves other than
to identify each field. If you want to know the logical *name* of each
field, you need the .proto schema.

On 10 Dec 2017 4:23 p.m., "Jim Baldwin"  wrote:

> Perhaps it might help if I understood the output of protoc --decode_raw.
>
> Here's an example of a .caffemodel file I'm trying to inspect.  Is there a
> description of what the numbers mean in this file?
>
> 1: "VGG_ILSVRC_16_layers"
> 100 {
>   1: "input-data"
>   2: "Python"
>   4: "data"
>   4: "im_info"
>   4: "gt_boxes"
>   10: 0
>   130 {
> 1: "roi_data_layer.layer"
> 2: "RoIDataLayer"
> 3: "\'num_classes\': 2"
>   }
> }
> 100 {
>   1: "data_input-data_0_split"
>   2: "Split"
>   3: "data"
>   4: "data_input-data_0_split_0"
>   4: "data_input-data_0_split_1"
>   10: 0
> }
>
> On Saturday, December 9, 2017 at 2:52:10 PM UTC-8, Jim Baldwin wrote:
>>
>> I have a protobuf file, and a .proto file that describes the schema.
>>
>> The .proto describes dozens of different messages that may be in the
>> protobuf file.
>>
>> I would like to know what messages can be found in the file.  I do a
>> protoc --decode_raw and get something out, but I don't see how to use that
>> to figure out how to extract messages from the file.
>>
>> I assume there's something I don't get about protobufs, but it seems to
>> me I should be able to take a protobuf data file and corresponding .proto
>> and turn it into a file that lets me see what the message hierarchy is in
>> the file.  JSON would be a great way to do that.
>>
>> What am I missing?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

You can and it does. The problem is that the wire format by itself doesn't
tell it **what message type** the root object is. So you need to tell it in
the additional parameter to --decode

On 10 Dec 2017 3:14 p.m., "Jim Baldwin"  wrote:

It's not really just a sequence; it's a hierarchy, isn't it?  Why can't I
use --decode  or something like that?


On Saturday, December 9, 2017 at 4:20:15 PM UTC-8, Ilia Mirkin wrote:

> On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin  wrote:
> > I have a protobuf file, and a .proto file that describes the schema.
> >
> > The .proto describes dozens of different messages that may be in the
> > protobuf file.
> >
> > I would like to know what messages can be found in the file.  I do a
> protoc
> > --decode_raw and get something out, but I don't see how to use that to
> > figure out how to extract messages from the file.
> >
> > I assume there's something I don't get about protobufs, but it seems to
> me I
> > should be able to take a protobuf data file and corresponding .proto and
> > turn it into a file that lets me see what the message hierarchy is in
> the
> > file.  JSON would be a great way to do that.
> >
> > What am I missing?
>
> An encoded protobuf is just a sequence of (tag, value) pairs. If you
> don't know which proto it is, decode_raw is the best you can do. If
> you do know which proto it is, you can use --decode instead and pass
> it a proto name to use for the decoding.
>
> Cheers,
>
>   -ilia
>
-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Guid conventions

[Marc Gravell]

One thought on the bytes vs string aspect: as shown in the SO question, it
isn't beyond imagining to want to use a guid/uuid with "map" - which
wouldn't be possible with (raw) bytes.

As an additional consideration for bytes : there are at least two binary
representations of guid/uuid - i.e. which order the bytes come vs the
visual string representation (either left to right byte equivalent, or a
crazy-endian version where individual portions at different 1/2/4-byte
lengths are individually reversed).

It is these ambiguities that makes me think - after hacking around it for a
long time - that a specific type might be useful.

Proposal

For efficiency, I would propose a new primitive keyword in .proto terms,
that is essentially the same as "bytes", but is mapped (when possible) to a
target platforms natural unique identifier primitive. If no such primitive
is available (JavaScript, for example), it would be decoded *creating a
string*, but expanding to the hex representations - so the byte 0xB7 would
contribute the two characters "B7" to the string. To be clarified: with of
without hyphen group separators. The reason for this string is to
facilitate use in maps, and the correct unit semantics, even on platforms
without a native uuid type. In terms of the binary payload: it would be
left-to-right byte-for-byte with the visual representation, so the uuid
starting "010203040506..." would contribute bytes 0x010203040506...

The size would always be 16 bytes but would be written with the usual
length prefix, so: 17 bytes overall. Any unexpected length prefix
encountered would result in a deserialization failure. Where a platform
doesn't have a native uuid type this 16 byte restriction would be enforced
at the point of assignment/addition.

Additional thought: "repeated uuid" (or whatever the keyword) *would be
allowed to use "packed" encoding. A receiver would divide the packed length
prefix by 16 to determine the number of elements.

In JSON it would be formatted in the text version described above for
platforms without a native uuid type.

---end

So: that's my idea of how uuids *should* be handled. But there's quite a
bit of work, and it sounds like not much dramatic call for them to be
added. So it probably won't happen, being realistic. But I just wanted to
brain-dump that.

Marc



On 8 Dec 2017 12:39 a.m., "Adam Cozzette" <acozze...@google.com> wrote:

> I haven't had to store a GUID/UUID in a proto before but it seems like
> string or bytes would be the best choice. You would definitely want to use
> bytes (not string) if you're using the binary representation, since string
> fields are for UTF-8 only. We could consider eventually creating a
> well-known type but I'm not sure how much demand there is for one.
>
> On Wed, Dec 6, 2017 at 11:46 AM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>> A question on Stack Overflow earlier (https://stackoverflow.com/que
>> stions/47674930/google-protobuf-proto-file-query/4767629) reminded me
>> that I'm not fully "up" on the conventions for using guids in protobuf.
>>
>> There's no primitive / keyword for them, and AFAIK no "well known type".
>> So : how do folks tend to handle guids? Strings? Bytes? *Should* there be a
>> stronger guid story? Or is this just a non-issue?
>>
>> Thoughts?
>>
>> Marc
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Validate .proto file in Java

[Marc Gravell]

Apparently https://github.com/square/wire/ includes a runtime .proto parser
for Java. That might help? It isn't the official one, note.

On 6 Dec 2017 8:28 p.m., "Omar Al-Safi"  wrote:

> Hello folks,
>
> I stubbled upon a requirements that I receive a textual representation of
> a proto file from an external service and then I need to save it. I want
> first to check if the text is a valid protobuf schema representation, is
> there way a to validate the syntax of a proto file during the runtime? I
> know that I can use the protoc compiler to validate a file and produce the
> compiled schema but my intention here just only to* validate the syntax*
> of the textual representation of a text if is a valid proto schema syntax.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Guid conventions

[Marc Gravell]

A question on Stack Overflow earlier (
https://stackoverflow.com/questions/47674930/google-protobuf-proto-file-query/4767629)
reminded me that I'm not fully "up" on the conventions for using guids in
protobuf.

There's no primitive / keyword for them, and AFAIK no "well known type". So
: how do folks tend to handle guids? Strings? Bytes? *Should* there be a
stronger guid story? Or is this just a non-issue?

Thoughts?

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Upgrading float fields to double

[Marc Gravell]

Not really, no. They take different amounts of space on the wire, and have
a different declared wire type (header). Some libraries may choose to be
gracious and apply the conversion silently, but other libraries could just
say "unexpected wire type" and stop processing.

You could perhaps do it as a `oneof`, and have the client check both?

On 15 Nov 2017 11:51 p.m., "James Philbin"  wrote:

> Hi,
>
> Is it safe to upgrade float fields to doubles for serialized protos? If
> not, why not?
>
> Thanks,
> James
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to combine FieldOptions with Default Values in Messages

[Marc Gravell]

requested claritication and *possibly* answered here:
https://stackoverflow.com/questions/47246474/protobuf-how-to-combine-fieldoptions-with-default-values-in-messages

On 12 Nov 2017 7:12 a.m.,  wrote:

> I have an message using FieldOptions but I also want to use default values
> as well, how do I combine two of those things ?
>
> extend google.protobuf.MessageOptions {
> optional CommandInfo cmd = 5678;
> }
>
>
> message TestCmd {
>
> optional bool BoolFlag = 1 [(info) = { name: "bool-flag" value: "false" 
> shorthand: "c" usage: "test"} ];
> optional string StringFlag = 2 [ (info) = { name: "string-flag" value: "" 
> shorthand: "o" usage: "test"}];
>
> }
>
>
>
> I want that TestCmd.BoolFlag default value to be bool value  "false", 
> currently I'm using string value then convert it to bool, How do I
>
> specify the default value in a type-safe way ?
>
>
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Integrate proto2 messages from a device to C#

[Marc Gravell]

protobuf-net should work with proto 2 - try here:
https://protogen.marcgravell.com

On 27 Oct 2017 6:16 p.m., "cleal"  wrote:

> I have to integrate a third part protocol from a gps device, they send me
> the .proto files, they are using the schema proto2.
>
> I should receive the binary messages and I need desarialize it in order to
> work with the location data.  But I see in forums that C# only works for
> proto3.  There is no way to migrate the messages because they are from a
> hardware device from a third part.
>
> I have never working with the Protocol Buffers, how I should start?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Reg: ProtocolBuf-net

[Marc Gravell]

k; you have a csv file... if you have a csv *file*, it is ... already
serialized ... as ... csv. So I'm unclear how this relates to protobuf. It
kinda sounds like you've asked a csv / aspx question on a thread that is
*completely unrelated* to either of those concepts. I'm going to give very
little answer here, mostly because I don't think this is the best place to
*get* the answer you want.

1: paging
2: don't display thousands of lines to a user - that is rarely useful
3: inbuilt controls like gridviews
4: perhaps ask on Stack Overflow with a question that gives *much* more
context and makes it clear exactly what you've tried and what didn't work
when you did so

Marc

On 26 October 2017 at 10:55, Mamy Andriamasinoro <
adrianemo.proj...@gmail.com> wrote:

> Hi Marc;
>
> I'm trying to serialize a csv file which contains thousands lines of query
> results. It works fine, but I'm stuck on how to display the results on an
> aspx page. Any idea please?
>
>
> Le jeudi 27 mai 2010 09:30:03 UTC+3, Marc Gravell a écrit :
>
>> With "full" .NET to .NET WCF, then switching the serialization layer to
>> use protobuf-net can (depending on a few factors) be as simple as a
>> configuration file change:
>> http://marcgravell.blogspot.com/2009/11/controlling-wcf-prot
>> obuf-net-at.html
>>
>>
>> <http://marcgravell.blogspot.com/2009/11/controlling-wcf-protobuf-net-at.html>*Unfortunately*,
>> this doesn't apply to Silverlight since Silverlight lacks that extension
>> point. There is no *silent* way of switching the serializer for
>> Silverlight. Instead the main mechanism for this would instead resolve
>> around exposing (on your WCF API) binary data - either byte[] or
>> Stream. You would, of course, then have to serialize/deserialize
>> manually. Nowhere near as elegant as you can achieve in full .NET, but it
>> still works. WCF essentially then becomes a plumbing layer, rather than
>> representing the actual data API.
>>
>> On the topic of data-contracts, you have a few options - if you have
>> existing data-contracts of the type:
>>
>> [DataContract]
>> public class MyData {
>> [DataMember]
>> public int Foo {get;set;}
>>
>> [DataMember]
>> public string Bar {get;set;}
>> }
>>
>> then all that is required is to associate a unique number with each
>> member (unique within the type, if you see what I mean). This can be done
>> using the Order property:
>>
>> [DataContract]
>> public class MyData {
>> [DataMember(Order=1)]
>> public int Foo {get;set;}
>>
>> [DataMember(Order=2)]
>> public string Bar {get;set;}
>> }
>>
>> or using protobuf-net's own attributes:
>>
>> [ProtoContract]
>> public class MyData {
>> [ProtoMember(1)]
>> public int Foo {get;set;}
>>
>> [ProtoMember(2)]
>> public string Bar {get;set;}
>> }
>>
>> It is important that the client and server agree about the numbers.
>>
>> Alternatively, if you *don't* have existing types you can also start from
>> a .proto file; the generator included with protobuf-net can emit suitable
>> types for you.
>>
>> Re transport; if you are using WCF over HTTP, it is usually worthwhile
>> enable MTOM; but I can't recall whether this is supported under Silverlight.
>>
>> Re serialization; if we assume you are throwing byte[] up and down the
>> wire, you should just need something like:
>>
>> MyData obj = new MyData { Foo = 123, Bar = "abc" };
>> byte[] blob;
>> using(var ms = new MemoryStream()) {
>> Serializer.Serialize(ms,obj);
>> blob = ms.ToArray();
>> }
>>
>> and then to deserialize:
>>
>> MyData obj;
>> using(var ms = new MemoryStream(blob)) {
>> obj = Serializer.Deserialize(ms);
>> }
>>
>> That should cover the main points. If there are more questions, please
>> let me know.
>>
>> Marc Gravell
>> (protobuf-net)
>>
>>
>>
>> On 27 May 2010 05:52, vikram <vikram...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am new to Protocol Buffer.
>>> I wanted to implement protocolbuffer for binary serialization for my
>>> WCF service.
>>>
>>> My requirement goes as given below:
>>>
>>> FrontEnd: SilverLight 3.0
>>> Middlelayer:  WCF Service (.NET 3.5)
>>>
>>> I would like to know what things need to be incopora

Re: [protobuf] Not able to serialize Object type

[Marc Gravell]

For completeness (mostly in case someone finds this via search in the
future), if you were using .proto schema-based generation, "oneof" is your
friend: https://protogen.marcgravell.com/#g01989f8da1f4ce8ab358756115478c37
(click "Generate" once the gist has loaded)

On 23 October 2017 at 09:48, Marc Gravell <marc.grav...@gmail.com> wrote:

> Yes, something like that; example: https://pastebin.com/CUvWz00L
>
> On 23 October 2017 at 09:38, Nihar Mishra <itnmis...@gmail.com> wrote:
>
>> Hi Marc,
>> Thanks a lot. Yes this is protobuf-net.
>> Still i don't get how to modify AttributeValueType class.
>>
>> Do you mean that i should write 4 protomembers for the 4 types?
>>
>> best Regards,
>> Nihar
>>
>>
>> On Mon, Oct 23, 2017 at 9:19 AM, Marc Gravell <marc.grav...@gmail.com>
>> wrote:
>>
>>>  I'm assuming this is protobuf-net; the message is right : the library
>>> can't work with "object itemField". Perhaps the best thing would be to
>>> treat this like a "oneof" (in .proto terms) and have an accessor per
>>> possible type. For example:
>>>
>>> [ProtoMember(n)]
>>> private int ValueInt32 {
>>> get => (int)itemField;
>>> set => itemField = value;
>>> }
>>> private bool ShouldSerializeValueInt32() => itemField is int;
>>>
>>> (with a different "n" per possible type)
>>>
>>> The "ShouldSerialize..." pattern is a .net idiom supported by
>>> protobuf-net that provides memberwise conditional serialization.
>>>
>>>
>>>
>>> On 23 Oct 2017 8:07 a.m., "Nihar Mishra" <itnmis...@gmail.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I have below class which is generated from .net client.
>>>> When i try to save in redis cache, i get the "Not able to serialize
>>>> Object type" error.
>>>> Please help me how to resolve it.
>>>>
>>>>   public partial class AttributeValueType
>>>> {
>>>> [DataMember]
>>>> private object itemField;
>>>>
>>>>
>>>> [System.Xml.Serialization.XmlElementAttribute("IntValue",
>>>> typeof(int))]
>>>> [System.Xml.Serialization.XmlElementAttribute("StringValue",
>>>> typeof(string))]
>>>> [System.Xml.Serialization.XmlElementAttribute("dateValue",
>>>> typeof(System.DateTime), DataType = "date")]
>>>> [System.Xml.Serialization.XmlElementAttribute("doubleValue",
>>>> typeof(double))]
>>>> public object Item
>>>> {
>>>> get { return this.itemField; }
>>>> set { this.itemField = value; }
>>>> }
>>>> }
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to protobuf+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to protobuf@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>
>
>
> --
> Regards,
>
> Marc
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Not able to serialize Object type

[Marc Gravell]

Yes, something like that; example: https://pastebin.com/CUvWz00L

On 23 October 2017 at 09:38, Nihar Mishra <itnmis...@gmail.com> wrote:

> Hi Marc,
> Thanks a lot. Yes this is protobuf-net.
> Still i don't get how to modify AttributeValueType class.
>
> Do you mean that i should write 4 protomembers for the 4 types?
>
> best Regards,
> Nihar
>
>
> On Mon, Oct 23, 2017 at 9:19 AM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>>  I'm assuming this is protobuf-net; the message is right : the library
>> can't work with "object itemField". Perhaps the best thing would be to
>> treat this like a "oneof" (in .proto terms) and have an accessor per
>> possible type. For example:
>>
>> [ProtoMember(n)]
>> private int ValueInt32 {
>> get => (int)itemField;
>> set => itemField = value;
>> }
>> private bool ShouldSerializeValueInt32() => itemField is int;
>>
>> (with a different "n" per possible type)
>>
>> The "ShouldSerialize..." pattern is a .net idiom supported by
>> protobuf-net that provides memberwise conditional serialization.
>>
>>
>>
>> On 23 Oct 2017 8:07 a.m., "Nihar Mishra" <itnmis...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I have below class which is generated from .net client.
>>> When i try to save in redis cache, i get the "Not able to serialize
>>> Object type" error.
>>> Please help me how to resolve it.
>>>
>>>   public partial class AttributeValueType
>>> {
>>> [DataMember]
>>> private object itemField;
>>>
>>>
>>> [System.Xml.Serialization.XmlElementAttribute("IntValue",
>>> typeof(int))]
>>> [System.Xml.Serialization.XmlElementAttribute("StringValue",
>>> typeof(string))]
>>> [System.Xml.Serialization.XmlElementAttribute("dateValue",
>>> typeof(System.DateTime), DataType = "date")]
>>> [System.Xml.Serialization.XmlElementAttribute("doubleValue",
>>> typeof(double))]
>>> public object Item
>>> {
>>> get { return this.itemField; }
>>> set { this.itemField = value; }
>>> }
>>> }
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Not able to serialize Object type

[Marc Gravell]

 I'm assuming this is protobuf-net; the message is right : the library
can't work with "object itemField". Perhaps the best thing would be to
treat this like a "oneof" (in .proto terms) and have an accessor per
possible type. For example:

[ProtoMember(n)]
private int ValueInt32 {
get => (int)itemField;
set => itemField = value;
}
private bool ShouldSerializeValueInt32() => itemField is int;

(with a different "n" per possible type)

The "ShouldSerialize..." pattern is a .net idiom supported by protobuf-net
that provides memberwise conditional serialization.



On 23 Oct 2017 8:07 a.m., "Nihar Mishra"  wrote:

>
> Hi,
>
> I have below class which is generated from .net client.
> When i try to save in redis cache, i get the "Not able to serialize Object
> type" error.
> Please help me how to resolve it.
>
>   public partial class AttributeValueType
> {
> [DataMember]
> private object itemField;
>
>
> [System.Xml.Serialization.XmlElementAttribute("IntValue",
> typeof(int))]
> [System.Xml.Serialization.XmlElementAttribute("StringValue",
> typeof(string))]
> [System.Xml.Serialization.XmlElementAttribute("dateValue",
> typeof(System.DateTime), DataType = "date")]
> [System.Xml.Serialization.XmlElementAttribute("doubleValue",
> typeof(double))]
> public object Item
> {
> get { return this.itemField; }
> set { this.itemField = value; }
> }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Null String in C# (CheckForNull)

[Marc Gravell]

Fields are optional but the implicit default for a string is a zero length
string, not a null length string. To be honest, either approach seems
perfectly reasonable as long as it is documented and any exception is clear
and obvious. For my separate implementation I chose to interpret nulls as
"meh, not set, ignore it", but... either is fine IMO.

On 3 October 2017 at 19:35, Everton Araujo  wrote:

> Hi all,
>
> I'm facing the situation where a string value is not allowed to be null in
> Protobuf 3 using C# (CheckNotNull).
> I do not understand it considering that fields are optional by default in
> proto3.
> Does anyone knows the original reason for this constraint? As far as I
> could see, it happens for string and bytearrays.
>
> Regards,
>
> Everton
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: How can i get bytes from a serialized message without the refix-length before each field?

[Marc Gravell]

(duplicate with existing answers 
here: 
https://stackoverflow.com/questions/45501605/is-it-possible-to-use-google-protobuffer-to-serialize-data-without-prefixing-the)

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Are protobufs serialised canonically?

[Marc Gravell]

Formally: no.

Practically: almost always

You shouldn't **demand** it.

Basically, it goes like this:

- the spec asks that writers *should* write fields in order
- the spec asks that readers *must* allow fields in any order
- data can be concatenated as a merge, meaning fields can appear out of
order
- how extension / unexpected fields are handled is often... inconsistent
- but in most cases: it'll probably work in the way you expect - just: it
isn't **required** to be identical

I'm aware of cases even in "protoc" (so: the C++ version) where it writes
fields out of order (relating to extension fields).

Additionally, the format itself allows for the same value to be expressed
in different ways - in particular as varint. Example: the value 1 is
usually expressed as the byte { 1 }, but actually { 129, 0 } and { 129,
128, 0 } should (if I have my math right) evaluate as 1. I am not aware of
anything that makes this alternative byte sequence formally invalid,
although I'd love to see an explicit / official answer on that.

Does that help any?

On 25 Jul 2017 9:26 p.m.,  wrote:

(Attempt 2 - I'm not sure if messages to this list and moderated or whether
my first one just got lost in the ether)

Are protobufs serialised canonically?  By that I mean is the same message
with the same fields populated with the same data guaranteed to serialise
to the same sequence of bytes, regardless of language/implementation
version?

The reason I ask is we're using serialised single-field protos as keys in a
Kafka topic, and I'm now wondering if this is safe.  The idea was to avoid
ambuiguity about how to interpret the contents of the key.  However, a
critical requirement of a key is that the same key must serialise to the
same bytes (Kafka is only aware of bytes - nothing else).

Is what I'm doing safe?  Is it safe for some subset of proto field types
(for example, I'm already pretty sure it's not safe for map types because
there is no canonical ordering, right?)?

Thanks,
Alex

-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.