Re: [protobuf] Default Values vs Missing Values
You can't distinguish an empty repeated from one that's not there at all. If you need that, you'll need a manual presence field. On Tue, Mar 29, 2016 at 2:02 AM, Yoav H wrote: > How do they handle collections (repeated, non packed) in this case? > The absence of the tag is not conclusive. > Actually, even packed collection (and strings, and binary data) suffer from > that, as you are "expected" to not include a packed collection with zero > bytes. > > > On Saturday, March 26, 2016 at 1:08:23 PM UTC-7, Ilia Mirkin wrote: >> >> Encoding is identical... just the API is different. In proto2, you >> have (in C++) FooMessage->has_field() which will tell you whether a >> field was present in the encoded version (or has been set prior if >> you're building a new message). The Java API has something rather >> similar... hasField() I think? >> >> On Sat, Mar 26, 2016 at 4:00 PM, Yoav H wrote: >> > Thanks all, >> > >> > Do you know where I can find the proto2 encoding guide? >> > The proto site has only the proto3 encoding described. >> > >> > On Saturday, March 26, 2016 at 12:21:39 PM UTC-7, Tim Kientzle wrote: >> >> >> >> >> >> > On Mar 26, 2016, at 11:43 AM, Yoav H wrote: >> >> > >> >> > Hi, >> >> > >> >> > I wanted ask regarding the decision to populate fields with default >> >> > values, even if they do not appear in the encoded message. >> >> > If I want to send a "patch" message, where I want to update just the >> >> > provided fields, how can I do that with protobuf (without adding >> >> > IsXXXSet >> >> > for every field)? >> >> > >> >> > Why not add another type, representing a default value? >> >> > So the schematics would be, if the field is missing, it is null, and >> >> > if >> >> > the field exists, but with this "missing value" type, it will get the >> >> > default value? >> >> >> >> As Ilia pointed out, proto2 still exists, is still supported, and can >> >> be >> >> used for >> >> cases where you require these particular semantics. >> >> >> >> For proto3, you might look at google.protobuf.FieldMask, which is a new >> >> standard message (one of the "well-known types") specifically designed >> >> to store a set of field names. You might be able to achieve what you >> >> want >> >> by >> >> providing a FieldMask with your data listing the specific fields to >> >> be updated. >> >> >> >> Tim >> >> >> >> >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Protocol Buffers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to protobuf+u...@googlegroups.com. >> > To post to this group, send email to prot...@googlegroups.com. >> > Visit this group at https://groups.google.com/group/protobuf. >> > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Getting the full (on disk) path to an input .proto from plugin
Hi all, I'm the author of protoc-gen-doc, a documentation generator plugin for protoc. I need to be able to get the full (on disk) path of an input file, but can't find a way to do so. The reason I need it is that I need to be able to open the file to do some custom parsing of the raw file contents (the info in the descriptors is not enough for this particular thing I need to do). FileDescriptor::name will give me the path relative to the source tree, but I can't find a way to get the corresponding SourceTree instance used by the compiler, so that I can use e.g. SourceTree::VirtualFileToDiskFile to resolve the full path. Am I stuck completely here? Thankful for any tips. Cheers, Elvis -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Streaming Serialization - Suggestion
On Mon, Mar 28, 2016 at 10:53 PM, Yoav H wrote: > They say on their website: "When evaluating new features, we look for > additions that are very widely useful or very simple". > What I'm suggesting here is both very useful (speeding up serialization > and eliminating memory duplication) and very simple (simple additions to > the encoding, no need to change the language). > So far, no response from the Google guys... > Actually there are already a "start embedding" tag and a "end embedding" tag in protobuf: https://developers.google.com/protocol-buffers/docs/encoding#structure 3 Start group groups (deprecated) 4 End group groups (deprecated) They are deprecated though. You mentioned it will be a performance gain, but what we experienced in google says otherwise. For example, in a lot places we are only interested in a few fields and want to skip through all other fields (if we are building a proxy, or the field is simply an unknown field). The start group/end group tag pair forces the parser to decode every single field in the a whole group even the whole group is to be ignored after parsing, and that's a very significant drawback. And adding a new wire tag type to protobuf is not a simple thing. Actually I don't think we have added any new wire type to protobuf before. There are a lot issues to consider. For example, isn't all code that switch on protobuf wire types now suddenly broken? if a new serializer uses the new wire type in its output, what will happen if the parsers can't understand it? Proto3 is already finalized and we will not add new wire types in proto3. Whether to add it in proto4 depends on whether we have a good use for it and whether we can mitigate the risks of rolling out a new wire type. > > > On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: >> >> This exact suggestion has been up for discussion long time ago(years?, >> before proto2?) >> >> When it comes to taking suggestions I'm only a 3rd party implementer but >> my understanding is that the design process of protocol buffers and its >> goals are internal to Google and they usually publish new versions of their >> code implementing new features before you can read about them in the >> documents. >> On Mar 27, 2016 5:31 AM, "Yoav H" wrote: >> >>> Any comment on this? >>> Will you consider this for proto3? >>> >>> On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: Hi, I have a suggestion fr improving the protobuf encoding. Is proto3 final? I like the simplicity of the encoding of protobuf. But I think it has one issue with serialization, using streams. The problem is with length delimited fields and the fact that they require knowing the length ahead of time. If we have a very long string, we need to encode the entire string before we know its length, so we basically duplicate the data in memory. Same is true for embedded messages, where we need to encode the entire embedded message before we can append it to the stream. I think there is a simple solution for both issues. For strings and byte arrays, a simple solution is to use "chunked encoding". Which means that the byte array is split into chunks and every chunk starts with the chunk length. End of array is indicated by length zero. For embedded messages, the solution is to have an "start embedding" tag and an "end embedding tag". Everything in between is the embedded message. By adding these two new features, serialization can be fully streamable and there is no need to pre-serialize big chunks in memory before writing them to the stream. Hope you'll find this suggestion useful and incorporate it into the protocol. Thanks, Yoav. -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to protobuf+u...@googlegroups.com. >>> To post to this group, send email to prot...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@g
[protobuf] Protobuf - Creating the ProtoBuf object from a physical File
Hi,
I have a requirement :
1. I have converted Example.proto file to Example.cs file code using the
protoc.exe command.
2. I want to read Example.cs file using FileStream and create the Example
object or atleast the TypeOf. I do not want to include it
in my project as I want to make it generic.
Could you please let me know how to achieve this.
Thanks & Regards
Raghu
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: Example.proto
#pragma warning disable 1591, 0612, 3021
#region Designer generated code
using pb = global::Google.Protobuf;
using pbc = global::Google.Protobuf.Collections;
using pbr = global::Google.Protobuf.Reflection;
using scg = global::System.Collections.Generic;
namespace ConsoleApplication1 {
/// Holder for reflection information generated from Example.proto
[global::System.Diagnostics.DebuggerNonUserCodeAttribute()]
public static partial class ExampleReflection {
#region Descriptor
/// File descriptor for Example.proto
public static pbr::FileDescriptor Descriptor {
get { return descriptor; }
}
private static pbr::FileDescriptor descriptor;
static ExampleReflection() {
byte[] descriptorData = global::System.Convert.FromBase64String(
string.Concat(
"Cg1FeGFtcGxlLnByb3RvEhNDb25zb2xlQXBwbGljYXRpb24xGhlnb29nbGUv",
"cHJvdG9idWYvYW55LnByb3RvIo0CCgZQZXJzb24SDAoEbmFtZRgBIAEoCRIK",
"CgJpZBgCIAEoBRINCgVlbWFpbBgDIAEoCRI2CgVwaG9uZRgEIAMoCzInLkNv",
"bnNvbGVBcHBsaWNhdGlvbjEuUGVyc29uLlBob25lTnVtYmVyEiEKA0FueRgF",
"IAEoCzIULmdvb2dsZS5wcm90b2J1Zi5BbnkaUgoLUGhvbmVOdW1iZXISDgoG",
"bnVtYmVyGAEgASgJEjMKBHR5cGUYAiABKA4yJS5Db25zb2xlQXBwbGljYXRp",
"b24xLlBlcnNvbi5QaG9uZVR5cGUiKwoJUGhvbmVUeXBlEgoKBk1PQklMRRAA",
"EggKBEhPTUUQARIICgRXT1JLEAJCFqoCE0NvbnNvbGVBcHBsaWNhdGlvbjFi",
"BnByb3RvMw=="));
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.AnyReflection.Descriptor, },
new pbr::GeneratedCodeInfo(null, new pbr::GeneratedCodeInfo[] {
new pbr::GeneratedCodeInfo(typeof(global::ConsoleApplication1.Person), global::ConsoleApplication1.Person.Parser, new[]{ "Name", "Id", "Email", "Phone", "Any" }, null, new[]{ typeof(global::ConsoleApplication1.Person.Types.PhoneType) }, new pbr::GeneratedCodeInfo[] { new pbr::GeneratedCodeInfo(typeof(global::ConsoleApplication1.Person.Types.PhoneNumber), global::ConsoleApplication1.Person.Types.PhoneNumber.Parser, new[]{ "Number", "Type" }, null, null, null)})
}));
}
#endregion
}
#region Messages
[global::System.Diagnostics.DebuggerNonUserCodeAttribute()]
public sealed partial class Person : pb::IMessage {
private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new Person());
public static pb::MessageParser Parser { get { return _parser; } }
public static pbr::MessageDescriptor Descriptor {
get { return global::ConsoleApplication1.ExampleReflection.Descriptor.MessageTypes[0]; }
}
pbr::MessageDescriptor pb::IMessage.Descriptor {
get { return Descriptor; }
}
public Person() {
OnConstruction();
}
partial void OnConstruction();
public Person(Person other) : this() {
name_ = other.name_;
id_ = other.id_;
email_ = other.email_;
phone_ = other.phone_.Clone();
Any = other.any_ != null ? other.Any.Clone() : null;
}
public Person Clone() {
return new Person(this);
}
/// Field number for the "name" field.
public const int NameFieldNumber = 1;
private string name_ = "";
public string Name {
get { return name_; }
set {
name_ = pb::Preconditions.CheckNotNull(value, "value");
}
}
/// Field number for the "id" field.
public const int IdFieldNumber = 2;
private int id_;
public int Id {
get { return id_; }
set {
id_ = value;
}
}
/// Field number for the "email" field.
public const int EmailFieldNumber = 3;
private string email_ = "";
public string Email {
get { return email_; }
set {
email_ = pb::Preconditions.CheckNotNull(value, "value");
}
}
/// Field number for the "phone" field.
public const int PhoneFieldNumber = 4;
private static readonly pb::FieldCodec _repeated_phone_codec
= pb::FieldCodec.ForMessage(34, global::ConsoleApplication1.Person.Types.PhoneNumber.Pa
[protobuf] Re: Getting the full (on disk) path to an input .proto from plugin
the compiler uses GOPATH from environment -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Streaming Serialization - Suggestion
On Wed, Mar 30, 2016 at 8:06 AM, 'Feng Xiao' via Protocol Buffers < protobuf@googlegroups.com> wrote: > > > On Mon, Mar 28, 2016 at 10:53 PM, Yoav H wrote: > >> They say on their website: "When evaluating new features, we look for >> additions that are very widely useful or very simple". >> What I'm suggesting here is both very useful (speeding up serialization >> and eliminating memory duplication) and very simple (simple additions to >> the encoding, no need to change the language). >> So far, no response from the Google guys... >> > Actually there are already a "start embedding" tag and a "end embedding" > tag in protobuf: > https://developers.google.com/protocol-buffers/docs/encoding#structure > > 3 Start group groups (deprecated) > 4 End group groups (deprecated) > > They are deprecated though. > > You mentioned it will be a performance gain, but what we experienced in > google says otherwise. For example, in a lot places we are only interested > in a few fields and want to skip through all other fields (if we are > building a proxy, or the field is simply an unknown field). The start > group/end group tag pair forces the parser to decode every single field in > the a whole group even the whole group is to be ignored after parsing, and > that's a very significant drawback. > This is definitely the use-case where delimiting makes perfect sense (proxy/middleware service that reads part of a message). The name 'protocol buffers' does kinda makes that use-case obvious. If using protobuf to simply serialize/deserialize, then start/end group would definitely benefit the streaming use-case. Shameless plug: https://github.com/protostuff/protostuff optimizes for the latter use-case and was mostly the reason it was created (java only though) > > And adding a new wire tag type to protobuf is not a simple thing. Actually > I don't think we have added any new wire type to protobuf before. There are > a lot issues to consider. For example, isn't all code that switch on > protobuf wire types now suddenly broken? if a new serializer uses the new > wire type in its output, what will happen if the parsers can't understand > it? > > Proto3 is already finalized and we will not add new wire types in proto3. > Whether to add it in proto4 depends on whether we have a good use for it > and whether we can mitigate the risks of rolling out a new wire type. > > >> >> >> On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: >>> >>> This exact suggestion has been up for discussion long time ago(years?, >>> before proto2?) >>> >>> When it comes to taking suggestions I'm only a 3rd party implementer but >>> my understanding is that the design process of protocol buffers and its >>> goals are internal to Google and they usually publish new versions of their >>> code implementing new features before you can read about them in the >>> documents. >>> On Mar 27, 2016 5:31 AM, "Yoav H" wrote: >>> Any comment on this? Will you consider this for proto3? On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: > > Hi, > > I have a suggestion fr improving the protobuf encoding. > Is proto3 final? > > I like the simplicity of the encoding of protobuf. > But I think it has one issue with serialization, using streams. > The problem is with length delimited fields and the fact that they > require knowing the length ahead of time. > If we have a very long string, we need to encode the entire string > before we know its length, so we basically duplicate the data in memory. > Same is true for embedded messages, where we need to encode the entire > embedded message before we can append it to the stream. > > I think there is a simple solution for both issues. > > For strings and byte arrays, a simple solution is to use "chunked > encoding". > Which means that the byte array is split into chunks and every chunk > starts with the chunk length. End of array is indicated by length zero. > > For embedded messages, the solution is to have an "start embedding" > tag and an "end embedding tag". > Everything in between is the embedded message. > > By adding these two new features, serialization can be fully > streamable and there is no need to pre-serialize big chunks in memory > before writing them to the stream. > > Hope you'll find this suggestion useful and incorporate it into the > protocol. > > Thanks, > Yoav. > > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com. To post to this group, send email to prot...@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. >>> -
[protobuf] Protobuf - How to create a FileDescriptor object from a Descriptor file
I created a Descriptor file from Example.proto file using the below command. .\protoc.exe --proto_path=D:\ExampleProject\ --include_imports D:\ExampleProject\Example.Proto --descriptor_set_out=D:\ExampleProject\Example.pb We want to read this file and create FileDescriptorSet object in C#. But both FileDescriptorSet and FileDescriptorProto are internal classes. Please let me know how to create FileDescriptor object from this Example.pb file. Note: We want to use C#. Thanks & Regards Raghu -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. Example.proto Description: Binary data Example.pb Description: Binary data
