[protobuf] How realistic are benchmarks such as NorthWind ?
I saw that ProtoBuf has been benchmarked using the Northwind data set- a data set of size 130K, with 3000 objects including orders and order line items. This is an excellent review: http://code.google.com/p/protobuf-net/wiki/Performance Is it not more realistic, to have a benchmark with a much larger file, in which we are interested only in a few records, and a few fields within those records. For example: 10,000 order line items, we want only a line item with a particular product code. Or we want to pick orders for a particular customer type, or with a particular description. Are there use cases where data is stored in Protocol Buffer Format in a file, and read into memory? Another issue is that the size seems rather small- it is only 256 bytes per object,- I would imagine there are many use cases where the objects are much bigger. Many use cases are going to be with much larger objects and will select m out N fields- where m will be 5 and N will be 20. This is because very rarely can an application want all of the information in a protocol buffer generated by another program. Any comments? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Nested messages accessor methods
Hi,
Given the .proto file as below:
message NestedObject {
required string field = 1;
}
message OuterObject {
required NestedObject nestedObject = 1;
}
Question: Can one define the nested object within the OuterObject as
required?
If you can use required, what will the getters/setters methods be?
I have tried with the required field but then no getters/setters are
generated.
Thanks
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How realistic are benchmarks such as NorthWind ?
Firstly, I must note that those benchmarks are specific to protobuf-net (a specific implementation), not protocol buffers (which covers a range of implementations). Re is it not more realistic; well, that depends entirely on what your use-case *is*. It /sounds/ like you are really talking about querying ad-hoc data; if so a file-based database may be more appropriate. But it depends entirely on your scenario. It /would/ be possible (with protobuf-net at least; I can't comment beyond that) to construct a type that represents the data that you *are* interested in - the other fields would be quietly dropped without having to fully process them, avoiding some CPU. Likewise, it is possible to read items in a non-buffered way (i.e. you only have 1 object directly available in memory; any others are discarded immediately, available for GC). However; again - it sounds like you *really* want a database. Which protocol buffers isn't. Marc Gravell On 14 May 2010 11:31, Kevin Apte- SOA and Cloud Computing Architect technicalarchitect2...@gmail.com wrote: I saw that ProtoBuf has been benchmarked using the Northwind data set- a data set of size 130K, with 3000 objects including orders and order line items. This is an excellent review: http://code.google.com/p/protobuf-net/wiki/Performance Is it not more realistic, to have a benchmark with a much larger file, in which we are interested only in a few records, and a few fields within those records. For example: 10,000 order line items, we want only a line item with a particular product code. Or we want to pick orders for a particular customer type, or with a particular description. Are there use cases where data is stored in Protocol Buffer Format in a file, and read into memory? Another issue is that the size seems rather small- it is only 256 bytes per object,- I would imagine there are many use cases where the objects are much bigger. Many use cases are going to be with much larger objects and will select m out N fields- where m will be 5 and N will be 20. This is because very rarely can an application want all of the information in a protocol buffer generated by another program. Any comments? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- Regards, Marc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Detect type of message
Hi,
If I want to serialize many message in the same pipe,
do I need one master message that includes all the others in
optional
and have a kind of messageType field in it to know which kind a
message is inside?
Like this,
message A1 {...}
message A2 {...}
message master
{
enum type {...}
required type t = 1;
optional A1 a1 = 2;
optional A2 a2 = 3;
}
Thanks,
Vince
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Detect type of message
Check out the docs:
http://code.google.com/apis/protocolbuffers/docs/techniques.html#union
On Fri, May 14, 2010 at 12:45 PM, Vince vincedup...@gmail.com wrote:
Hi,
If I want to serialize many message in the same pipe,
do I need one master message that includes all the others in
optional
and have a kind of messageType field in it to know which kind a
message is inside?
Like this,
message A1 {...}
message A2 {...}
message master
{
enum type {...}
required type t = 1;
optional A1 a1 = 2;
optional A2 a2 = 3;
}
Thanks,
Vince
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Nested messages accessor methods
Yes, you can. What is your target language? The accessor names are generated
differently - in C++ you would get lower-cased names. Have you looked at the
generated header? You may also want to see
http://code.google.com/apis/protocolbuffers/docs/style.html
On Fri, May 14, 2010 at 1:29 AM, Jacques jacques.de...@gmail.com wrote:
Hi,
Given the .proto file as below:
message NestedObject {
required string field = 1;
}
message OuterObject {
required NestedObject nestedObject = 1;
}
Question: Can one define the nested object within the OuterObject as
required?
If you can use required, what will the getters/setters methods be?
I have tried with the required field but then no getters/setters are
generated.
Thanks
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] How to get rid of metadata only extensions dependencies
So, let's say I'm developing a plugin which, say, add or removes builder
pattern to java generator (this is actually a real use case). So I created a
.proto file with my plugin metadata info:
MyPluginMetadata.proto:
import google/protobuf/descriptor.proto
extend google.protobuf.MessageOptions {
optional bool use_builder_pattern = 50001;
}
Now I can just do:
MyAppMessages.proto:
import path/to/MyPluginMetadata.proto
message Foo {
option (use_builder_pattern) = true;
}
Now, when I ask protoc to generate code for MyAppMessages.proto, I
get MyAppMessages.java which depends on MyPluginMetadata.java, even
though MyPluginMetadata.proto solely provides metadata extensions. Is there
a way to avoid that?
Thanks,
-Gatis
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Detect type of message
I believe that's the way to do it.
I think you can also use the extends, but looks like other language
ports (like the C port) do not all support the extend mechanism.
( yet).
-john
On May 14, 12:45 pm, Vince vincedup...@gmail.com wrote:
Hi,
If I want to serialize many message in the same pipe,
do I need one master message that includes all the others in
optional
and have a kind of messageType field in it to know which kind a
message is inside?
Like this,
message A1 {...}
message A2 {...}
message master
{
enum type {...}
required type t = 1;
optional A1 a1 = 2;
optional A2 a2 = 3;
}
Thanks,
Vince
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Can I add a new item into an existing list of enumerations safely?
Hi
This is my first attempt at defining a command pattern :
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
}
enum Type { POP = 0, PUSH = 1 }
I deploy code using this file.
Later, I want to add a new command, but not break deployed clients.
Can I do so safely:
enum Type { POP = 0, PUSH = 1, TOP = 2 }
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
optional Top top = 5;
}
If not, what's the suggested way for achieving something like this?
I read up on 'extend' - would that be a better way? My concern is that
the other language ports (The C one, for example) do not yet support
extend.
thanx
-john
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Can I add a new item into an existing list of enumerations safely?
Clients with the old binary will not be able to read messages containing the
new enum value. This would get treated as an unknown field, meaning that the
type field would not be sent in an older client. Since your field is also
required, it would fail to parse. So, with this example, you would need to
make sure that all clients are updated before sending messages containing
the enum value.
The solution would be to use optional fields, and define an UNKNOWN enum
value.
e.g.:
enum Type { POP = 0, PUSH = 1, UNKNOWN = 2 } // set unknown = 2 for
backward compatibility, assuming POP and PUSH are already defined and in the
wild.
message Command {
optional Type type = 1 [default = UNKNOWN];
...
}
When you add TOP, you can start sending messages with that value before all
clients are updated. The message will now parse successfully on old clients,
who will get UNKNOWN when they access type().
On Fri, May 14, 2010 at 3:07 PM, john mani john.m...@gmail.com wrote:
Hi
This is my first attempt at defining a command pattern :
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
}
enum Type { POP = 0, PUSH = 1 }
I deploy code using this file.
Later, I want to add a new command, but not break deployed clients.
Can I do so safely:
enum Type { POP = 0, PUSH = 1, TOP = 2 }
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
optional Top top = 5;
}
If not, what's the suggested way for achieving something like this?
I read up on 'extend' - would that be a better way? My concern is that
the other language ports (The C one, for example) do not yet support
extend.
thanx
-john
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How to get rid of metadata only extensions dependencies
No, there isn't any way to remove the dependency. The compiler doesn't try
to detect why you defined an import.
On Fri, May 14, 2010 at 2:38 PM, Igor Gatis igorga...@gmail.com wrote:
So, let's say I'm developing a plugin which, say, add or removes builder
pattern to java generator (this is actually a real use case). So I created a
.proto file with my plugin metadata info:
MyPluginMetadata.proto:
import google/protobuf/descriptor.proto
extend google.protobuf.MessageOptions {
optional bool use_builder_pattern = 50001;
}
Now I can just do:
MyAppMessages.proto:
import path/to/MyPluginMetadata.proto
message Foo {
option (use_builder_pattern) = true;
}
Now, when I ask protoc to generate code for MyAppMessages.proto, I
get MyAppMessages.java which depends on MyPluginMetadata.java, even
though MyPluginMetadata.proto solely provides metadata extensions. Is there
a way to avoid that?
Thanks,
-Gatis
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Can I add a new item into an existing list of enumerations safely?
Thanks.
To make this generic and allow for multiple commands to be added in
future, would a good strategy be
to define:
enum Type { POP = 0, PUSH = 1, UNKNOWN = 100 }, and later add TOP,
TAIL, etc. etc. as items after PUSH (meaning with values 2, 3 ... ).
So, later we have: enum Type { POP = 0, PUSH = 1, TOP = 2, TAIL = 3,
UNKNOWN = 100 } and so on ...
Or it really does not matter?
Is this the recommended approach for this pattern? Or should I be
using extends?
-john
On May 14, 3:15 pm, Jason Hsueh jas...@google.com wrote:
Clients with the old binary will not be able to read messages containing the
new enum value. This would get treated as an unknown field, meaning that the
type field would not be sent in an older client. Since your field is also
required, it would fail to parse. So, with this example, you would need to
make sure that all clients are updated before sending messages containing
the enum value.
The solution would be to use optional fields, and define an UNKNOWN enum
value.
e.g.:
enum Type { POP = 0, PUSH = 1, UNKNOWN = 2 } // set unknown = 2 for
backward compatibility, assuming POP and PUSH are already defined and in the
wild.
message Command {
optional Type type = 1 [default = UNKNOWN];
...
}
When you add TOP, you can start sending messages with that value before all
clients are updated. The message will now parse successfully on old clients,
who will get UNKNOWN when they access type().
On Fri, May 14, 2010 at 3:07 PM, john mani john.m...@gmail.com wrote:
Hi
This is my first attempt at defining a command pattern :
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
}
enum Type { POP = 0, PUSH = 1 }
I deploy code using this file.
Later, I want to add a new command, but not break deployed clients.
Can I do so safely:
enum Type { POP = 0, PUSH = 1, TOP = 2 }
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
optional Top top = 5;
}
If not, what's the suggested way for achieving something like this?
I read up on 'extend' - would that be a better way? My concern is that
the other language ports (The C one, for example) do not yet support
extend.
thanx
-john
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Can I add a new item into an existing list of enumerations safely?
It doesn't matter that much. I only set UNKNOWN = 2 on the assumption that
pop and push were already assigned and you needed to preserve wire
compatibility. If I were starting from scratch I would probably define
UNKNOWN as the first value and set it to 0. Making it the first value
defined makes it the implicit default value - even if its integer value is
non-zero you can still do this. Mostly it depends on how you want your proto
file to look.
Whether to use this approach or extensions is up to you. This approach works
nicely with switch/case. You could do the same with extensions, but it's a
little bit more verbose. Extensions are primarily useful so that you don't
have to have a centralized definition - other users of this proto can define
their own commands as extensions. Plus there's the lack of support in other
implementations that you've mentioned.
On Fri, May 14, 2010 at 4:53 PM, john mani john.m...@gmail.com wrote:
Thanks.
To make this generic and allow for multiple commands to be added in
future, would a good strategy be
to define:
enum Type { POP = 0, PUSH = 1, UNKNOWN = 100 }, and later add TOP,
TAIL, etc. etc. as items after PUSH (meaning with values 2, 3 ... ).
So, later we have: enum Type { POP = 0, PUSH = 1, TOP = 2, TAIL = 3,
UNKNOWN = 100 } and so on ...
Or it really does not matter?
Is this the recommended approach for this pattern? Or should I be
using extends?
-john
On May 14, 3:15 pm, Jason Hsueh jas...@google.com wrote:
Clients with the old binary will not be able to read messages containing
the
new enum value. This would get treated as an unknown field, meaning that
the
type field would not be sent in an older client. Since your field is also
required, it would fail to parse. So, with this example, you would need
to
make sure that all clients are updated before sending messages containing
the enum value.
The solution would be to use optional fields, and define an UNKNOWN enum
value.
e.g.:
enum Type { POP = 0, PUSH = 1, UNKNOWN = 2 } // set unknown = 2 for
backward compatibility, assuming POP and PUSH are already defined and in
the
wild.
message Command {
optional Type type = 1 [default = UNKNOWN];
...
}
When you add TOP, you can start sending messages with that value before
all
clients are updated. The message will now parse successfully on old
clients,
who will get UNKNOWN when they access type().
On Fri, May 14, 2010 at 3:07 PM, john mani john.m...@gmail.com wrote:
Hi
This is my first attempt at defining a command pattern :
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
}
enum Type { POP = 0, PUSH = 1 }
I deploy code using this file.
Later, I want to add a new command, but not break deployed clients.
Can I do so safely:
enum Type { POP = 0, PUSH = 1, TOP = 2 }
message Command {
required Type type = 1;
optional Value value = 2;
optional Push push = 3;
optional Pop pop = 4;
optional Top top = 5;
}
If not, what's the suggested way for achieving something like this?
I read up on 'extend' - would that be a better way? My concern is that
the other language ports (The C one, for example) do not yet support
extend.
thanx
-john
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group athttp://
groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
