[protobuf] Storing a Protocol Buffer inside another Protocol Buffer

2011-08-09 Thread Dave
Hi,

We have some legacy code, that provides an RPC library for several
services.  All RPC messages have a handler name, which is used to
determine which callback, a message should be sent to when it is
received (when a service starts up it registers 1 or more handlers).
I have been working towards replacing the marshalling/unmarshalling
code with google protocol buffers, and currently have messages like
this:

message RPCBuffer {
   required string handler = 1;
   extensions 100 to max;
}

message example {

extend RPCBuffer {
optional string name = 110;
optional int32 num = 111;
}
}

Now the legacy code behaves the same way as before, when it receives a
message it uses a ParseFrom method to construct the RPCBuffer protocol
buffer. It then gets the 'handler', and uses it to send the entire
buffer to the appropriate callback.  The callback then uses the
extensions API to access data from inside the buffer.

This works, but I can see it getting messy, as over time the different
services that extend RPCBuffer, have to make sure that they use unique
identifiers.


An alternative approach maybe this:

message RPCBuffer {
   required string handler = 1;
   required bytes messageBuffer = 2;
}

message example {
required string name = 1;
required int32 num = 2;
}

The code will still, use the handler name to determine which callback
to call, but instead of passing the entire protocol buffer, it would
pass the 'messageBuffer'.  The callback, would then construct the
'example' protocol buffer, calling ParseFrom and passing in the
messageBuffer.  The callback, can then access the data members uses
the usual google protocol buffer API.

The only problem I have with the second approach, is that there are
effectively two unmarshalls (ParseFrom() is called twice, once on
the RPCBuffer, and then once on the example buffer).   Will this
provide significant overhead in terms of memory and CPU ?

Any comments/suggestions ?

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.



Re: [protobuf] Storing a Protocol Buffer inside another Protocol Buffer

2011-08-09 Thread Daniel Wright
Both methods that you mention will work and are fairly common.  The first is
generally preferred because you're less likely to have type errors, and
debugging tools can show you more of RPCBuffer (e.g. the rpc library could
print the DebugString of the RPCBuffer in debugging mode, showing exactly
what's in the message).

In terms of performance, the second approach will make one extra string copy
of messageBuffer -- it shouldn't be a significant issue unless the code is
extremely cpu and memory sensitive.

Within Google we handle the unique id issue by using changelist numbers --
our source control system generates unique numbers for every change, so it's
pretty easy to just use those numbers as the field ids.

Daniel

On Tue, Aug 9, 2011 at 7:52 AM, Dave dave.johns...@me.com wrote:

 Hi,

 We have some legacy code, that provides an RPC library for several
 services.  All RPC messages have a handler name, which is used to
 determine which callback, a message should be sent to when it is
 received (when a service starts up it registers 1 or more handlers).
 I have been working towards replacing the marshalling/unmarshalling
 code with google protocol buffers, and currently have messages like
 this:

 message RPCBuffer {
   required string handler = 1;
   extensions 100 to max;
 }

 message example {

extend RPCBuffer {
optional string name = 110;
optional int32 num = 111;
}
 }

 Now the legacy code behaves the same way as before, when it receives a
 message it uses a ParseFrom method to construct the RPCBuffer protocol
 buffer. It then gets the 'handler', and uses it to send the entire
 buffer to the appropriate callback.  The callback then uses the
 extensions API to access data from inside the buffer.

 This works, but I can see it getting messy, as over time the different
 services that extend RPCBuffer, have to make sure that they use unique
 identifiers.


 An alternative approach maybe this:

 message RPCBuffer {
   required string handler = 1;
   required bytes messageBuffer = 2;
 }

 message example {
required string name = 1;
required int32 num = 2;
 }

 The code will still, use the handler name to determine which callback
 to call, but instead of passing the entire protocol buffer, it would
 pass the 'messageBuffer'.  The callback, would then construct the
 'example' protocol buffer, calling ParseFrom and passing in the
 messageBuffer.  The callback, can then access the data members uses
 the usual google protocol buffer API.

 The only problem I have with the second approach, is that there are
 effectively two unmarshalls (ParseFrom() is called twice, once on
 the RPCBuffer, and then once on the example buffer).   Will this
 provide significant overhead in terms of memory and CPU ?

 Any comments/suggestions ?

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.