On 05/10/2012 07:52 AM, Evan Jones wrote:
On May 9, 2012, at 15:26 , Jeremy Stribling wrote:
* There are two nodes, 1 and 2, running version A of the software.
* They exchange messages containing protobuf P, which contains a string field F.
* We write a new version B of the software, which changes field F to an integer 
as an optimization.
* We upgrade node 1, but node 2.
* If node 1 sends a protobuf P to node 2, I want node 2 to be able to access 
field F as a string, even though the wire format sent by node 1 was an integer.

I think you can achieve your goals by building a layer on top of the existing protocol 
buffer parsing, possibly in combination with some custom options, a protoc plugin, and 
maybe a small tweak to the existing C++ code generator. You do the breaking change by 
effectively "renaming" the field, then using a protoc plugin to make it 
invisible to the application. To make this concrete, your Version A looks like:

message P {
        optional string F = 1;

Then Version B looks like the following:

message P {
        optional string old_F = 1 [(custom_upgrade_option) = 
        optional int32 F = 2;

With this structure, Version B can always parse a Version A message. Senders will always 
ensure there is only one version in the message, so the only thing you are 
"losing" here is a field number, which isn't a huge deal. However, you but now 
want to automatically convert old_F to F. This can be done without changing the guts of 
the parser by writing a protoc plugin that generates a member function based on the 
custom option:

void UpgradeToLatest() {
     if (has_old_F()) {

You then need to make sure that Version B of the software calls this everywhere it is 
needed. Maybe this argues that what is needed is a "post-processing" insertion 
point in ::MergePartialFromCodedStream? Then your protoc plugin could insert this call 
after a protocol buffer message is successfully parsed, so the application would only 
ever have to deal with the integer version.

Yep, I think something like that could work. Thanks, I'll have to explore how best to add a post-processing insertion point there, if we decide to go that route.

In the other direction, I don't understand how the downgrading can possibly be 
done at the receiver, since it doesn't know how to do the downgrade (unless you 
are thinking about mobile code?). So in your example, Node 1 must create a 
Version A protocol buffer message when sending to Node 2. This means you need 
*some* sort of handshaking between Node 1 and Node 2, to indicate supported 

This is reason I proposed adding some other member function that takes a 
"target_version", so the sender knows what to emit. If sending the same message 
to multiple recipients, you'll need to send the lowest version in the group. Based on the 
above, your plugin could emit:

void DowngradeToVersion(int target_version) {
     if (target_version<  0xB&&  has_F()) {

There are many other ways you could do this, but it seems to me that this 
proposal is a way to do it without complicating the base protocol buffers 
library with application-specific details.

Downgrading at the sender is not an option, because the "sender" might be writing something to persistent storage that can be read by any version of the program -- there might be no direct connection over which to relay versions. It is possible to do the downgrading at the receiver by having two separate processes, likely connected over a local socket -- one that holds the main logic of your program, and one which is responsible only for translation. Then, as part of your upgrade, you can first upgrade the translation program separately on all nodes, so they know how to downgrade from newer versions of the data. This upgrade would be easy, and completely non-disruptive to the main logic process. After all translation programs in the system have been upgraded, you can start the (possibly long) process of upgrading the other processes, one by one, without worrying much about the effect they have on the non-upgraded nodes. As long as there's a stable interface between the two processes that can withstand restarts at either end, this should be possible. This is what's described in Sameer's thesis.

So the challenge I'm pondering is how to plug in calls to such a program from somewhere in the protobuf processing path, for only the case where the incoming message's version is not natively supported by the program. Perhaps, as you suggest, a post-processing insertion point in MergePartialFromCodedStream is the right way to go. I'll report back if I make any progress on this.


You received this message because you are subscribed to the Google Groups "Protocol 
Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
For more options, visit this group at 

Reply via email to