On 05/10/2012 07:52 AM, Evan Jones wrote:
On May 9, 2012, at 15:26 , Jeremy Stribling wrote:
* There are two nodes, 1 and 2, running version A of the software.
* They exchange messages containing protobuf P, which contains a string field F.
* We write a new version B of the software, which changes field F to an integer
as an optimization.
* We upgrade node 1, but node 2.
* If node 1 sends a protobuf P to node 2, I want node 2 to be able to access
field F as a string, even though the wire format sent by node 1 was an integer.
I think you can achieve your goals by building a layer on top of the existing protocol
buffer parsing, possibly in combination with some custom options, a protoc plugin, and
maybe a small tweak to the existing C++ code generator. You do the breaking change by
effectively "renaming" the field, then using a protoc plugin to make it
invisible to the application. To make this concrete, your Version A looks like:
message P {
optional string F = 1;
}
Then Version B looks like the following:
message P {
optional string old_F = 1 [(custom_upgrade_option) =
"some_upgrade_code"];
optional int32 F = 2;
}
With this structure, Version B can always parse a Version A message. Senders will always
ensure there is only one version in the message, so the only thing you are
"losing" here is a field number, which isn't a huge deal. However, you but now
want to automatically convert old_F to F. This can be done without changing the guts of
the parser by writing a protoc plugin that generates a member function based on the
custom option:
void UpgradeToLatest() {
if (has_old_F()) {
set_F(some_upgrade_code(get_old_F()));
clear_old_F();
}
}
You then need to make sure that Version B of the software calls this everywhere it is
needed. Maybe this argues that what is needed is a "post-processing" insertion
point in ::MergePartialFromCodedStream? Then your protoc plugin could insert this call
after a protocol buffer message is successfully parsed, so the application would only
ever have to deal with the integer version.
Yep, I think something like that could work. Thanks, I'll have to
explore how best to add a post-processing insertion point there, if we
decide to go that route.
In the other direction, I don't understand how the downgrading can possibly be
done at the receiver, since it doesn't know how to do the downgrade (unless you
are thinking about mobile code?). So in your example, Node 1 must create a
Version A protocol buffer message when sending to Node 2. This means you need
*some* sort of handshaking between Node 1 and Node 2, to indicate supported
versions.
This is reason I proposed adding some other member function that takes a
"target_version", so the sender knows what to emit. If sending the same message
to multiple recipients, you'll need to send the lowest version in the group. Based on the
above, your plugin could emit:
void DowngradeToVersion(int target_version) {
if (target_version< 0xB&& has_F()) {
set_old_F(some_downgrade_code(get_F()));
clear_F();
}
}
There are many other ways you could do this, but it seems to me that this
proposal is a way to do it without complicating the base protocol buffers
library with application-specific details.
Downgrading at the sender is not an option, because the "sender" might
be writing something to persistent storage that can be read by any
version of the program -- there might be no direct connection over which
to relay versions. It is possible to do the downgrading at the receiver
by having two separate processes, likely connected over a local socket
-- one that holds the main logic of your program, and one which is
responsible only for translation. Then, as part of your upgrade, you
can first upgrade the translation program separately on all nodes, so
they know how to downgrade from newer versions of the data. This
upgrade would be easy, and completely non-disruptive to the main logic
process. After all translation programs in the system have been
upgraded, you can start the (possibly long) process of upgrading the
other processes, one by one, without worrying much about the effect they
have on the non-upgraded nodes. As long as there's a stable interface
between the two processes that can withstand restarts at either end,
this should be possible. This is what's described in Sameer's thesis.
So the challenge I'm pondering is how to plug in calls to such a program
from somewhere in the protobuf processing path, for only the case where
the incoming message's version is not natively supported by the
program. Perhaps, as you suggest, a post-processing insertion point in
MergePartialFromCodedStream is the right way to go. I'll report back if
I make any progress on this.
Jeremy
--
You received this message because you are subscribed to the Google Groups "Protocol
Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
