I'm working on a project to upgrade- and downgrade-proof a distributed
system that uses protobufs to communicate data between instances of a C
++ program.  I'm trying to cover all possible cases for data schema
changes between versions of my programs, and I was hoping to get some
insight from the community on what the best practice is for the
following tricky scenario.

To reduce serialization type and protobuf message size, the format of
a field in a message is changed between incompatible types.  For
example, a string field gets changed to an int, or perhaps a field
gets changed from one message type to another.  Because this is being
done as an optimization, it makes no sense to keep both versions of
the data around, so I think whether we change the field ID is not
relevant -- we only ever want to have one version of the field in any
particular protobuf.

Of course, this makes communicating between versions of the program
very difficult, and I think it requires there to be some kind of
translator code to transform the field from one format to the other.
Ideally, this transformation would be invisible to the rest of the
program.  One ugly thought I had was to have a version field in every
message, and then in the autogenerated C++ serialize code, maybe in
MergePartialCodedFromStream, I could insert a call to an external
translator program that would transform the input bytes into something
that could be decoded by the version of the message expected by this
instance of the program.  I don't think there's an insertion point
defined for this part of the code, so I'd have to write my own script
to do it.  The external translator program could be upgraded
independently of the main program, so older versions would know how to
intepret the fields of the newer versions.

I'm wondering if anyone has experience with a scenario like this, and
if there's a more elegant way to solve it.  If not, what do folks
think of this business of an external translator program?  Foolish
nonsense?  Worthy of a proper insertion point?



You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
For more options, visit this group at 

Reply via email to