I know the answer is that it doesn't support this, but here's what I'm
wanting to set up to give a clearer view.
Suppose you have a set of nested messages that represent the application
state of an entire application, in this case a game and its entities and
various other stuff. Here's a trimmed down example.
message Game{
required string name = 1;
// other info
message Entity {
required int32 uid = 1;
// other stuff, position, orientation, etc
}
repeated Entity entities = 2;
}
Basically what I want to do here, to avoid having to write a bunch of
external code to track last transmitted data set, etc, is to have a
persistent instance of this Game message. At first it will be empty, and
during the update each frame, or possible more infrequently, the game will
iterate its own internal lists and compare the newest data that's in the
real game objects with the data cached in these messages. The idea here is
that these messages represent the latest state of the world that has been
sent to the network client. When the real data differs from the cached data
in the message, I update that data and serialize that to a network packet
to send, and I'm left with the last sent network state in the message
hierarchy. If the state is small enough, I may even create an empty Game
message on the stack during this process that represents the network
packet, and as I'm looping through checking dirty data, build an updated
game state message with only the changed data since the last network
transmission.
With all this in mind, there is the issue of frequent iteration of a large
nested message structure, and the cache misses that come with that. Not so
bad on x86 architecture, but pretty bad on current gaming consoles. The
other issue is with frequently hammering the dynamic memory allocation.
This part can be alleviated a lot with how its used, like not creating
temporary messages on the stack to fill in. To avoid this, I can just
update the cached messages in place, and then send those messages
individually, instead of building up a full new Game message that contains
all the diffs of the entire hierarchy. I would kinda prefer to build up the
full snapshots of the changed state, as that would greatly simplify a
number of aspects of the application. I can maintain persistant Game
message allocations simply for building the diff state, but I was hoping
there was an option to utilize the stack more for temporary messages. One
goal for this type of application(a remote debugger), especially on certain
platforms like the consoles, is for a minimal additional performance and
memory footprint, because they are often already running on the upper end
of their capabilities.
For these reasons it would be pretty useful if there was an option for the
compiler to generate statically nested variables, and max length string
buffer according to proto markup. I'll press on and depending on whether or
not it turns out to badly effect performance maybe do some compiler
modifications myself.
Thanks for the help.
On Thursday, July 19, 2012 8:22:29 AM UTC-5, Evan Jones wrote:
>
> On Jul 18, 2012, at 16:14 , Jeremy wrote:
> > I understand, but if one wants to keep a large persistent message
> allocated and walk over it frequently, there is a price to pay on cache
> misses that can be significant.
>
> I guess you are wishing that the memory layout was completely contiguous?
> Eg. if you have three string fields, that their memory would be laid out
> one field after another? Chances are good that with most dynamic memory
> allocators, if you allocate this specific sized message at one time, the
> fields will *likely* be contiguous or close to it, but obviously there are
> no guarantees. I would personally be surprised if these cache misses would
> be an important performance difference, but as normal there is only one way
> to tell: measure it.
>
> If you want something like this in protobuf though, you would need to
> change a *lot* of the internals. This would not be a simple change. I
> suggest trying to re-use a message, and seeing if the performance is
> acceptable or not. If not, you'll need to find some other serialization
> solution. Good luck,
>
> Evan
>
> --
> http://evanjones.ca/
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/protobuf/-/3FyoW8V5NooJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
