Re: [protobuf] allocate google protobuf C++ objects on shared memory

['Adam Cozzette' via Protocol Buffers]

One thing that might help for your situation is to allocate protos on an
Arena .
Protocol buffers don't have support for C++ allocators but this is the
closest thing to that. When you create an Arena you can pass in an initial
block of memory to use, so you could potentially create a block of shared
memory and have your protos use that memory for allocations that way. I've
never heard of anyone trying to do this, though, so I can't say for sure
whether this is a good idea or not. The thing with arenas is that they
cannot free up any memory until they're destroyed, so you would have to be
careful about that as well.

As you mentioned, the other problem is that you can't control where string
fields are allocated, because the API uses std::string, and the default
allocator is built into its type. Within Google we have an option
ctype=STRING_PIECE (see here
)
that you can set on a string field so that it exposes an API based on the
raw byte array instead of std::string, but unfortunately we haven't been
able to open source it yet because that requires a fair amount of cleanup
work to happen.

On Thu, Feb 16, 2017 at 12:12 AM, tarun kumar  wrote:

> Hi protobuf owners and developers ,
>
> To give the background we have extensively used the google protobuf
>
> 1) for messaging between processes  (because its ability to serialize and
> desiralize very efficiently)
>
> 2) as well as for datamodel within application. (as its reflection
> capabilities and mapping files give us the ability to keep datamodel
> pluggable to application to high extent).
>
> For case 2 we iterate through the mapping (where we have source as queries
> to database, target as generated proto objects and mapping of database
> columns and probuf class member variables) and create maps of protobuf
> objects which works as base datmodel (reference data) for our application.
> Thus our datamodel initialization is plugged out of source code , so far so
> good.
>
> Now we want to move this entire reference data into shared memory as we
> are running multiple instances (80 of them :) ) of same application, and we
> want to have the feature where this reference data can be updated in real
> time (intraday , till now it was only initialized at start of the day and
> no change was allowed intraday) if data changes in underlined database.
>
> If the reference data is in shared memory one component can update the
> same if something is updated in underlined database and all the application
> will automatically get the updated data.
>
> Keeping these bunch of maps in shared memory is an easy task for me (using
> boost ipc shared memory segment manager) . But as map gets allocated into
> shared memory, objects inside maps and there member variables should also
> be allocated to the same shared memory. It it would be my own written
>  class as members of map the task was easy I would have used primitive data
> types int , float , char array and in most complex case
> boost:ipc:basic_string if required.
>
> (http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/allocators_
> containers.html -> example is close to what i am trying to do)
>
>
> *But in protobuf I do not find any way to represent strings as char array
> , and the only data type is string which gets generated as std::string. so
> I am stuck here as due to this I am not able to allocate these member
> strings on shared memory and thus the protbuf objects on shared memory. *
>
> It would be great if you help me on this . My protobuf objects are pretty
> simple few int, float and string members (fixed byte char array will also
> suffice.). Thanks in advance.
>
> Regards,
> Tarun Kumar
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] allocate google protobuf C++ objects on shared memory

[tarun kumar]

Hi protobuf owners and developers , 

To give the background we have extensively used the google protobuf 

1) for messaging between processes  (because its ability to serialize and 
desiralize very efficiently)

2) as well as for datamodel within application. (as its reflection 
capabilities and mapping files give us the ability to keep datamodel 
pluggable to application to high extent).

For case 2 we iterate through the mapping (where we have source as queries 
to database, target as generated proto objects and mapping of database 
columns and probuf class member variables) and create maps of protobuf 
objects which works as base datmodel (reference data) for our application. 
Thus our datamodel initialization is plugged out of source code , so far so 
good.

Now we want to move this entire reference data into shared memory as we are 
running multiple instances (80 of them :) ) of same application, and we 
want to have the feature where this reference data can be updated in real 
time (intraday , till now it was only initialized at start of the day and 
no change was allowed intraday) if data changes in underlined database. 

If the reference data is in shared memory one component can update the same 
if something is updated in underlined database and all the application will 
automatically get the updated data. 

Keeping these bunch of maps in shared memory is an easy task for me (using 
boost ipc shared memory segment manager) . But as map gets allocated into 
shared memory, objects inside maps and there member variables should also 
be allocated to the same shared memory. It it would be my own written 
 class as members of map the task was easy I would have used primitive data 
types int , float , char array and in most complex case 
boost:ipc:basic_string if required. 

(http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/allocators_containers.html
 
-> example is close to what i am trying to do)


*But in protobuf I do not find any way to represent strings as char array , 
and the only data type is string which gets generated as std::string. so I 
am stuck here as due to this I am not able to allocate these member strings 
on shared memory and thus the protbuf objects on shared memory. *

It would be great if you help me on this . My protobuf objects are pretty 
simple few int, float and string members (fixed byte char array will also 
suffice.). Thanks in advance.

Regards,
Tarun Kumar


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.