I haven't worked with a message broker myself but I think you might find that gRPC can work well in that scenario, too. gRPC is not really tied to web services although that might be a common use case. It supports different variations <https://grpc.io/docs/guides/concepts.html#rpc-life-cycle> of streaming requests and responses, so you're not limited to the typical send-one-request-and-wait-for-the-response model. You could stream a bunch of messages to the server without having to wait for a response for each one. I'm not sure if that necessarily helps with the message broker setup but I think there's a lot of flexibility there.
On Mon, Jul 2, 2018 at 1:24 PM John Lilley <[email protected]> wrote: > Adam, > Thanks! all good suggestions. These confirm what I'm discovering on my > own. Simple framing packet containing the message name and bytes. I've > found the classes/methods for creating a concrete message given its name > and byte-buffer. So now I'm trying to find a technique for generating both > server-side and client-side code with request/response methods operating on > concrete message type. It seems like gRPC is the only "silver bullet", but > it is tied to web services. Meanwhile we're heading down a message-broker > path, so marrying these two things together is the trick. LMK if you think > of something clever or if gRPC can be tricked into using a message-queue > instead. > Cheers, > john > > > On Mon, Jul 2, 2018 at 2:05 PM Adam Cozzette <[email protected]> wrote: > >> I realized I missed the fact that in your subject line you mentioned >> reference counting. In that case it sounds like you might want to use >> std::shared_ptr instead of std::unique_ptr, but that should also work. >> >> Arenas are good for performance but come at the cost of being a little >> more difficult to work with, so if you just want to make sure the message >> is destroyed after use in a simple way then I would guess arenas aren't the >> best option. The simplest thing would be to put your message directly on >> the stack so that it gets destroyed when you return from the function, but >> of course that doesn't work if you need the message to outlive the function >> call. So if putting it on the stack doesn't work for your use case, I would >> recommend going with std::unique_ptr or std::shared_ptr. >> >> About your polymorphism question, I think a good solution would be to >> create your own class or interface and have that wrap the proto message, >> since that way your dispatcher could operate on multiple kinds of message >> without necessarily having to know the concrete message type. This >> polymorphism could handle your message->method mapping since you could >> have a common interface with a separate implementation for each message >> type. Serialized protocol buffers do not indicate their type or size, so >> you usually have to set up your own framing to at least indicate the >> message size. It's common to use a varint size prefix before the message >> payload, though that's not the only way to do it. To indicate the type of >> the message, probably the easiest approach is to have a parent message that >> contains all possible message types inside a oneof >> <https://developers.google.com/protocol-buffers/docs/proto#oneof> field. >> >> After writing all this it occurs to me that you might just want to go >> with an RPC framework like gRPC <https://grpc.io/>, since that would set >> up all the infrastructure you need for having service endpoints operating >> on protocol buffers--have you looked into gRPC by any chance? >> >> On Mon, Jul 2, 2018 at 11:19 AM John Lilley <[email protected]> wrote: >> >>> >>> Thanks Adam! >>> >>> I'll look into arena allocation, since all I really want is >>> destroy-after-use. Will that do it? >>> >>> Regarding "polymorphism", what I'm really after is some way to manage >>> and act upon a group of messages, something like: >>> 1. Get incoming packet from a queue, deserialize it and create the >>> message object. >>> 2. Send the message object (hence the need for a base class) to a >>> "dispatcher" >>> 3. Dispatcher invokes the appropriate processing method which creates a >>> response message. >>> 4. Serialize the response and send packet on return queue. >>> >>> Steps 1 and 3 are the parts that I'm grappling with, both in Java and >>> C++. Any tips on >>> -- The best way to examine the packet and create the correct message? I >>> have some freedom to put framing around the packet if necessary. >>> -- The best way to create and invoke a message->method mapping in Java / >>> C++? >>> >>> Thanks >>> john >>> >>> >>> On Fri, Jun 29, 2018 at 4:13 PM Adam Cozzette <[email protected]> >>> wrote: >>> >>>> I've found that std::unique_ptr usually works well with messages in >>>> C++. Probably the only case where things get a little bit complicated is if >>>> you're using arena allocation >>>> <https://developers.google.com/protocol-buffers/docs/reference/arenas>, >>>> since in that case the arena owns the message and you must not call delete >>>> on it. Within Google we have a special pointer type that keeps track of >>>> whether a message was arena-allocated or not, and makes sure to call delete >>>> only if the message was heap-allocated. >>>> >>>> About inheritance, there is no way to make all messages inherit from a >>>> specified class. However, all messages do inherit from >>>> google::protobuf::MessageLite (and google::protobuf::Message if you're not >>>> using the lite runtime), so that can be useful if you want to write code >>>> that operates on arbitrary message types. >>>> >>>> On Thu, Jun 28, 2018 at 12:31 PM John Lilley <[email protected]> wrote: >>>> >>>>> I have two (maybe unrelated) questions: >>>>> -- Is there a recommended way to wrap "smart pointers" around protobuf >>>>> message objects in C++? >>>>> -- Is there a way to make all messages inherit from the same base >>>>> class or interface? Note I am not trying to make message definitions >>>>> inherit from each other; I only want the classes generated C++ code to >>>>> inherit from a class that I control.' >>>>> Thanks >>>>> john >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
