Hi Michael, > I'm relatively new to 0MQ and like what I see so far. I believe in my > initial attempt to apply 0MQ to clean up some legacy socket code, I may > have uncovered a use case that 0MQ does not yet address that may be > generally useful for high-performance messaging. > > The code in question takes a few separate data structures (matrices and > associated metadata) and sends it to another process over a socket. The > most straightforward way for me to accomplish this with 0MQ is to create > a message (I know the data structure's size ahead of time), pack the > structures into contiguous memory managed by the message, and send with > the 0MQ API. The issue that I'm running into is that the matrices are > rather large, and I don't want to incur the penalty of copying them into > contiguous memory as required by the 0MQ API. > > 0MQ seems to have a mechanism to avoid excessive copying by providing > the memory for the underlying message. However, this functionality > requires the input data to be already stored in a single chunk of > contiguous memory. > > Has anyone considered implementing a message in 0MQ that would allow a > developer to send several regions of memory as a message's content? One > idea to accomplish this would be to create a separate message > constructor that took a list of memory regions (base + length) > representing the message's content. Another idea to accomplish this is > to provide a streaming interface that would allow a developer to append > to a message without having to copy data. > > What are your thoughts on this idea? If this is something that would be > generally useful to others, I'd be happy to contribute to an > investigation or development effort.
You should read the subsequent email from Brian Granger to get better idea of what issues are involved in zero-copy. In general, using zero-copy techniques from garbage collected languages is either impossible (Java?) or problematic (python). However, if what you are interested in is C/C++, the idea described above is doable. zmq_msg_t already allows storing different types of messages (VSM which is a small message stored directly in the structure vs. dynamically allocated block of memory). There's no reason why there shouldn't be a message type with multiple memory blocks. Martin _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
