Actually, which do you think would result in a better design decision? Would using message passing result in a more scalable architecture? Where I could just change a variable to increase throughput on better processors.
On Tue, Jan 14, 2014 at 3:09 PM, Kenneth Adam Miller < [email protected]> wrote: > Well, I'm just a type safety dork, so I tend to think that even losing > type information over one pointer, even if I know where that pointer is > going to end up what type it represents on the other side, is a bad thing. > Plus it's a performance thing that's just unnecessary, but I don't think > it's a big deal. These aren't objects, they are indeed raw buffers, as you > assumed. > > Also, awesome about the boost find! Appreciate you so much, you are a > beast. But I'm actually still in a sprint, so there's no version or commit > with which these hypothetical discussions directly coincide, you're helping > me get it right the first time. > > > On Tue, Jan 14, 2014 at 2:48 PM, Lindley French <[email protected]>wrote: > >> A visit to the Boost libraries reveals there's a brand-new Boost.Lockfree >> library that must have arrived with one of the last few versions. You >> should seriously consider simply replacing your std::lists with >> boost::lockfree::queues using your existing logic, and see if that gives >> you the performance you're looking for before you make any massive changes. >> >> >> On Tue, Jan 14, 2014 at 3:40 PM, Lindley French <[email protected]>wrote: >> >>> I'm going to caution you about passing pointers through inproc. It may >>> be possible to do safely, but I haven't yet figured out how to manage >>> ownership semantics in an environment where messages (pointers) can be >>> silently dropped. >>> >>> I didn't imagine serialization would be a problem since you referred to >>> "buffers"; I thought these would be raw byte buffers. If you actually mean >>> lists of objects, then yes, you'll need to serialize to use inproc. There >>> are a number of options for serialization in C++; there's >>> Boost.Serialization, Google Protobufs, a few others. You can also do it >>> manually if your objects are simple. >>> >>> Qt Signals & Slots is another solution for inter-thread communication >>> similar to inproc which has the expected C++ object semantics and therefore >>> doesn't require serialization. The downside is it's really only useful for >>> one-to-one, one-to-many, or many-to-one semantics. This covers a lot, but I >>> don't think it has a way to cover one-to-any, which is really what you want >>> (and what the zmq push socket is designed for). >>> >>> >>> On Tue, Jan 14, 2014 at 2:42 PM, Kenneth Adam Miller < >>> [email protected]> wrote: >>> >>>> Yeah, it's in C/++. >>>> >>>> >>>> On Tue, Jan 14, 2014 at 1:39 PM, Charles Remes <[email protected]>wrote: >>>> >>>>> If you are doing this from C and can access the raw memory, an inproc >>>>> socket can pass pointers around. If you are using a managed language or >>>>> one >>>>> where accessing raw memory is difficult, you’ll want to figure out how to >>>>> “fake” passing a pointer (or an object reference). In your case it seems >>>>> like serializing/deserializing would be a big performance hit. That said, >>>>> if that is the direction you must go then pick something fast like msgpack >>>>> as your serializer. >>>>> >>>>> >>>>> On Jan 14, 2014, at 1:29 PM, Kenneth Adam Miller < >>>>> [email protected]> wrote: >>>>> >>>>> @AJ No, but I understand exactly why you suggested that. It's because >>>>> I haven't explained that thread 1 is doing critical work and it needs to >>>>> offload tasks to other threads as quickly as possible. >>>>> >>>>> @Lindley, Thanks so much for helping me see the truth! I was getting >>>>> awful confused considering all the different bolony that could go on if I >>>>> was stuck with semaphores, and I couldn't really re-envision it. Is there >>>>> any kind of convenience function or core utility for de-serializing the >>>>> data you receive over inproc messages? >>>>> >>>>> >>>>> On Tue, Jan 14, 2014 at 12:49 PM, AJ Lewis <[email protected]>wrote: >>>>> >>>>>> In the zeromq example, couldn't you just skip thread 1 entirely? >>>>>> Then the >>>>>> PULL socket from thread 2 takes uncompressed input from the source, >>>>>> compresses it, and shoves it out the PUSH socket to thread 3 for >>>>>> output. >>>>>> >>>>>> In this case, the PULL socket is the uncompressed pool and the PUSH >>>>>> socket >>>>>> is the compressed pool. Just make sure your uncompressed pool >>>>>> doesn't fill >>>>>> up faster than thread 2 can compress it, or you'll need to implement >>>>>> some >>>>>> logic to prevent it from using up all the memory. >>>>>> >>>>>> AJ >>>>>> >>>>>> On Tue, Jan 14, 2014 at 01:16:32PM -0500, Lindley French wrote: >>>>>> > In this case your "buffers" are really just messages, aren't they? >>>>>> A thread >>>>>> > grabs one (receives a message), processes it, and writes the result >>>>>> into >>>>>> > another buffer (sends a message). >>>>>> > >>>>>> > The hard part is that ZeroMQ sockets don't like to be touched by >>>>>> multiple >>>>>> > threads, which complicates the many-to-many pattern you have going >>>>>> here. >>>>>> > I'm no expert, but I would suggest.... >>>>>> > >>>>>> > Each "pool", A and B, becomes a single thread with two ZMQ inproc >>>>>> sockets, >>>>>> > one push and one pull. These are both bound to well-known >>>>>> endpoints. All >>>>>> > the thread does is continually shove messages from the pull socket >>>>>> to the >>>>>> > push socket. >>>>>> > >>>>>> > Each thread in "Thread set 1" has a push inproc socket connected to >>>>>> pool >>>>>> > A's pull socket. >>>>>> > >>>>>> > Each thread in "Thread set 2" has a pull inproc socket connected to >>>>>> pool >>>>>> > A's push socket and a push inproc socket connected to pool B's pull >>>>>> socket. >>>>>> > For each message it receives, it just processes it and spits it out >>>>>> the >>>>>> > other socket. >>>>>> > >>>>>> > The thread in "Thread set 3" has a pull inproc socket connected to >>>>>> pool B's >>>>>> > push socket. It just continually receives messages and outputs them. >>>>>> > >>>>>> > This may seem complicated because concepts that were distinct before >>>>>> > (buffer pools and worker threads) are now the same thing: they're >>>>>> both just >>>>>> > threads with sockets. The critical difference is that the "buffer >>>>>> pools" >>>>>> > bind to well-known endpoints, so you can only have a few of them, >>>>>> while the >>>>>> > worker threads connect to those well-known endpoints, so you can >>>>>> have as >>>>>> > many as you like. >>>>>> > >>>>>> > Will this perform as well as your current code? I don't know. >>>>>> Profile it >>>>>> > and find out. >>>>>> > >>>>>> > >>>>>> > On Tue, Jan 14, 2014 at 12:23 PM, Kenneth Adam Miller < >>>>>> > [email protected]> wrote: >>>>>> > >>>>>> > > So, I have two pools of shared buffers; pool A, which is a set of >>>>>> buffers >>>>>> > > of uncompressed data, and pool B, for compressed data. I three >>>>>> sets of >>>>>> > > threads. >>>>>> > > >>>>>> > > Thread set 1 pulls from pool A, and fills buffers it receives >>>>>> from pool A >>>>>> > > up with uncompressed data. >>>>>> > > >>>>>> > > Thread set 2 is given a pool from A that has recently been >>>>>> filled. It >>>>>> > > pulls a buffer from pool B, compresses from A into B, and then >>>>>> returns the >>>>>> > > buffer it was given, cleared, back to pool A. >>>>>> > > >>>>>> > > Thread set 3 is a single thread, that is continually handed >>>>>> compressed >>>>>> > > data from thread set 2, which it outputs. When data is finished >>>>>> output, it >>>>>> > > returns the buffer to pool B, cleared. >>>>>> > > >>>>>> > > Can anybody describe a scheme to me that will allow thread sets 1 >>>>>> & 2 to >>>>>> > > scale? >>>>>> > > >>>>>> > > Also, suppose for pools A and B, I'm using shared queues that are >>>>>> just C++ >>>>>> > > stl lists. When I pop from the front, I use a lock for removal to >>>>>> make sure >>>>>> > > that removal is deterministic. When I enqueue, I use a separate >>>>>> lock to >>>>>> > > ensure that the internals of the STL list is respected (don't >>>>>> want two >>>>>> > > threads receiving iterators to the same beginning node, that >>>>>> would probably >>>>>> > > corrupt the container or cause data loss, or both). Is this the >>>>>> appropriate >>>>>> > > way to go about it? Thread sets 1 & 2 will likely have more than >>>>>> one >>>>>> > > thread, but there's no guarantee that thread sets 1 & 2 will have >>>>>> equal >>>>>> > > threads. >>>>>> > > >>>>>> > > I was reading the ZeroMQ manual, and I read the part about >>>>>> multi-threading >>>>>> > > and message passing, and I was wondering what approaches should >>>>>> be taken >>>>>> > > with message passing when data is inherently shared between >>>>>> threads. >>>>>> > > >>>>>> > > _______________________________________________ >>>>>> > > zeromq-dev mailing list >>>>>> > > [email protected] >>>>>> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> > > >>>>>> > > >>>>>> >>>>>> > _______________________________________________ >>>>>> > zeromq-dev mailing list >>>>>> > [email protected] >>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >>>>>> >>>>>> -- >>>>>> AJ Lewis >>>>>> Software Engineer >>>>>> Quantum Corporation >>>>>> >>>>>> Work: 651 688-4346 >>>>>> email: [email protected] >>>>>> >>>>>> ---------------------------------------------------------------------- >>>>>> The information contained in this transmission may be confidential. >>>>>> Any disclosure, copying, or further distribution of confidential >>>>>> information is not permitted unless such privilege is explicitly granted >>>>>> in >>>>>> writing by Quantum. Quantum reserves the right to have electronic >>>>>> communications, including email and attachments, sent across its networks >>>>>> filtered through anti virus and spam software programs and retain such >>>>>> messages in order to comply with applicable data security and retention >>>>>> requirements. Quantum is not responsible for the proper and complete >>>>>> transmission of the substance of this communication or for any delay in >>>>>> its >>>>>> receipt. >>>>>> _______________________________________________ >>>>>> zeromq-dev mailing list >>>>>> [email protected] >>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> zeromq-dev mailing list >>>>> [email protected] >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> zeromq-dev mailing list >>>>> [email protected] >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
