Yeah, it's in C/++.
On Tue, Jan 14, 2014 at 1:39 PM, Charles Remes <[email protected]> wrote: > If you are doing this from C and can access the raw memory, an inproc > socket can pass pointers around. If you are using a managed language or one > where accessing raw memory is difficult, you’ll want to figure out how to > “fake” passing a pointer (or an object reference). In your case it seems > like serializing/deserializing would be a big performance hit. That said, > if that is the direction you must go then pick something fast like msgpack > as your serializer. > > > On Jan 14, 2014, at 1:29 PM, Kenneth Adam Miller < > [email protected]> wrote: > > @AJ No, but I understand exactly why you suggested that. It's because I > haven't explained that thread 1 is doing critical work and it needs to > offload tasks to other threads as quickly as possible. > > @Lindley, Thanks so much for helping me see the truth! I was getting awful > confused considering all the different bolony that could go on if I was > stuck with semaphores, and I couldn't really re-envision it. Is there any > kind of convenience function or core utility for de-serializing the data > you receive over inproc messages? > > > On Tue, Jan 14, 2014 at 12:49 PM, AJ Lewis <[email protected]> wrote: > >> In the zeromq example, couldn't you just skip thread 1 entirely? Then the >> PULL socket from thread 2 takes uncompressed input from the source, >> compresses it, and shoves it out the PUSH socket to thread 3 for output. >> >> In this case, the PULL socket is the uncompressed pool and the PUSH socket >> is the compressed pool. Just make sure your uncompressed pool doesn't >> fill >> up faster than thread 2 can compress it, or you'll need to implement some >> logic to prevent it from using up all the memory. >> >> AJ >> >> On Tue, Jan 14, 2014 at 01:16:32PM -0500, Lindley French wrote: >> > In this case your "buffers" are really just messages, aren't they? A >> thread >> > grabs one (receives a message), processes it, and writes the result into >> > another buffer (sends a message). >> > >> > The hard part is that ZeroMQ sockets don't like to be touched by >> multiple >> > threads, which complicates the many-to-many pattern you have going here. >> > I'm no expert, but I would suggest.... >> > >> > Each "pool", A and B, becomes a single thread with two ZMQ inproc >> sockets, >> > one push and one pull. These are both bound to well-known endpoints. All >> > the thread does is continually shove messages from the pull socket to >> the >> > push socket. >> > >> > Each thread in "Thread set 1" has a push inproc socket connected to pool >> > A's pull socket. >> > >> > Each thread in "Thread set 2" has a pull inproc socket connected to pool >> > A's push socket and a push inproc socket connected to pool B's pull >> socket. >> > For each message it receives, it just processes it and spits it out the >> > other socket. >> > >> > The thread in "Thread set 3" has a pull inproc socket connected to pool >> B's >> > push socket. It just continually receives messages and outputs them. >> > >> > This may seem complicated because concepts that were distinct before >> > (buffer pools and worker threads) are now the same thing: they're both >> just >> > threads with sockets. The critical difference is that the "buffer pools" >> > bind to well-known endpoints, so you can only have a few of them, while >> the >> > worker threads connect to those well-known endpoints, so you can have as >> > many as you like. >> > >> > Will this perform as well as your current code? I don't know. Profile it >> > and find out. >> > >> > >> > On Tue, Jan 14, 2014 at 12:23 PM, Kenneth Adam Miller < >> > [email protected]> wrote: >> > >> > > So, I have two pools of shared buffers; pool A, which is a set of >> buffers >> > > of uncompressed data, and pool B, for compressed data. I three sets of >> > > threads. >> > > >> > > Thread set 1 pulls from pool A, and fills buffers it receives from >> pool A >> > > up with uncompressed data. >> > > >> > > Thread set 2 is given a pool from A that has recently been filled. It >> > > pulls a buffer from pool B, compresses from A into B, and then >> returns the >> > > buffer it was given, cleared, back to pool A. >> > > >> > > Thread set 3 is a single thread, that is continually handed compressed >> > > data from thread set 2, which it outputs. When data is finished >> output, it >> > > returns the buffer to pool B, cleared. >> > > >> > > Can anybody describe a scheme to me that will allow thread sets 1 & 2 >> to >> > > scale? >> > > >> > > Also, suppose for pools A and B, I'm using shared queues that are >> just C++ >> > > stl lists. When I pop from the front, I use a lock for removal to >> make sure >> > > that removal is deterministic. When I enqueue, I use a separate lock >> to >> > > ensure that the internals of the STL list is respected (don't want two >> > > threads receiving iterators to the same beginning node, that would >> probably >> > > corrupt the container or cause data loss, or both). Is this the >> appropriate >> > > way to go about it? Thread sets 1 & 2 will likely have more than one >> > > thread, but there's no guarantee that thread sets 1 & 2 will have >> equal >> > > threads. >> > > >> > > I was reading the ZeroMQ manual, and I read the part about >> multi-threading >> > > and message passing, and I was wondering what approaches should be >> taken >> > > with message passing when data is inherently shared between threads. >> > > >> > > _______________________________________________ >> > > zeromq-dev mailing list >> > > [email protected] >> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > >> > > >> >> > _______________________________________________ >> > zeromq-dev mailing list >> > [email protected] >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> -- >> AJ Lewis >> Software Engineer >> Quantum Corporation >> >> Work: 651 688-4346 >> email: [email protected] >> >> ---------------------------------------------------------------------- >> The information contained in this transmission may be confidential. Any >> disclosure, copying, or further distribution of confidential information is >> not permitted unless such privilege is explicitly granted in writing by >> Quantum. Quantum reserves the right to have electronic communications, >> including email and attachments, sent across its networks filtered through >> anti virus and spam software programs and retain such messages in order to >> comply with applicable data security and retention requirements. Quantum is >> not responsible for the proper and complete transmission of the substance >> of this communication or for any delay in its receipt. >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
