I agree with this. The final paragraph is key. cr
> On Feb 6, 2015, at 9:41 AM, Thomas Rodgers <[email protected]> wrote: > > Having thought about this for a couple of more days, I want to at least take > a stab at arguing against "threadsafe" sockets - > > libzmq's thread safety guarantees, to me anyway, are very clear, unsurprising > and non-controversial - I cannot share a socket with another thread without a > full fence. > > The kinds of systems I generally build have very strict requirements on > overall latency, to the point that most of my networking IO is done through > kernel-bypass libraries and NICs that support this, for raw TCP and UDP > multicast. The latency sensitive code that does IO is in it's own thread, > with exclusive access to the NICs which are accessed via kernel bypass. > Coordination with other threads in the same process is done via inproc pair > sockets. Pair sockets + very small messages (small enough that libzmq does > not need to perform allocation) provide a very nice interface to a lock free > queue with low overhead using a single atomic CAS operation. Atomic > operations are cheap, but they are not free (~30 clocks on x86). Adding a > mutex, even one that is never contended, to the socket will essentially > triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the > message on the pipe, one atomic CAS to release the mutex). I would like to > have the option to avoid this. > > If a wrapper wants thread safe sockets to enable certain use-cases that may > be more idiomatic for the language in question, it can provide the full > fence. AZMQ <https://github.com/zeromq/azmq> does exactly this by default, > but you have the option to opt out of it. It does this because Boost Asio by > default allows it's sockets to be used from multiple threads for async IO and > I need to guard more than just exclusive access to the ZeroMQ socket a the > fence in this case. Putting a mutex inside of the libzmq socket, essentially > doubles the overhead for no gain in useful functionality and runs completely > counter to one of C and C++'s overarching principles: "don't pay for what you > don't use". > > If a class of apps really demands short lived exclusive access to a socket, > provide a pool abstraction. The pool is thread safe, obtain a socket, perform > I/O in a single thread, return the socket to the pool. > > On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier <[email protected] > <mailto:[email protected]>> wrote: > > > On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens <[email protected] > <mailto:[email protected]>> wrote: > The discussion about thread safety was quite short iirc, though that > contributor did discuss other things... at length. I merged his > "thread safe socket" change rapidly, then we reverted it after a few > days, and he disappeared. It was rather brute force and I suspect did > not work at all, it simply wrapped all accesses to the socket > structure in mutexes. No discussion at the time of multipart data and > atomic send/recv. > > My memory of the conversation at the time is pretty dim, I agree the changes > were ugly and untested and the contributor was difficult to reason with and > seemed to want to make the changes based on no real need at all. > > > As for socket safety, I've no strong opinion. I see that many people > expect that to work and hit errors when it doesn't. I see that nanomsg > has threadsafe sockets and no multipart. I see that sharing sockets > across threads would make some actor models simpler, which is nice. > > This is the classic problem with thread safe anything. Threads are hard, and > there is a balance between the complexity of making a thread safe construct > and the skill required of a programmer to use "unsafe" construct in a safe > manner. I still think if the concrete problem is very short lived threads > causing slow joiner problems, then the simple solution is pools (troupes of > actors?). > > -Michel > > > > > > > > On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier > <[email protected] <mailto:[email protected]>> wrote: > > I think Brian has some good points here, there are numerous unrelated issues > > being discussed in this thread. > > > > A few points that I have: > > > > Multi part messages have also bothered me. However as a Python programmer i > > see Min's points about the expense of buffer creation. To my knowledge > > zproto does not (yet) have Python generation support either, or maybe > > something like generated cffi or ctypes wrappers around the zproto generated > > C code. That being said there are a variety of serialization libraries for > > Python. With some ctypes and mmap magic they can also be done "zero copy" > > but it's not pretty: > > > > https://gist.github.com/michelp/7522179 > > <https://gist.github.com/michelp/7522179> > > > > Multi part envelops are also how multi-hop routing is done. I don't see how > > the new ideas handle that. I don't think we can just say "multi hop routing > > is bad" and get rid of it. > > > > "Thread safe" sockets do not sound appealing to me. We did that, had a long > > and contentious discussion with the person championing them, merged it, then > > reverted it and that person is now no longer in the community. Pieter was > > the most vocal opponent to them then and now he wants them back. Of course, > > anyone can change their mind, but the only current argument I hear now for > > them though is improving the performance of short lived threads, but that > > can be solved, more correctly in my opinion, with thread or connection > > pools. If you creating and tearing down threads that rapidly then you have > > two problems. > > > > -Michel > > > > On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <[email protected] > > <mailto:[email protected]>> wrote: > >> > >> After catching up on this thread, I feel like at least three problems are > >> being conflated into one problem. I'll state what I see being discussed > >> from my perspective: > >> > >> 1. "Using multi part messages as a way to route to clients from a router > >> socket is overly complicated and not how new users expect things to work" > >> > >> 2. "Using multi part messages for message serialization is costly, and > >> potentially confusing to others." > >> > >> 3. "ZeroMQ sockets are not thread safe." > >> > >> While on an implementation level these three problems may be related, on a > >> conceptual level I don't see them as related. I may agree with some of > >> these problem statements and not others. > >> > >> For me, my first priority is to always have the ability to get back a nice > >> agnostic blob of bytes from ZeroMQ. This makes it easy to make ZeroMQ > >> socket use compatible with standard io interfaces in Go. Structure for > >> what > >> is contained in those bytes is a concern of a different layer. Sometimes I > >> use zproto for this (which I like), and other times I don't. > >> > >> As a demonstration that the problems are different problems, I solved #1 > >> for myself in goczmq without addressing anything else. > >> > >> I would assert some of the confusion in this discussion is that we're > >> talking about multiple problem statements at the same time. > >> > >> Cheers - and it was great meeting people this week! > >> > >> Brian > >> > >> > >> > >> > >> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <[email protected] > >> <mailto:[email protected]>> wrote: > >>> > >>> Ironically, in my testing of high message rate), allowing multipart > >>> creates significant costs. Multipart is just one way of getting > >>> zero-copy, and even then only works on writing, not reading. > >>> > >>> For high performance brokers like Malamute I'd *really* like to be > >>> moving blobs around instead of lists of blobs. > >>> > >>> > >>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <[email protected] > >>> <mailto:[email protected]>> > >>> wrote: > >>> > M> Perhaps it is because I spend my days in a higher level language > >>> > M> like Python, but zproto is not an attractive option. > >>> > > >>> > Same here. I will read in detail about it shortly, but it may not make > >>> > it into my toolbox as a multipart replacement. Multipart looked very > >>> > cool when I found 0MQ, but I've ended up not using it much. I'm not > >>> > doing high performance stuff though. Simplicity and ease of use are > >>> > tops on my list. > >>> > > >>> > -- Gregg > >>> > > >>> > _______________________________________________ > >>> > zeromq-dev mailing list > >>> > [email protected] <mailto:[email protected]> > >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >>> > <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> > >>> _______________________________________________ > >>> zeromq-dev mailing list > >>> [email protected] <mailto:[email protected]> > >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >>> <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> > >> > >> > >> > >> _______________________________________________ > >> zeromq-dev mailing list > >> [email protected] <mailto:[email protected]> > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> > >> > > > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] <mailto:[email protected]> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> > > > _______________________________________________ > zeromq-dev mailing list > [email protected] <mailto:[email protected]> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> > > > _______________________________________________ > zeromq-dev mailing list > [email protected] <mailto:[email protected]> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > <http://lists.zeromq.org/mailman/listinfo/zeromq-dev> > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
