On Fri, Feb 6, 2015 at 1:08 PM, Thomas Rodgers <[email protected]> wrote:
> libzmq has had io_vec like send/receive for a while (zmq_sendiov, > zmq_recviov), there is no documentation for them however. > Yes, though with it's current implementation, using sendiov is the same as several calls to zmq_send(..., SNDMORE), making it no better or worse than making those several calls yourself. -MinRK > > On Fri, Feb 6, 2015 at 2:40 PM, Michel Pelletier < > [email protected]> wrote: > >> Something else that occurred to me today, perhaps incorrectly, is that >> there was mention that nanomsg doesn't have multipart messages, but it does >> still preserve multipart messages in a way, but supporting scatter and >> gather io vectors. Perhaps emulating this well known API pattern is the >> right approach to moving forward that still keep everyone happy and brings >> us closer to the ability to have thread safety. >> >> -Michel >> >> On Fri, Feb 6, 2015 at 12:28 PM, MinRK <[email protected]> wrote: >> >>> If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental >>> branch?) preserved multi-hop while separating routing from content by >>> making the routing id a sequence rather than a single value. That way, it >>> seems that the same push/pop behavior that happens on a ROUTER-DEALER >>> device today could still work on CLIENT-SERVER. The push/pop would be >>> happening on the routing stack instead of message content. >>> >>> If a message were something like: >>> >>> { >>> frame_t frames[]; >>> int nframes; >>> routing_id_t route[]; >>> int nroutes; >>> } >>> >>> it would seem that you could have multi-part content (selfishly, in the >>> way that affects me - messages composed of non-contiguous memory), >>> multi-hop routing, and send it all with a single `zmq_send_newfangled` >>> call, allowing future attempts at making `zmq_send_newfangled` a threadsafe >>> call. >>> >>> There may be reasons this would be super gross and horrible, but it's an >>> idea, anyway. >>> >>> -MinRK >>> >>> >>> On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers <[email protected]> >>> wrote: >>> >>>> Adding a mutex, even one that is never contended, to the socket will >>>>> essentially triple this (one atomic CAS to acquire the mutex, one atomic >>>>> CAS to put the message on the pipe, one atomic CAS to release the mutex). >>>> >>>> >>>> This is a bit of a "blue sky" view of the cost of acquiring a mutex. >>>> For the adventurous of spirit, chase down the call path of pthread_mutex >>>> sometime in GLIBC. It is substantially more involved than a single pair of >>>> 'lock; cmpxchg' instructions, but it tries really hard to make that the >>>> rough cost of the happy path. >>>> >>>> On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers <[email protected]> >>>> wrote: >>>> >>>>> Having thought about this for a couple of more days, I want to at >>>>> least take a stab at arguing against "threadsafe" sockets - >>>>> >>>>> libzmq's thread safety guarantees, to me anyway, are very clear, >>>>> unsurprising and non-controversial - I cannot share a socket with another >>>>> thread without a full fence. >>>>> >>>>> The kinds of systems I generally build have very strict requirements >>>>> on overall latency, to the point that most of my networking IO is done >>>>> through kernel-bypass libraries and NICs that support this, for raw TCP >>>>> and >>>>> UDP multicast. The latency sensitive code that does IO is in it's own >>>>> thread, with exclusive access to the NICs which are accessed via kernel >>>>> bypass. Coordination with other threads in the same process is done via >>>>> inproc pair sockets. Pair sockets + very small messages (small enough that >>>>> libzmq does not need to perform allocation) provide a very nice interface >>>>> to a lock free queue with low overhead using a single atomic CAS >>>>> operation. >>>>> Atomic operations are cheap, but they are not free (~30 clocks on x86). >>>>> Adding a mutex, even one that is never contended, to the socket will >>>>> essentially triple this (one atomic CAS to acquire the mutex, one atomic >>>>> CAS to put the message on the pipe, one atomic CAS to release the mutex). >>>>> I >>>>> would like to have the option to avoid this. >>>>> >>>>> If a wrapper wants thread safe sockets to enable certain use-cases >>>>> that may be more idiomatic for the language in question, it can provide >>>>> the >>>>> full fence. AZMQ <https://github.com/zeromq/azmq> does exactly this >>>>> by default, but you have the option to opt out of it. It does this because >>>>> Boost Asio by default allows it's sockets to be used from multiple threads >>>>> for async IO and I need to guard more than just exclusive access to the >>>>> ZeroMQ socket a the fence in this case. Putting a mutex inside of the >>>>> libzmq socket, essentially doubles the overhead for no gain in useful >>>>> functionality and runs completely counter to one of C and C++'s >>>>> overarching >>>>> principles: "don't pay for what you don't use". >>>>> >>>>> If a class of apps really demands short lived exclusive access to a >>>>> socket, provide a pool abstraction. The pool is thread safe, obtain a >>>>> socket, perform I/O in a single thread, return the socket to the pool. >>>>> >>>>> On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> The discussion about thread safety was quite short iirc, though that >>>>>>> contributor did discuss other things... at length. I merged his >>>>>>> "thread safe socket" change rapidly, then we reverted it after a few >>>>>>> days, and he disappeared. It was rather brute force and I suspect did >>>>>>> not work at all, it simply wrapped all accesses to the socket >>>>>>> structure in mutexes. No discussion at the time of multipart data and >>>>>>> atomic send/recv. >>>>>>> >>>>>> >>>>>> My memory of the conversation at the time is pretty dim, I agree the >>>>>> changes were ugly and untested and the contributor was difficult to >>>>>> reason >>>>>> with and seemed to want to make the changes based on no real need at all. >>>>>> >>>>>> >>>>>>> >>>>>>> As for socket safety, I've no strong opinion. I see that many people >>>>>>> expect that to work and hit errors when it doesn't. I see that >>>>>>> nanomsg >>>>>>> has threadsafe sockets and no multipart. I see that sharing sockets >>>>>>> across threads would make some actor models simpler, which is nice. >>>>>>> >>>>>> >>>>>> This is the classic problem with thread safe anything. Threads are >>>>>> hard, and there is a balance between the complexity of making a thread >>>>>> safe >>>>>> construct and the skill required of a programmer to use "unsafe" >>>>>> construct >>>>>> in a safe manner. I still think if the concrete problem is very short >>>>>> lived threads causing slow joiner problems, then the simple solution is >>>>>> pools (troupes of actors?). >>>>>> >>>>>> -Michel >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier >>>>>>> <[email protected]> wrote: >>>>>>> > I think Brian has some good points here, there are numerous >>>>>>> unrelated issues >>>>>>> > being discussed in this thread. >>>>>>> > >>>>>>> > A few points that I have: >>>>>>> > >>>>>>> > Multi part messages have also bothered me. However as a Python >>>>>>> programmer i >>>>>>> > see Min's points about the expense of buffer creation. To my >>>>>>> knowledge >>>>>>> > zproto does not (yet) have Python generation support either, or >>>>>>> maybe >>>>>>> > something like generated cffi or ctypes wrappers around the zproto >>>>>>> generated >>>>>>> > C code. That being said there are a variety of serialization >>>>>>> libraries for >>>>>>> > Python. With some ctypes and mmap magic they can also be done >>>>>>> "zero copy" >>>>>>> > but it's not pretty: >>>>>>> > >>>>>>> > https://gist.github.com/michelp/7522179 >>>>>>> > >>>>>>> > Multi part envelops are also how multi-hop routing is done. I >>>>>>> don't see how >>>>>>> > the new ideas handle that. I don't think we can just say "multi >>>>>>> hop routing >>>>>>> > is bad" and get rid of it. >>>>>>> > >>>>>>> > "Thread safe" sockets do not sound appealing to me. We did that, >>>>>>> had a long >>>>>>> > and contentious discussion with the person championing them, >>>>>>> merged it, then >>>>>>> > reverted it and that person is now no longer in the community. >>>>>>> Pieter was >>>>>>> > the most vocal opponent to them then and now he wants them back. >>>>>>> Of course, >>>>>>> > anyone can change their mind, but the only current argument I hear >>>>>>> now for >>>>>>> > them though is improving the performance of short lived threads, >>>>>>> but that >>>>>>> > can be solved, more correctly in my opinion, with thread or >>>>>>> connection >>>>>>> > pools. If you creating and tearing down threads that rapidly then >>>>>>> you have >>>>>>> > two problems. >>>>>>> > >>>>>>> > -Michel >>>>>>> > >>>>>>> > On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <[email protected]> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> After catching up on this thread, I feel like at least three >>>>>>> problems are >>>>>>> >> being conflated into one problem. I'll state what I see being >>>>>>> discussed >>>>>>> >> from my perspective: >>>>>>> >> >>>>>>> >> 1. "Using multi part messages as a way to route to clients from a >>>>>>> router >>>>>>> >> socket is overly complicated and not how new users expect things >>>>>>> to work" >>>>>>> >> >>>>>>> >> 2. "Using multi part messages for message serialization is >>>>>>> costly, and >>>>>>> >> potentially confusing to others." >>>>>>> >> >>>>>>> >> 3. "ZeroMQ sockets are not thread safe." >>>>>>> >> >>>>>>> >> While on an implementation level these three problems may be >>>>>>> related, on a >>>>>>> >> conceptual level I don't see them as related. I may agree with >>>>>>> some of >>>>>>> >> these problem statements and not others. >>>>>>> >> >>>>>>> >> For me, my first priority is to always have the ability to get >>>>>>> back a nice >>>>>>> >> agnostic blob of bytes from ZeroMQ. This makes it easy to make >>>>>>> ZeroMQ >>>>>>> >> socket use compatible with standard io interfaces in Go. >>>>>>> Structure for what >>>>>>> >> is contained in those bytes is a concern of a different layer. >>>>>>> Sometimes I >>>>>>> >> use zproto for this (which I like), and other times I don't. >>>>>>> >> >>>>>>> >> As a demonstration that the problems are different problems, I >>>>>>> solved #1 >>>>>>> >> for myself in goczmq without addressing anything else. >>>>>>> >> >>>>>>> >> I would assert some of the confusion in this discussion is that >>>>>>> we're >>>>>>> >> talking about multiple problem statements at the same time. >>>>>>> >> >>>>>>> >> Cheers - and it was great meeting people this week! >>>>>>> >> >>>>>>> >> Brian >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <[email protected]> >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>> Ironically, in my testing of high message rate), allowing >>>>>>> multipart >>>>>>> >>> creates significant costs. Multipart is just one way of getting >>>>>>> >>> zero-copy, and even then only works on writing, not reading. >>>>>>> >>> >>>>>>> >>> For high performance brokers like Malamute I'd *really* like to >>>>>>> be >>>>>>> >>> moving blobs around instead of lists of blobs. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin < >>>>>>> [email protected]> >>>>>>> >>> wrote: >>>>>>> >>> > M> Perhaps it is because I spend my days in a higher level >>>>>>> language >>>>>>> >>> > M> like Python, but zproto is not an attractive option. >>>>>>> >>> > >>>>>>> >>> > Same here. I will read in detail about it shortly, but it may >>>>>>> not make >>>>>>> >>> > it into my toolbox as a multipart replacement. Multipart >>>>>>> looked very >>>>>>> >>> > cool when I found 0MQ, but I've ended up not using it much. >>>>>>> I'm not >>>>>>> >>> > doing high performance stuff though. Simplicity and ease of >>>>>>> use are >>>>>>> >>> > tops on my list. >>>>>>> >>> > >>>>>>> >>> > -- Gregg >>>>>>> >>> > >>>>>>> >>> > _______________________________________________ >>>>>>> >>> > zeromq-dev mailing list >>>>>>> >>> > [email protected] >>>>>>> >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> >>> _______________________________________________ >>>>>>> >>> zeromq-dev mailing list >>>>>>> >>> [email protected] >>>>>>> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> _______________________________________________ >>>>>>> >> zeromq-dev mailing list >>>>>>> >> [email protected] >>>>>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> >> >>>>>>> > >>>>>>> > >>>>>>> > _______________________________________________ >>>>>>> > zeromq-dev mailing list >>>>>>> > [email protected] >>>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> > >>>>>>> _______________________________________________ >>>>>>> zeromq-dev mailing list >>>>>>> [email protected] >>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> zeromq-dev mailing list >>>>>> [email protected] >>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >>>>>> >>>>> >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
