I can see why. When I finally tracked down the place it happens in dist.cpp, I realized it isn't as simple as changing a setting...the logic just shrugs if the write fails. Hmm.
On Thu, Jan 16, 2014 at 4:04 PM, Charles Remes <[email protected]> wrote: > I don’t think there is any resistance to having pluggable HWM policies. No > one has wanted this badly enough to write it. > > On Jan 16, 2014, at 2:38 PM, Lindley French <[email protected]> wrote: > > That will work, of course....I'm just curious what the resistance is to > letting the HWM policy be settable. > > > On Thu, Jan 16, 2014 at 3:27 PM, Goswin von Brederlow > <[email protected]>wrote: > >> On Thu, Jan 16, 2014 at 02:45:31PM -0500, Lindley French wrote: >> > > This is a common issue. If you can?t recover from dropped messages, >> > PUB/SUB is not the correct pattern. >> > >> > In many cases, this is correct. I do not believe inproc is one of those >> > cases, however. With inproc, you should have full control of who is >> > subscribing and when they come up relative to the publisher. If your >> > subscribers aren't running when you expect them to be running, that's a >> > bug. If they aren't running fast enough, dropping messages *might* be a >> > solution, or it might not. I don't feel that's a decision that can be >> made >> > in the general case. >> > >> > Let me put it this way: If I need one-to-many semantics with >> backpressure >> > and filtering, what should I use? PUB is the only one-to-many socket >> type. >> > I can write my own filtering code, keep a vector of push sockets, etc. >> but >> > that seems to defeat the point of ZMQ patterns. PUB is exactly what I >> want >> > in every way *except* the HWM behavior. >> > >> > >> > On Thu, Jan 16, 2014 at 2:11 PM, Charles Remes <[email protected]> >> wrote: >> > >> > > This is a common issue. If you can?t recover from dropped messages, >> > > PUB/SUB is not the correct pattern. >> > > >> > > This of PUB as a radio antenna. It broadcasts regardless of whether >> or not >> > > anyone is listening. If there are no listeners, every packet gets >> dropped. >> > > If a listener is slow, then packets will get dropped. If you need >> further >> > > guarantees about delivery, then you need to build some kind of >> protocol >> > > (ack, nak, ack window, etc) on top of DEALER/ROUTER. >> > > >> > > Also, as of libzmq 3.3, I believe the default HWM is 1000 (to prevent >> > > memory exhaustion in the default configuration). If you want >> ?infinite? >> > > then setsockopt to -1. >> > > >> > > As for the dropped messages on inproc, you need to be careful to >> confirm >> > > that a listener (SUB) is actually up, running and *connected* before >> you >> > > start PUB?ing otherwise the PUB socket will drop messages. >> Synchronization >> > > for this is discussed in the guide. Alternately, just have your PUB >> ?sleep? >> > > for a second after the SUB bind/connects and you should be okay. >> > > >> > > >> > > On Jan 16, 2014, at 1:03 PM, Lindley French <[email protected]> >> wrote: >> > > >> > > Well, aside from the router issue I do like the arrangement for easily >> > > handling different messages in different places. However, there may >> be a >> > > fatal flaw at the moment: PUB's desire to drop messages at the HWM. >> While >> > > making "drop" a default behavior for PUB is fine, I really don't like >> that >> > > it's the *only* behavior possible. >> > > >> > > Then again, that may or may not be the issue here. I haven't touched >> the >> > > HWM, so it should still be 0 which is theoretically infinite. >> Nonetheless, >> > > a bunch of my messages in a row vanished into the ether somewhere >> between >> > > PUB and SUB inproc sockets. >> > > >> > > >> > > On Thu, Jan 16, 2014 at 1:02 PM, Lindley French <[email protected] >> >wrote: >> > > >> > >> I tend to stuff in as many different features as I can when I'm first >> > >> learning something new, it helps me get a feel for it. >> > >> >> > >> You should have seen my first major python program..... >> > >> >> > >> >> > >> On Thu, Jan 16, 2014 at 12:46 PM, Charles Remes < >> [email protected]>wrote: >> > >> >> > >>> Create a socket for each worker thread and have your main thread >> resend >> > >>> the message down the appropriate socket. Sometimes it isn?t a good >> idea to >> > >>> try and shoe-horn every zeromq socket pattern into your app. :) >> > >>> >> > >>> On Jan 16, 2014, at 9:54 AM, Lindley French <[email protected]> >> wrote: >> > >>> >> > >>> > A problem I was wrestling with was, how do I deal with a TCP >> > >>> connection where messages of different types may arrive, and may >> need to be >> > >>> dealt with in different threads? The TCP socket can't be touched >> directly >> > >>> by multiple threads, of course. The obvious solution was to >> immediately >> > >>> forward messages arriving on the TCP socket to an inproc socket. >> > >>> > >> > >>> > I then took it one step further: why not make that inproc socket >> a PUB >> > >>> socket and make the first part of each message be a topic >> identifier, so >> > >>> that whichever thread knows how to deal with a particular message >> can just >> > >>> subscribe to it and ignore the rest? >> > >>> > >> > >>> > That's a great design, right up until I try to do it with the TCP >> > >>> socket being a ROUTER. Now, no matter what the first part of the >> sent >> > >>> message is, the identity will end up being the first part on the >> receiving >> > >>> end. The PUB/SUB won't work without some tweaking. >> > >>> > >> > >>> > I don't want to just drop the identity; that's useful >> information. I >> > >>> could swap the first two parts; that will work, but it's >> unintuitive and >> > >>> could cause confusion down the road. >> > >>> > >> > >>> > Any other ideas? >> >> Add a splitter that simply checks the first frame, looks up the right >> target socket and sends the remainder on the message onwards. >> Then you can use a simple PUSH/PULL pattern. >> >> MfG >> Goswin >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
