Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
sust...@250bpm.com said: > A straightforward behaviour for zmq_term would be to block until all the > sockets are deallocated -- each depending on it's SO_LINGER policy. > > (This could obviously result in deadlock on zmq_term, but that's beside > the point once again I think.) > > Thoughts? For the record, I also agree; I think there is now consensus on what the correct semantics for zmq_close() and zmq_term() should be. Good discussion! -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
i like this. and the default behaviour (depending on SO_LINGER) would more naturally follow the expected behaviour. On Jul 9, 2010, at 6:29 AM, Martin Sustrik wrote: A straightforward behaviour for zmq_term would be to block until all the sockets are deallocated -- each depending on it's SO_LINGER policy. (This could obviously result in deadlock on zmq_term, but that's beside the point once again I think.) Thoughts? Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Andrew Hume (best -> Telework) +1 732-886-1886 and...@research.att.com (Work) +1 973-360-8651 AT&T Labs - Research; member of USENIX and LOPSA ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
Hi, I would say the zmq_close behaviour should mimic Berkeley socket close behaviour, including SO_LINGER option and all the shutdown modes it provides. (One thing that strikes me is that there's no timout for trying to send the data in the default case, which just begs to be exploited in a DoS attack. But that's beside the point.) The only real problem is zmq_term behaviour. 0MQ context is meant to play the same role as OS plays for BSD sockets. Following the analogy, closing context should drop all the pending data same way as OS drops all the data pending in TCP tx buffers on shutdown. The problem with this approach is that the behaviour is non-deterministic (you don't know whether data were sent when shutting down). Non-deterministic behaviour is OK for OS shutdown, but doesn't work well for application shutdown (i.e. zmq_term). A straightforward behaviour for zmq_term would be to block until all the sockets are deallocated -- each depending on it's SO_LINGER policy. (This could obviously result in deadlock on zmq_term, but that's beside the point once again I think.) Thoughts? Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
opening and closing lots of sockets is arguably bad. agreed. not being able to close off a communication channel with deterministic semantics is bad. i don't understand the difficulty of being able to ask the publisher, that is someone who has done zmq_send's, if all those sends have been sent. On Jul 8, 2010, at 8:58 AM, Chuck Remes wrote: Flushing messages from closing sockets (or terminated contexts) is only interesting at the final stage of an application's lifecycle. Earlier in this thread someone mentioned that it's probably a buggy "pattern" if one is opening and closing lots of sockets during a program's lifetime. I agree. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Andrew Hume (best -> Telework) +1 732-886-1886 and...@research.att.com (Work) +1 973-360-8651 AT&T Labs - Research; member of USENIX and LOPSA ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Jul 8, 2010, at 6:55 AM, Matt Weinstein wrote:
> IMHO the least surprising close semantics depend on the class of socket.
>
> For example, REQ/REP or PAIR should not bother to flush, because they
> are a communicating "pair" (temporal/ry or not), and if the
> correspondent has gone away, that's probably more important.
> STREAM sockets probably should finish, PUB sockets should probably
> finish. etc.
>
> A more important is that whereas TCP is in the kernel, ØMQ is not, and
> their lifetimes are different, which means you could still close(),
> then exit() from another thread, and it would be an unpleasant
> "surprise" if you expected close() to finish sending, and it didn't.
>
> Right now, I'd say the simplest way might be to add a way of finding
> out if a socket is "busy" doing stuff you asked it to, which allows
> you to wait if you want to. You could do this e.g. by adding
> getsockopt(ZMQ_IOPEND).
>
> Then you can hang around waiting for completion:
>
> for (;;) {
> int64_t working;
> int rc = zmq_getsockopt (psocket, ZMQ_IOPEND, &working, sizeof working);
> assert (rc == 0);
> if (! working) {
> rc = zmq_close(psocket);
> assert(rc == 0);
> break;
> }
> usleep(1);
> }
>
> Obviously this can be wrapped, error checked, etc.
I think this is a reasonable suggestion. Right now there is no way to check 0mq
for queued messages, queue depth, pending I/O, etc. A mechanism to do so would
be nice.
I'm not in favor of adding more blocking behavior to the system. I prefer
everything to be asynchronous so that I can tend to other work. Blocking
behavior ruins my throughput.
Flushing messages from closing sockets (or terminated contexts) is only
interesting at the final stage of an application's lifecycle. Earlier in this
thread someone mentioned that it's probably a buggy "pattern" if one is opening
and closing lots of sockets during a program's lifetime. I agree.
cr
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
IMHO the least surprising close semantics depend on the class of socket.
For example, REQ/REP or PAIR should not bother to flush, because they
are a communicating "pair" (temporal/ry or not), and if the
correspondent has gone away, that's probably more important.
STREAM sockets probably should finish, PUB sockets should probably
finish. etc.
A more important is that whereas TCP is in the kernel, ØMQ is not, and
their lifetimes are different, which means you could still close(),
then exit() from another thread, and it would be an unpleasant
"surprise" if you expected close() to finish sending, and it didn't.
Right now, I'd say the simplest way might be to add a way of finding
out if a socket is "busy" doing stuff you asked it to, which allows
you to wait if you want to. You could do this e.g. by adding
getsockopt(ZMQ_IOPEND).
Then you can hang around waiting for completion:
for (;;) {
int64_t working;
int rc = zmq_getsockopt (psocket, ZMQ_IOPEND, &working, sizeof working);
assert (rc == 0);
if (! working) {
rc = zmq_close(psocket);
assert(rc == 0);
break;
}
usleep(1);
}
Obviously this can be wrapped, error checked, etc.
On Jul 8, 2010, at 5:38 AM, Pieter Hintjens wrote:
> On Thu, Jul 8, 2010 at 11:19 AM, Martin Lucina
> wrote:
>
>> Almost any UNIX system call can return EINTR if it's interrupted at
>> just
>> the right time, that doesn't make it a blocking call.
>
> If it does almost nothing, then blocking and non-blocking are the
> same... close() is synchronous even if it does nothing. fclose()
> flushes data and that could conceivably block exactly as I'm
> suggesting zmq_close() would block, if it flushed messages to the OS.
>
> Given that we don't have any scenarios that prove or disprove this
> solution, it's pretty irrelevant. We need to flush messages at some
> stage before exiting the process, and we need to add this without
> changing the API.
>
> Opinions from others would be welcome. Does a zmq_close() which
> flushes pending messages make sense? Or is that too surprising?
>
> I'm sure when Sustrik gets back from his week off he'll have a good
> answer to this :-)
>
> -Pieter
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Thu, Jul 8, 2010 at 11:19 AM, Martin Lucina wrote: > Almost any UNIX system call can return EINTR if it's interrupted at just > the right time, that doesn't make it a blocking call. If it does almost nothing, then blocking and non-blocking are the same... close() is synchronous even if it does nothing. fclose() flushes data and that could conceivably block exactly as I'm suggesting zmq_close() would block, if it flushed messages to the OS. Given that we don't have any scenarios that prove or disprove this solution, it's pretty irrelevant. We need to flush messages at some stage before exiting the process, and we need to add this without changing the API. Opinions from others would be welcome. Does a zmq_close() which flushes pending messages make sense? Or is that too surprising? I'm sure when Sustrik gets back from his week off he'll have a good answer to this :-) -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On 8/7/2010, "Pieter Hintjens" wrote: >On Wed, Jul 7, 2010 at 6:39 PM, Martin Lucina wrote: > >> Read up on the manuals for close() and setsockopt(). >> >> Following the principle of least surprise and consistency with standard >> sockets, zmq_close() should not be a blocking call by default since >> close() isn't either. > >The standard Linux man pages on close() and setsockopt() are not very >helpful. Do you have a URL to whatever manual page you're referring >to? You're right, the manual pages are confusing. Quoting Stevens, UNIX Network Programming Vol. 1, 3rd ed., pg., 94, which is a much better source: close Function The normal Unix close function is also used to close a socket and terminate a TCP connection. #include int close (int sockfd) ; Returns: 0 if OK, -1 on error The default action of close with a TCP socket is to mark the socket as closed and return to the process immediately. The socket descriptor is no longer usable by the process: It cannot be used as an argument to read or write. But, TCP will try to send any data that is already queued to be sent to the other end, and after this occurs, the normal TCP connection termination sequence takes place (Section 2.6). In Section 7.5, we will describe the SO_LINGER socket option, which lets us change this default action with a TCP socket. In that section, we will also describe what a TCP application must do to be guaranteed that the peer application has received any outstanding data. >[...] >Further, close() _is_ a blocking call according to the man pages I'm >looking at (which is why it may return EINTR) though it's unclear >whether close() flushes data or not. Almost any UNIX system call can return EINTR if it's interrupted at just the right time, that doesn't make it a blocking call. close() doesn't have to wait for anything to happen since it does not make any guarantees of the state of the operation after it completes (other than the fd being closed), therefore it's not a blocking call. >You're comparing TCP sockets with 0MQ sockets but it's not the right >comparison because send() is working so differently. What we're doing >is caching data in application memory and deciding on semantics for >flushing that cache. As you said yourself, the accurate comparison is >between 0MQ sockets and a file system. The general principle is the same for both a TCP socket and a file, is it not? close() tells the OS that we're done with this file/socket. Nowhere at that point do you *expect data loss as a normal consequence of calling close()*. The file analogy just makes it easier to understand. >So, does fclose() flush data? Yes, it does. Does it do a fsync to >ensure the data is actually written to disk? No, that's the OS's >problem. It's precisely the same here. Correct, but as I wrote above, close() flushes data but does not block until said data ends up anywhere. -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Wed, Jul 7, 2010 at 6:39 PM, Martin Lucina wrote: > Read up on the manuals for close() and setsockopt(). > > Following the principle of least surprise and consistency with standard > sockets, zmq_close() should not be a blocking call by default since > close() isn't either. The standard Linux man pages on close() and setsockopt() are not very helpful. Do you have a URL to whatever manual page you're referring to? However... following the principle of least surprise, send() would block and apps would not lose data just because they forgot to sleep(). We're already breaking the principle of least surprise in a major way, obviously. IMO it would be considerably less surprising to have zmq_close() flush data, than to have data loss. Further, close() _is_ a blocking call according to the man pages I'm looking at (which is why it may return EINTR) though it's unclear whether close() flushes data or not. > As I mentioned in my original email you can alter the default behaviour > of close() on sockets to be blocking, or even to do roughly what > zmq_close() does now by manipulating the SO_LINGER socket option. You're comparing TCP sockets with 0MQ sockets but it's not the right comparison because send() is working so differently. What we're doing is caching data in application memory and deciding on semantics for flushing that cache. As you said yourself, the accurate comparison is between 0MQ sockets and a file system. So, does fclose() flush data? Yes, it does. Does it do a fsync to ensure the data is actually written to disk? No, that's the OS's problem. It's precisely the same here. Still, it's moot IMO, the only difference between flushing in zmq_close() and zmq_term() will be for applications that create and destroy many sockets, which is probably a buggy pattern. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On 7/7/2010, "Dhammika Pathirana" wrote: >Not really, TCP close() just indicates "done reading and writing". But >the OS can still send a RST (ie. receiving more data for the closed >socket), and terminate the connection. Sure, which is fine. After a close() we don't want to receive any more messages sent to the 0MQ socket either. -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On 7/7/2010, "Pieter Hintjens" wrote: >On Wed, Jul 7, 2010 at 9:35 AM, Pieter Hintjens wrote: > >> Does anyone see problems with this approach? > >Sorry, for some reason Mato's original email slipped past me... > >Mato, +1 on making zmq_term a blocking call. > I'd probably make >zmq_close a blocking call as well, and offer a socket option to >disable this per socket. Read up on the manuals for close() and setsockopt(). Following the principle of least surprise and consistency with standard sockets, zmq_close() should not be a blocking call by default since close() isn't either. As I mentioned in my original email you can alter the default behaviour of close() on sockets to be blocking, or even to do roughly what zmq_close() does now by manipulating the SO_LINGER socket option. Now, I've never actually seen any code that uses SO_LINGER but if enough people want that then it could be implemented for 0MQ sockets. >Do we have use cases for applications that need to kill 0MQ right >away? I'd suggest that this can be done e.g. by destroying the >context and not calling zmq_term(). zmq_term() is the only way to destroy a context at the moment. -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On 7/7/10, Martin Lucina wrote: > > On 7/7/2010, "Dhammika Pathirana" wrote: > > >But how is this different from network or remote host queuing/dropping > >messages? > >Sending queued messages doesn't really guarantee delivery of messages. > > > Of course this change doesn't *guarantee* anything has been delivered. > But that's not the point. Imagine you're using a standard TCP socket, > and you do: > > accept (...); > write (...); > close (...); > > Surely you expect that the data you sent down that socket with the > write() should pop out the other end *if that other end is still there*? > Not really, TCP close() just indicates "done reading and writing". But the OS can still send a RST (ie. receiving more data for the closed socket), and terminate the connection. > With the current zmq_close() semantics that won't happen. Bringing this > in line with the standard close() semantics is what my change proposes. > I see, but there's this subtle TCP behavior. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Tue, Jul 6, 2010 at 7:51 PM, Martin Lucina wrote: >> And I'm looking for a way to dynamically change the number of >> concurrent user threads available for a context, so maybe it's time >> for zmq_setcontextopt()? ;-) > > Probably. get/setcontextopt() are the equivalent of sysctl() or similar in > the OS space. Adding get/setcontextopt() would be a good change IMO because it will allow us to remove that 'number of I/O threads' option on new contexts, and allow dynamic reconfiguring of the context without breaking the API each time. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Wed, Jul 7, 2010 at 9:35 AM, Pieter Hintjens wrote: > Does anyone see problems with this approach? Sorry, for some reason Mato's original email slipped past me... Mato, +1 on making zmq_term a blocking call. I'd probably make zmq_close a blocking call as well, and offer a socket option to disable this per socket. Do we have use cases for applications that need to kill 0MQ right away? I'd suggest that this can be done e.g. by destroying the context and not calling zmq_term(). -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Wed, Jul 7, 2010 at 9:23 AM, Martin Lucina wrote: > The confusion with the API changes in the 2.0.7 release and the resulting > stable release discussion came from the fact that Martin Sustrik and > myself had defined the 2.0.x series as "Beta" until such point that it > was declared "Stable". It's now obvious that the community has > declared 2.0.x "Stable", which from my point of view basically means > 2.0.x will be frozen and changes like this would go into a 2.1.x release. It was obviously time to declare 0MQ/2.0.x as Stable, given the amount of work resting on it, both as apps and as language bindings. I don't see that we need to start designing 0MQ/3.x. We're not yet finished making 2.x work properly, and the current semantics for close/term seem like unfinished work. Your analogy with writing to a file is accurate; we do not need guarantees that the data has been written but simply assurance that it's been handed off to the operating system. It seems obvious that closing a socket should clear out all messages waiting to be sent. Terminating 0MQ would do this for all open sockets. If people need the current semantics (which seem pointless but you never know) they could be requested on a per-socket basis using setsockopt(). Does anyone see problems with this approach? -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On 7/7/2010, "Peter Alexander" wrote: >Is it time to layout a road-map document and start a zeromq3 >development branch on GitHub? This should be where changes that break >backwards compatibility will go and can take shape for the next >generation of 0mq. > >I realize the following is known by everybody, but there seems to have >been some confusion at times still. > >" In principle, in subsequent releases, the major number is increased >when there are significant jumps in functionality [breakable], the >minor number is incremented when only minor features or significant >fixes have been added [non-breakable], and the revision number is >incremented when minor bugs are fixed." [1] The confusion with the API changes in the 2.0.7 release and the resulting stable release discussion came from the fact that Martin Sustrik and myself had defined the 2.0.x series as "Beta" until such point that it was declared "Stable". It's now obvious that the community has declared 2.0.x "Stable", which from my point of view basically means 2.0.x will be frozen and changes like this would go into a 2.1.x release. The reason I'm bringing this up is that I personally have the resources and motivation to maintain one "released" branch. So if these changes become 2.1.x stuff, then it's extremely unlikely that I will make any more 2.0.x releases. People are of course free to ask for more 2.0.x maintenance releases in exchange for money. -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On 7/7/2010, "Dhammika Pathirana" wrote: >But how is this different from network or remote host queuing/dropping >messages? >Sending queued messages doesn't really guarantee delivery of messages. Of course this change doesn't *guarantee* anything has been delivered. But that's not the point. Imagine you're using a standard TCP socket, and you do: accept (...); write (...); close (...); Surely you expect that the data you sent down that socket with the write() should pop out the other end *if that other end is still there*? With the current zmq_close() semantics that won't happen. Bringing this in line with the standard close() semantics is what my change proposes. If you need a better example, imagine what you're writing to is a file and not a socket :-) -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
But how is this different from network or remote host queuing/dropping
messages?
Sending queued messages doesn't really guarantee delivery of messages.
This gets even worse as TCP sends RST (ECONNRESET) on receiving data
to a closed socket. In http world they work around this by sender
doing a half close, receiver reading EOF and closing its end.
On 7/6/10, Martin Lucina wrote:
> Hi all,
>
> while implementing a 0MQ architecture which needs to dynamically create and
> destroy sockets during operation I ran into the current behaviour of
> zmq_close() being semantically different from the standard close() system
> call.
>
> Consider a scenario where we wish to send a bunch of messages, then close
> the socket:
>
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_close (s)
>
> The current behaviour is that zmq_close() will discard any messages which
> have been queued ("sent") with zmq_send() but have not yet been pushed out
> to the network. Contrast this with the behaviour of the close() system call
> on a standard socket where the call means "please make this socket go away,
> but finish sending any outstanding data on it asynchronously if you
> can"[1].
>
> In my opinion the proper solution is to use the same semantics as the
> close() system call, in other words, zmq_close() shall invalidate the
> socket from the caller's point of view so no further operations may be
> performed on it, but 0MQ shall send any outstanding messages in the
> background *as long as a endpoint for those messages still exists* before
> destroying the socket "for real".
>
> This would mean a second change to the API which would make zmq_term() a
> blocking call, since it would need to wait until all outstanding messages
> are sent. The analogous functionality for the close() system call is
> handled by the OS kernel -- obviously if the OS shuts down then data will
> be lost.
>
> The downside is that zmq_term() could freeze for an arbitrary amount of
> time if the remote end is "stuck". For applications where this is
> undesirable it would mean adding a "KILL" flag or separate zmq_term_kill()
> function which means "we don't care, really go away now".
>
> Please let me know your opinions on this change; ultimately I think it's
> the right way to go especially if OS integration of 0MQ sockets is (a long
> way) down the road.
>
> -mato
>
> [1] This behaviour can be changed using the SO_LINGER option, we'd probably
> want to implement a similar option for 0MQ sockets.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
Hi Martin On Tue, Jul 6, 2010 at 1:51 PM, Martin Lucina wrote: > matt_weinst...@yahoo.com said: >> > In my opinion the proper solution is to use the same semantics as the >> > close() system call, in other words, zmq_close() shall invalidate the >> > socket from the caller's point of view so no further operations may be >> > performed on it, but 0MQ shall send any outstanding messages in the >> > background *as long as a endpoint for those messages still exists* >> > before >> > destroying the socket "for real". >> > >> Would this be logical to implement as a new zmq_setsockopt() option? > > Ultimately the semantics will *have* to change if 0MQ sockets would be > integrated into the OS. > > If your primary priority is backward compatibility, then yes, the "new" > behaviour would have to become a socket option. I'm not convinced that > keeping incorrect behaviour for the sake of backward compatibility is a > good idea and my view is that the current behaviour is definitely incorrect > in the long run. Is it time to layout a road-map document and start a zeromq3 development branch on GitHub? This should be where changes that break backwards compatibility will go and can take shape for the next generation of 0mq. I realize the following is known by everybody, but there seems to have been some confusion at times still. " In principle, in subsequent releases, the major number is increased when there are significant jumps in functionality [breakable], the minor number is incremented when only minor features or significant fixes have been added [non-breakable], and the revision number is incremented when minor bugs are fixed." [1] [1] http://en.wikipedia.org/wiki/Software_versioning > >> >> > This would mean a second change to the API which would make >> > zmq_term() a >> > blocking call, since it would need to wait until all outstanding >> > messages >> > are sent. The analogous functionality for the close() system call is >> > handled by the OS kernel -- obviously if the OS shuts down then data >> > will >> > be lost. >> >> And I'm looking for a way to dynamically change the number of >> concurrent user threads available for a context, so maybe it's time >> for zmq_setcontextopt()? ;-) > > Probably. get/setcontextopt() are the equivalent of sysctl() or similar in > the OS space. > > -mato > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
Oh, I forgot the mention another point about this.
I have started to put time.sleep(?) calls at the end of my zmq
applications because of this. But, this has a huge problem that is
almost impossible to fix with the current API. The amount of time I
need to sleep before calling zmq_term is unpredictable and depends on
things like:
* This number of outstanding messages.
* The size of the outstanding messages.
There isn't really any way of knowing these things in general, so if
my application will have lots of large messages, I have to put in
*really* long sleeps before shutting down. I would much rather have
zmq_term block for exactly the right amount of time.
Cheers,
Brian
On Tue, Jul 6, 2010 at 10:24 AM, Martin Lucina wrote:
> Hi all,
>
> while implementing a 0MQ architecture which needs to dynamically create and
> destroy sockets during operation I ran into the current behaviour of
> zmq_close() being semantically different from the standard close() system
> call.
>
> Consider a scenario where we wish to send a bunch of messages, then close
> the socket:
>
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_close (s)
>
> The current behaviour is that zmq_close() will discard any messages which
> have been queued ("sent") with zmq_send() but have not yet been pushed out
> to the network. Contrast this with the behaviour of the close() system call
> on a standard socket where the call means "please make this socket go away,
> but finish sending any outstanding data on it asynchronously if you
> can"[1].
>
> In my opinion the proper solution is to use the same semantics as the
> close() system call, in other words, zmq_close() shall invalidate the
> socket from the caller's point of view so no further operations may be
> performed on it, but 0MQ shall send any outstanding messages in the
> background *as long as a endpoint for those messages still exists* before
> destroying the socket "for real".
>
> This would mean a second change to the API which would make zmq_term() a
> blocking call, since it would need to wait until all outstanding messages
> are sent. The analogous functionality for the close() system call is
> handled by the OS kernel -- obviously if the OS shuts down then data will
> be lost.
>
> The downside is that zmq_term() could freeze for an arbitrary amount of
> time if the remote end is "stuck". For applications where this is
> undesirable it would mean adding a "KILL" flag or separate zmq_term_kill()
> function which means "we don't care, really go away now".
>
> Please let me know your opinions on this change; ultimately I think it's
> the right way to go especially if OS integration of 0MQ sockets is (a long
> way) down the road.
>
> -mato
>
> [1] This behaviour can be changed using the SO_LINGER option, we'd probably
> want to implement a similar option for 0MQ sockets.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgran...@calpoly.edu
elliso...@gmail.com
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
Martin,
On Tue, Jul 6, 2010 at 10:24 AM, Martin Lucina wrote:
> Hi all,
>
> while implementing a 0MQ architecture which needs to dynamically create and
> destroy sockets during operation I ran into the current behaviour of
> zmq_close() being semantically different from the standard close() system
> call.
I have run into this issue as well.
> Consider a scenario where we wish to send a bunch of messages, then close
> the socket:
>
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_close (s)
>
> The current behaviour is that zmq_close() will discard any messages which
> have been queued ("sent") with zmq_send() but have not yet been pushed out
> to the network. Contrast this with the behaviour of the close() system call
> on a standard socket where the call means "please make this socket go away,
> but finish sending any outstanding data on it asynchronously if you
> can"[1].
The other issue with the current API is that it is non-deterministic.
Depending on the timing of when zmq_close is called, the messages
may or may not get sent.
> In my opinion the proper solution is to use the same semantics as the
> close() system call, in other words, zmq_close() shall invalidate the
> socket from the caller's point of view so no further operations may be
> performed on it, but 0MQ shall send any outstanding messages in the
> background *as long as a endpoint for those messages still exists* before
> destroying the socket "for real".
+1
> This would mean a second change to the API which would make zmq_term() a
> blocking call, since it would need to wait until all outstanding messages
> are sent. The analogous functionality for the close() system call is
> handled by the OS kernel -- obviously if the OS shuts down then data will
> be lost.
+1
> The downside is that zmq_term() could freeze for an arbitrary amount of
> time if the remote end is "stuck". For applications where this is
> undesirable it would mean adding a "KILL" flag or separate zmq_term_kill()
> function which means "we don't care, really go away now".
>
> Please let me know your opinions on this change; ultimately I think it's
> the right way to go especially if OS integration of 0MQ sockets is (a long
> way) down the road.
I think this would be a great change in the API.
Cheers,
Brian
> -mato
>
> [1] This behaviour can be changed using the SO_LINGER option, we'd probably
> want to implement a similar option for 0MQ sockets.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgran...@calpoly.edu
elliso...@gmail.com
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
matt_weinst...@yahoo.com said: > > In my opinion the proper solution is to use the same semantics as the > > close() system call, in other words, zmq_close() shall invalidate the > > socket from the caller's point of view so no further operations may be > > performed on it, but 0MQ shall send any outstanding messages in the > > background *as long as a endpoint for those messages still exists* > > before > > destroying the socket "for real". > > > Would this be logical to implement as a new zmq_setsockopt() option? Ultimately the semantics will *have* to change if 0MQ sockets would be integrated into the OS. If your primary priority is backward compatibility, then yes, the "new" behaviour would have to become a socket option. I'm not convinced that keeping incorrect behaviour for the sake of backward compatibility is a good idea and my view is that the current behaviour is definitely incorrect in the long run. > > > This would mean a second change to the API which would make > > zmq_term() a > > blocking call, since it would need to wait until all outstanding > > messages > > are sent. The analogous functionality for the close() system call is > > handled by the OS kernel -- obviously if the OS shuts down then data > > will > > be lost. > > And I'm looking for a way to dynamically change the number of > concurrent user threads available for a context, so maybe it's time > for zmq_setcontextopt()? ;-) Probably. get/setcontextopt() are the equivalent of sysctl() or similar in the OS space. -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_close() semantics and handling outstanding messages
On Jul 6, 2010, at 1:24 PM, Martin Lucina wrote:
> Hi all,
>
> while implementing a 0MQ architecture which needs to dynamically
> create and
> destroy sockets during operation I ran into the current behaviour of
> zmq_close() being semantically different from the standard close()
> system
> call.
>
> Consider a scenario where we wish to send a bunch of messages, then
> close
> the socket:
>
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_send (s, ...)
> zmq_close (s)
>
> The current behaviour is that zmq_close() will discard any messages
> which
> have been queued ("sent") with zmq_send() but have not yet been
> pushed out
> to the network. Contrast this with the behaviour of the close()
> system call
> on a standard socket where the call means "please make this socket
> go away,
> but finish sending any outstanding data on it asynchronously if you
> can"[1].
>
> In my opinion the proper solution is to use the same semantics as the
> close() system call, in other words, zmq_close() shall invalidate the
> socket from the caller's point of view so no further operations may be
> performed on it, but 0MQ shall send any outstanding messages in the
> background *as long as a endpoint for those messages still exists*
> before
> destroying the socket "for real".
>
Would this be logical to implement as a new zmq_setsockopt() option?
> This would mean a second change to the API which would make
> zmq_term() a
> blocking call, since it would need to wait until all outstanding
> messages
> are sent. The analogous functionality for the close() system call is
> handled by the OS kernel -- obviously if the OS shuts down then data
> will
> be lost.
And I'm looking for a way to dynamically change the number of
concurrent user threads available for a context, so maybe it's time
for zmq_setcontextopt()? ;-)
>
> The downside is that zmq_term() could freeze for an arbitrary amount
> of
> time if the remote end is "stuck". For applications where this is
> undesirable it would mean adding a "KILL" flag or separate
> zmq_term_kill()
> function which means "we don't care, really go away now".
>
> Please let me know your opinions on this change; ultimately I think
> it's
> the right way to go especially if OS integration of 0MQ sockets is
> (a long
> way) down the road.
>
> -mato
>
> [1] This behaviour can be changed using the SO_LINGER option, we'd
> probably
> want to implement a similar option for 0MQ sockets.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
