Re: [zeromq-dev] MDP protocol, detecting dead workers
Using srcfd is prolemtic, zeromq handle reconnection and the srcfd might change. To solve your problem I would change the design and continue sending heartbeat during long job and change to worker to two threads model. Alternatively you set a maximum time for a job, after which you consider the worker dead. If not dead you can handle the reconnection then. On Feb 14, 2017 17:33, "Gyorgy Szekely" wrote: > Hi, > The implemented protocol (ZMQ-RFC 7/MDP) has application level mutual > heartbeating between the broker and the worker. And this works fine: both > parties detect if the other side dies via missing heartbeats. The problem > appears when the worker is assigned a long running job, heartbeating is > _disabled_ while the job is being processed (as per 7/MDP specifies). This > enables the worker to be single threaded, and avoids typical multithreaded > issues (eg. processing thread hangs, heartbeating thread runs; worker in > inconsistent state). > > When a worker crashes during job processing my application doesn't realize > this since no messages are flowing (the broker is waiting for the job > result), but the libzmq detects this, as the socket is always closed. My > goal is to always keep in sync the number of underlying sockets in libzmq > and Worker related objects in my application. > > I've googled around and found a few libzmq features that would suit my > needs: > - ZMQ_IDENTITY_FD - this was introduced and shortly removed from the lib > - ZMQ_SRCFD - deprecated, but it's exactly what I need! > - "Peer-Address" metadata, the recommended replacement for ZMQ_SRCFD, but > not suitable for my needs > > I know fd's should be handled with care (monitor events are asynchronous, > fd's get reused), but ZMQ_SRCFD solves my problem with the following > ruleset: > 1. When a Worker registers (first message over a connection) save the > underlying fd > - and - > 2. Check that this fd is in use by another Worker, if it is: that Worker > is dead since libzmq reused its file descriptor > > 3. If a Worker's fd is in closed state for a longer period (heartbeat > expiry time), then it crashed and the fd was not re-used (get this info > from monitor) > > I don't know if this is considered as an ugly hack by hardcore zeromq > users, but it looks like a legitimate ZMQ_SRCFD use-case to me. It would be > nice if it wasn't removed in the upcoming versions. > Any feedback welcome! > > Regards, > Gyorgy > > > > On Mon, Feb 13, 2017 at 10:21 PM, Greg Young > wrote: > >> I believe the term here is application level heartbeats. >> >> It should also be supported that clients can heartbeat to server. It >> is not always that all clients want similar heartbeat timeouts. >> >> On Mon, Feb 13, 2017 at 4:07 PM, Michal Vyskocil >> wrote: >> > Hi, >> > >> > You can take inspiration from malamute broker >> > https://github.com/zeromq/malamute >> > >> > There clients pings server regularly. The same does MQTT (just it's a >> > server, who pings clients). >> > >> > Sadly malamute is vulnerable to the same problem, that received service >> > request may get lost. Solution would be to let client to send a request >> > again after timeout, however wasn't yet implemented. >> > >> > ___ >> > zeromq-dev mailing list >> > zeromq-dev@lists.zeromq.org >> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> >> -- >> Studying for the Turing test >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] MDP protocol, detecting dead workers
Hi, The implemented protocol (ZMQ-RFC 7/MDP) has application level mutual heartbeating between the broker and the worker. And this works fine: both parties detect if the other side dies via missing heartbeats. The problem appears when the worker is assigned a long running job, heartbeating is _disabled_ while the job is being processed (as per 7/MDP specifies). This enables the worker to be single threaded, and avoids typical multithreaded issues (eg. processing thread hangs, heartbeating thread runs; worker in inconsistent state). When a worker crashes during job processing my application doesn't realize this since no messages are flowing (the broker is waiting for the job result), but the libzmq detects this, as the socket is always closed. My goal is to always keep in sync the number of underlying sockets in libzmq and Worker related objects in my application. I've googled around and found a few libzmq features that would suit my needs: - ZMQ_IDENTITY_FD - this was introduced and shortly removed from the lib - ZMQ_SRCFD - deprecated, but it's exactly what I need! - "Peer-Address" metadata, the recommended replacement for ZMQ_SRCFD, but not suitable for my needs I know fd's should be handled with care (monitor events are asynchronous, fd's get reused), but ZMQ_SRCFD solves my problem with the following ruleset: 1. When a Worker registers (first message over a connection) save the underlying fd - and - 2. Check that this fd is in use by another Worker, if it is: that Worker is dead since libzmq reused its file descriptor 3. If a Worker's fd is in closed state for a longer period (heartbeat expiry time), then it crashed and the fd was not re-used (get this info from monitor) I don't know if this is considered as an ugly hack by hardcore zeromq users, but it looks like a legitimate ZMQ_SRCFD use-case to me. It would be nice if it wasn't removed in the upcoming versions. Any feedback welcome! Regards, Gyorgy On Mon, Feb 13, 2017 at 10:21 PM, Greg Young wrote: > I believe the term here is application level heartbeats. > > It should also be supported that clients can heartbeat to server. It > is not always that all clients want similar heartbeat timeouts. > > On Mon, Feb 13, 2017 at 4:07 PM, Michal Vyskocil > wrote: > > Hi, > > > > You can take inspiration from malamute broker > > https://github.com/zeromq/malamute > > > > There clients pings server regularly. The same does MQTT (just it's a > > server, who pings clients). > > > > Sadly malamute is vulnerable to the same problem, that received service > > request may get lost. Solution would be to let client to send a request > > again after timeout, however wasn't yet implemented. > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > -- > Studying for the Turing test > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Updating zguide
Accepted. Thanks! On Feb 14, 2017 09:49, "Doron Somech" wrote: > Luca, Kevin you were invited to the api wiki. Let me know once you joined > and I will make you admins. > > On Feb 14, 2017 11:03, "Doron Somech" wrote: > >> It is another wikidot project. >> >> Will try to get access to it. >> >> On Mon, Feb 13, 2017 at 11:40 PM, Kevin Sapper >> wrote: >> >>> I was able to generate and upload the latest version of the czmq api >>> using Pieters secret Perl tools (github.com/zeromq/ztools). I can do >>> the same for libzmq. But I'm not sure to which wiki as I don't have access >>> to the original one? The same as czmq? I that case we need to adjust the >>> DNS. >>> >>> Am 04.01.2017 08:11 schrieb "Doron Somech" : >>> I'm not the admin of those websites, both zguide and czmq. So we might need to create new ones. Because it is all generated it not much loss and I do control the DNS. Do you want to try and generate the docs into new wikidot site and then I will redirect the DNS? On Tue, Jan 3, 2017 at 10:34 PM, Luca Boccassi >>> > wrote: > While on the subject of Wikidot access, could you please give myself > (luca.boccassi) and sappo access to the CZMQ page? > > https://github.com/zeromq/czmq/issues/1563 > http://czmq.zeromq.org/ > > When I try to edit I get a permission denied error > > On Tue, 2017-01-03 at 21:51 +0200, Doron Somech wrote: > > Done > > > > On Tue, Jan 3, 2017 at 7:11 PM, Kyle Kelley > wrote: > > > > > I'm rgbkrk on wikidot and my email is rgb...@gmail.com > > > > > > On Mon, Jan 2, 2017 at 11:26 PM, Doron Somech > wrote: > > > > Kevin what is your wikidot username and email? > > > > > > > > On Tue, Jan 3, 2017 at 12:24 AM, Kyle Kelley > wrote: > > > >> > > > >> Just as an aside, in case my message went through twice - > somehow my > > > >> emails were getting bounced and my account was disabled. Sorry > about > > > >> that! > > > >> > > > >> On Mon, Jan 2, 2017 at 2:18 PM, Benjamin Henrion < > zoo...@gmail.com> > > > wrote: > > > >> > On Mon, Jan 2, 2017 at 8:16 PM, Kevin Sapper < > kevinsappe...@gmail.com > > > > > > > >> > wrote: > > > >> >> Hi Kyle, > > > >> >> > > > >> >> it is a semi manual process. There a build and upload script > in the > > > >> >> repository that has be invoked by someone with permissions > to the > > > >> >> wikidot > > > >> >> page. > > > >> >> > > > >> >> Who has access to the zguide wikidot? @zoobab, @bluca, > @somdoron? > > > >> > > > > >> > I don't have access to it. > > > >> > > > > >> > Last edit is from 2013. > > > >> > > > > >> > -- > > > >> > Benjamin Henrion > > > >> > FFII Brussels - +32-484-566109 - +32-2-3500762 > > > >> > "In July 2005, after several failed attempts to legalise > software > > > >> > patents in Europe, the patent establishment changed its > strategy. > > > >> > Instead of explicitly seeking to sanction the patentability of > > > >> > software, they are now seeking to create a central European > patent > > > >> > court, which would establish and enforce patentability rules > in their > > > >> > favor, without any possibility of correction by competing > courts or > > > >> > democratically elected legislators." > > > >> > ___ > > > >> > zeromq-dev mailing list > > > >> > zeromq-dev@lists.zeromq.org > > > >> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > >> > > > >> > > > >> > > > >> -- > > > >> Kyle Kelley (@rgbkrk; lambdaops.com) > > > >> ___ > > > >> zeromq-dev mailing list > > > >> zeromq-dev@lists.zeromq.org > > > >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > > > > > > > > ___ > > > > zeromq-dev mailing list > > > > zeromq-dev@lists.zeromq.org > > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > > > > -- > > > Kyle Kelley (@rgbkrk; lambdaops.com) > > > ___ > > > zeromq-dev mailing list > > > zeromq-dev@lists.zeromq.org > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zero
Re: [zeromq-dev] Updating zguide
Done. On Di, Feb 14, 2017 at 10:48 , Doron Somech wrote: Luca, Kevin you were invited to the api wiki. Let me know once you joined and I will make you admins. On Feb 14, 2017 11:03, "Doron Somech" wrote: It is another wikidot project. Will try to get access to it. On Mon, Feb 13, 2017 at 11:40 PM, Kevin Sapper wrote: I was able to generate and upload the latest version of the czmq api using Pieters secret Perl tools (github.com/zeromq/ztools). I can do the same for libzmq. But I'm not sure to which wiki as I don't have access to the original one? The same as czmq? I that case we need to adjust the DNS. Am 04.01.2017 08:11 schrieb "Doron Somech" : I'm not the admin of those websites, both zguide and czmq. So we might need to create new ones. Because it is all generated it not much loss and I do control the DNS. Do you want to try and generate the docs into new wikidot site and then I will redirect the DNS? On Tue, Jan 3, 2017 at 10:34 PM, Luca Boccassi wrote: While on the subject of Wikidot access, could you please give myself (luca.boccassi) and sappo access to the CZMQ page? https://github.com/zeromq/czmq/issues/1563 http://czmq.zeromq.org/ When I try to edit I get a permission denied error On Tue, 2017-01-03 at 21:51 +0200, Doron Somech wrote: > Done > > On Tue, Jan 3, 2017 at 7:11 PM, Kyle Kelley wrote: > > > I'm rgbkrk on wikidot and my email is rgb...@gmail.com > > > > On Mon, Jan 2, 2017 at 11:26 PM, Doron Somech wrote: > > > Kevin what is your wikidot username and email? > > > > > > On Tue, Jan 3, 2017 at 12:24 AM, Kyle Kelley wrote: > > >> > > >> Just as an aside, in case my message went through twice - somehow my > > >> emails were getting bounced and my account was disabled. Sorry about > > >> that! > > >> > > >> On Mon, Jan 2, 2017 at 2:18 PM, Benjamin Henrion > > wrote: > > >> > On Mon, Jan 2, 2017 at 8:16 PM, Kevin Sapper > > > > > >> > wrote: > > >> >> Hi Kyle, > > >> >> > > >> >> it is a semi manual process. There a build and upload script in the > > >> >> repository that has be invoked by someone with permissions to the > > >> >> wikidot > > >> >> page. > > >> >> > > >> >> Who has access to the zguide wikidot? @zoobab, @bluca, @somdoron? > > >> > > > >> > I don't have access to it. > > >> > > > >> > Last edit is from 2013. > > >> > > > >> > -- > > >> > Benjamin Henrion > > >> > FFII Brussels - +32-484-566109 - +32-2-3500762 > > >> > "In July 2005, after several failed attempts to legalise software > > >> > patents in Europe, the patent establishment changed its strategy. > > >> > Instead of explicitly seeking to sanction the patentability of > > >> > software, they are now seeking to create a central European patent > > >> > court, which would establish and enforce patentability rules in their > > >> > favor, without any possibility of correction by competing courts or > > >> > democratically elected legislators." > > >> > ___ > > >> > zeromq-dev mailing list > > >> > zeromq-dev@lists.zeromq.org > > >> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > >> > > >> > > >> > > >> -- > > >> Kyle Kelley (@rgbkrk; lambdaops.com) > > >> ___ > > >> zeromq-dev mailing list > > >> zeromq-dev@lists.zeromq.org > > >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > > > > ___ > > > zeromq-dev mailing list > > > zeromq-dev@lists.zeromq.org > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > -- > > Kyle Kelley (@rgbkrk; lambdaops.com) > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Updating zguide
Luca, Kevin you were invited to the api wiki. Let me know once you joined and I will make you admins. On Feb 14, 2017 11:03, "Doron Somech" wrote: > It is another wikidot project. > > Will try to get access to it. > > On Mon, Feb 13, 2017 at 11:40 PM, Kevin Sapper > wrote: > >> I was able to generate and upload the latest version of the czmq api >> using Pieters secret Perl tools (github.com/zeromq/ztools). I can do the >> same for libzmq. But I'm not sure to which wiki as I don't have access to >> the original one? The same as czmq? I that case we need to adjust the DNS. >> >> Am 04.01.2017 08:11 schrieb "Doron Somech" : >> >>> I'm not the admin of those websites, both zguide and czmq. So we might >>> need to create new ones. >>> Because it is all generated it not much loss and I do control the DNS. >>> >>> Do you want to try and generate the docs into new wikidot site and then >>> I will redirect the DNS? >>> >>> On Tue, Jan 3, 2017 at 10:34 PM, Luca Boccassi >>> wrote: >>> While on the subject of Wikidot access, could you please give myself (luca.boccassi) and sappo access to the CZMQ page? https://github.com/zeromq/czmq/issues/1563 http://czmq.zeromq.org/ When I try to edit I get a permission denied error On Tue, 2017-01-03 at 21:51 +0200, Doron Somech wrote: > Done > > On Tue, Jan 3, 2017 at 7:11 PM, Kyle Kelley wrote: > > > I'm rgbkrk on wikidot and my email is rgb...@gmail.com > > > > On Mon, Jan 2, 2017 at 11:26 PM, Doron Somech wrote: > > > Kevin what is your wikidot username and email? > > > > > > On Tue, Jan 3, 2017 at 12:24 AM, Kyle Kelley wrote: > > >> > > >> Just as an aside, in case my message went through twice - somehow my > > >> emails were getting bounced and my account was disabled. Sorry about > > >> that! > > >> > > >> On Mon, Jan 2, 2017 at 2:18 PM, Benjamin Henrion < zoo...@gmail.com> > > wrote: > > >> > On Mon, Jan 2, 2017 at 8:16 PM, Kevin Sapper < kevinsappe...@gmail.com > > > > > >> > wrote: > > >> >> Hi Kyle, > > >> >> > > >> >> it is a semi manual process. There a build and upload script in the > > >> >> repository that has be invoked by someone with permissions to the > > >> >> wikidot > > >> >> page. > > >> >> > > >> >> Who has access to the zguide wikidot? @zoobab, @bluca, @somdoron? > > >> > > > >> > I don't have access to it. > > >> > > > >> > Last edit is from 2013. > > >> > > > >> > -- > > >> > Benjamin Henrion > > >> > FFII Brussels - +32-484-566109 - +32-2-3500762 > > >> > "In July 2005, after several failed attempts to legalise software > > >> > patents in Europe, the patent establishment changed its strategy. > > >> > Instead of explicitly seeking to sanction the patentability of > > >> > software, they are now seeking to create a central European patent > > >> > court, which would establish and enforce patentability rules in their > > >> > favor, without any possibility of correction by competing courts or > > >> > democratically elected legislators." > > >> > ___ > > >> > zeromq-dev mailing list > > >> > zeromq-dev@lists.zeromq.org > > >> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > >> > > >> > > >> > > >> -- > > >> Kyle Kelley (@rgbkrk; lambdaops.com) > > >> ___ > > >> zeromq-dev mailing list > > >> zeromq-dev@lists.zeromq.org > > >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > > > > ___ > > > zeromq-dev mailing list > > > zeromq-dev@lists.zeromq.org > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > -- > > Kyle Kelley (@rgbkrk; lambdaops.com) > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.
Re: [zeromq-dev] Updating zguide
It is another wikidot project. Will try to get access to it. On Mon, Feb 13, 2017 at 11:40 PM, Kevin Sapper wrote: > I was able to generate and upload the latest version of the czmq api using > Pieters secret Perl tools (github.com/zeromq/ztools). I can do the same > for libzmq. But I'm not sure to which wiki as I don't have access to the > original one? The same as czmq? I that case we need to adjust the DNS. > > Am 04.01.2017 08:11 schrieb "Doron Somech" : > >> I'm not the admin of those websites, both zguide and czmq. So we might >> need to create new ones. >> Because it is all generated it not much loss and I do control the DNS. >> >> Do you want to try and generate the docs into new wikidot site and then I >> will redirect the DNS? >> >> On Tue, Jan 3, 2017 at 10:34 PM, Luca Boccassi >> wrote: >> >>> While on the subject of Wikidot access, could you please give myself >>> (luca.boccassi) and sappo access to the CZMQ page? >>> >>> https://github.com/zeromq/czmq/issues/1563 >>> http://czmq.zeromq.org/ >>> >>> When I try to edit I get a permission denied error >>> >>> On Tue, 2017-01-03 at 21:51 +0200, Doron Somech wrote: >>> > Done >>> > >>> > On Tue, Jan 3, 2017 at 7:11 PM, Kyle Kelley wrote: >>> > >>> > > I'm rgbkrk on wikidot and my email is rgb...@gmail.com >>> > > >>> > > On Mon, Jan 2, 2017 at 11:26 PM, Doron Somech >>> wrote: >>> > > > Kevin what is your wikidot username and email? >>> > > > >>> > > > On Tue, Jan 3, 2017 at 12:24 AM, Kyle Kelley >>> wrote: >>> > > >> >>> > > >> Just as an aside, in case my message went through twice - somehow >>> my >>> > > >> emails were getting bounced and my account was disabled. Sorry >>> about >>> > > >> that! >>> > > >> >>> > > >> On Mon, Jan 2, 2017 at 2:18 PM, Benjamin Henrion < >>> zoo...@gmail.com> >>> > > wrote: >>> > > >> > On Mon, Jan 2, 2017 at 8:16 PM, Kevin Sapper < >>> kevinsappe...@gmail.com >>> > > > >>> > > >> > wrote: >>> > > >> >> Hi Kyle, >>> > > >> >> >>> > > >> >> it is a semi manual process. There a build and upload script >>> in the >>> > > >> >> repository that has be invoked by someone with permissions to >>> the >>> > > >> >> wikidot >>> > > >> >> page. >>> > > >> >> >>> > > >> >> Who has access to the zguide wikidot? @zoobab, @bluca, >>> @somdoron? >>> > > >> > >>> > > >> > I don't have access to it. >>> > > >> > >>> > > >> > Last edit is from 2013. >>> > > >> > >>> > > >> > -- >>> > > >> > Benjamin Henrion >>> > > >> > FFII Brussels - +32-484-566109 - +32-2-3500762 >>> > > >> > "In July 2005, after several failed attempts to legalise >>> software >>> > > >> > patents in Europe, the patent establishment changed its >>> strategy. >>> > > >> > Instead of explicitly seeking to sanction the patentability of >>> > > >> > software, they are now seeking to create a central European >>> patent >>> > > >> > court, which would establish and enforce patentability rules in >>> their >>> > > >> > favor, without any possibility of correction by competing >>> courts or >>> > > >> > democratically elected legislators." >>> > > >> > ___ >>> > > >> > zeromq-dev mailing list >>> > > >> > zeromq-dev@lists.zeromq.org >>> > > >> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > > >> >>> > > >> >>> > > >> >>> > > >> -- >>> > > >> Kyle Kelley (@rgbkrk; lambdaops.com) >>> > > >> ___ >>> > > >> zeromq-dev mailing list >>> > > >> zeromq-dev@lists.zeromq.org >>> > > >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > > > >>> > > > >>> > > > >>> > > > ___ >>> > > > zeromq-dev mailing list >>> > > > zeromq-dev@lists.zeromq.org >>> > > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > > >>> > > >>> > > >>> > > -- >>> > > Kyle Kelley (@rgbkrk; lambdaops.com) >>> > > ___ >>> > > zeromq-dev mailing list >>> > > zeromq-dev@lists.zeromq.org >>> > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > > >>> > ___ >>> > zeromq-dev mailing list >>> > zeromq-dev@lists.zeromq.org >>> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >>> >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
