Re: [zeromq-dev] problem with sending and recieving in a simple PAIR setup
Aren't you missing indentation in rtx functions?
On Sat, Aug 18, 2018, 01:20 Michael Hansen wrote:
> Hello, I am quite new to ZMQ which seems like a really nice library.
> However i seem to be running into some issues already.
> I have to make a setup with just 2 peers who can send to each other.
> There is no order in how the clients communicate - however in generel data
> flows
> from one to the other and commands from the other to the first.
>
> I made a small test as below but communication seem to hang up and only
> one part is sending.
> Also - sometimes when i try to run this, i get the following error:
>
> 'Resource temporarily unavailable (bundled/zeromq/src/signaler.cpp:301)
>
> Abort trap: 6'
>
>
> Here are the 2 components, what am I doing wrong?
>
>
>
>
> # ==
> # = SERVER A
> # ==
>
>
> import threading
> import time
> import zmq
>
>
> port =
> context = zmq.Context()
> socket = context.socket(zmq.PAIR)
> socket.connect("tcp://127.0.0.1:%s" % port)
>
>
> def rtx(socket):
> print('A started receiving...')
> while 1:
> print(b'A RECEIVED : %s' % socket.recv())
>
>
> def ttx(socket):
> print('A started transmitting...')
> while 1:
> socket.send(b'Message from A')
> time.sleep(1)
>
>
> threading.Thread(target=rtx, args=(socket,)).start()
> threading.Thread(target=ttx, args=(socket,)).start()
>
>
>
>
> # ==
> # = SERVER B
> # ==
>
>
> import threading
> import time
> import zmq
>
>
> port =
> context = zmq.Context()
> socket = context.socket(zmq.PAIR)
> socket.bind("tcp://127.0.0.1:%s" % port)
>
>
> def rtx(socket):
> print('B started receiving...')
> while 1:
> print(b'B RECEIVED : %s' % socket.recv())
>
>
> def ttx(socket):
> print('B started transmitting...')
> while 1:
> socket.send(b'Message from B')
> time.sleep(1)
>
>
> threading.Thread(target=rtx, args=(socket,)).start()
> threading.Thread(target=ttx, args=(socket,)).start()
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Slow joiner syndrome solution for XSUB/XPUB broker based system
You are right - my method will cause high CPU usage for the period of the
sync, which is certainly a disadvantage.
The advantage though is keeping both the broker and the
publishers/subscribers simple - The sync is done "under the hood" by the
infrastructure which encapsulates zmq from the users, so that both
subscribers and publishers are kept simple and not aware of the broker in
the middle. The broker is also kept as simple as the zmq example for
XPUB/XSUB proxy.
We found a bug in our sync implementation which was the reason I opened
this thread - now that it is solved I can commit that this solution works
from both sides - subscribers and publishers, details below for those who
are interested.
Thanks,
Tomer
In our framework, "controllers" and "agents" all use a "BrokerInterface"
which is a zeromq wrapper with PUB socket and a SUB socket.
A controller is an entity on the bus which sends commands to agents, and
expects replies.
An agent is an entity on the bus which receives commands from controllers
and sends replies.
>From zeromq perspective, both are publishers and subscribers, therefore
they use BrokerInterface class for IPC.
BrokerInterface class
Init() - connects the SUB and PUB sockets to the bus's (broker) subscribers
and publishers endpoints
Subscribe(topic) - subscribes to via the SUB socket
Send(msg) - send message via the PUB socket (message contains a topic and
data)
Recv() - Receive a message via the SUB socket
Sync() - Synchronize for Send() and Subscribe() - subscribe("Hello"), then
send "Hello" messages until the first one is received, then
unsubscribe("Hello")
Bug description
In one of our unit tests, we forked a broker, and a test process using the
BrokerInterface.
The test subscribed to topics A,B,C, called Sync() method (which subscribed
to Hello, sent Hello until received, then unsubscribed Hello), and sent
messages to topics A,B,C which were received as expected.
Then it subscribed to topic D, called Sync() again, then sent messages to
topic D which were lost.
We found out that if we add a delay before sending messages to topic D,
everything works.
Root cause
The first Sync() added a subscription to "Hello" in the broker, sent
messages, and then unsubscribed from "Hello". We know that the subscribe
was completed because we received an "Hello" message.
What we don't know for certain is that the unsubscribe completed - and in
fact it wasn't...
Recall that the Sync() after subscribe() is done to ensure that previous
subscriptions are registered in the broker.
What happened was that the 2nd Sync() used the same "Hello" topic to which
we already subscribed-unsubscribed from in the 1st Sync(), but the
unsubscribe() part hasn't happened yet, causing the "Hello" messages from
the 2nd Sync() to be received, marking this synchronization done, and
failing the test since the subscription to topic "D" hadn't occurred yet.
Fix
The fix is to make sure that the Sync() topic is unique for the process
which is using it.
The sync topic we now use contains the process name and a unique identifier
which is changed in every Sync() call.
On Tue, May 29, 2018 at 4:13 AM, Chris Billington <
chrisjbilling...@gmail.com> wrote:
> I think that mostly makes sense. I'm not sure if zmq_proxy will work,
> because it will probably attempt to forward subscription requests read from
> the XPUB to the PULL socket. However I think it will work if you go back to
> the original spec of using a PUB socket for the senders and a XSUB in the
> broker.
>
> The only downside of that approach compared to mine I think is that the
> 'sending repeatedly' of SYNC messages will require some small sleep in
> between sends to not peg the CPU, leading to a likely slower than necessary
> time to establish that the subscription is complete, whereas my method has
> no sleeping and so will complete more deterministically and in a shorter
> time. But the simplicity of your approach might be better all things
> considered.
>
> -Chris
>
> On Mon, May 28, 2018 at 9:46 PM, Tomer Eliyahu
> wrote:
>
>> Thanks for sharing Chris, this is interesting.
>>
>> From the subscribers perspective, I think that using the same sync
>> mechanism I described for the publishers can solve the subscribers side so
>> that they know the subscription is complete in the broker while keeping the
>> broker "dumb" -
>> Subscribe first to the real topics, then perform a sync - subscribe to
>> "SYNC", then send messages via the PUB interface (or PUSH) with "SYNC"
>> topic until the first message is received.
>> Once this happens, you know for certain that the previous subscriptions
>> were also received by the broker.
>>
>&
Re: [zeromq-dev] Slow joiner syndrome solution for XSUB/XPUB broker based system
Thanks for sharing Chris, this is interesting. >From the subscribers perspective, I think that using the same sync mechanism I described for the publishers can solve the subscribers side so that they know the subscription is complete in the broker while keeping the broker "dumb" - Subscribe first to the real topics, then perform a sync - subscribe to "SYNC", then send messages via the PUB interface (or PUSH) with "SYNC" topic until the first message is received. Once this happens, you know for certain that the previous subscriptions were also received by the broker. Then the broker can go back to using zmq_proxy() instead of handling subscriptions.. What do you think? On Mon, May 28, 2018 at 1:21 PM, Chris Billington < chrisjbilling...@gmail.com> wrote: > I've handled this problem by avoiding using a PUB socket for the senders > of messages: > > a) senders of messages send them on a PUSH socket and the broker forwards > from a PULL to a XPUB. This means that there is no slow joiner problem with > the senders starting up (PUSH won't drop messages), but has the downside > that the messages are *always* sent to the broker even if there are no > subscribers. They will instead be dropped by the XPUB if there are no > subscribers. > > b) Subscribers request and wait for subscription confirmation messages > from the broker when they subscribe to a topic so calling code can be sure > they are subscribed before starting the senders. > > See here for my Python project that implements this (the EventBroker and > Event classes): > > https://bitbucket.org/cbillington/zprocess/src/ > default/zprocess/process_tree.py?at=default&fileviewer=file- > view-default#process_tree.py-102 > > > On Mon, May 28, 2018 at 7:40 PM, Tomer Eliyahu > wrote: > >> Hi Gyorgy, >> >> Thank you - but assuming the subscriber connect and subscribe happen long >> before the publisher starts, is there still a risk for the slow joiner >> problem? >> >> Assume the following flow: >> broker: >>zmq_bind(frontend, "ipc:///tmp/publishers"); >>zmq_bind(backend, "ipc:///tmp/subscribers"); >>zmq_proxy(frontend, backend, NULL); >> >> >> >> subscriber: >>zmq_connect(sub_socket, "ipc:///tmp/subscribers"); >> >> >> >> >> >> publisher: >>zmq_connect(pub_socket, "ipc:///tmp/publishers"); >>zmq_connect(sub_socket, "ipc:///tmp/subscribers"); >> >> >> >> >> >> >> Bottom line - is there some sort of synchronization done under the hood >> by ZMQ when the publisher first sends a message with the topic on which the >> subscriber subscribed? or is this all handled between the broker and the >> subscriber? >> >> Thanks, >> Tomer >> >> On Mon, May 28, 2018 at 12:23 PM, Gyorgy Szekely >> wrote: >> >>> Hi Tomer >>> As far as I know the message from the publisher will reach the broker. >>> According to the docs, the PUB socket drops messages in mute-state (HWM >>> reached), and it's not the case here. The message will be sent as soon as >>> the connection is established, and the socket termination blocks until the >>> send is complete. Unless you set linger to zero. >>> >>> The slow joiner problem means that subscriptions may not be active by >>> the time the publisher send the message. Either because the subscriber is >>> not yet running, or because the subscribe calls themselves are asynchronous >>> (by the time setsockopt(SUNSCRIBE) returns the broker is not yet aware of >>> this). The zmq guide shows mitigations for this problem in the Advanced >>> Publish Subscribe chapter. >>> >>> Regards, >>> Gyorgy >>> >>> On Mon, May 28, 2018 at 11:06 AM, Tomer Eliyahu >> > wrote: >>> >>>> Hi, >>>> >>>> >>>> >>>> I know this topic was probably discussed before, I couldn't find a >>>> proper solution, so I implemented something a bit different. I'm not sure >>>> if this solves all pitfalls, i'll be greatfull for comments. >>>> >>>> >>>> >>>> We have a system with a XPUB-XSUB broker running as a separate process >>>> in the system (binds frontend to ipc:///tmp/publishers and backend to >>>> ipc:///tmp/subscribers). >>>> >>>> >>>> >>>> Clients of the broker have both SUB socket for receiving mes
Re: [zeromq-dev] Slow joiner syndrome solution for XSUB/XPUB broker based system
Hi Gyorgy, Thank you - but assuming the subscriber connect and subscribe happen long before the publisher starts, is there still a risk for the slow joiner problem? Assume the following flow: broker: zmq_bind(frontend, "ipc:///tmp/publishers"); zmq_bind(backend, "ipc:///tmp/subscribers"); zmq_proxy(frontend, backend, NULL); subscriber: zmq_connect(sub_socket, "ipc:///tmp/subscribers"); publisher: zmq_connect(pub_socket, "ipc:///tmp/publishers"); zmq_connect(sub_socket, "ipc:///tmp/subscribers"); Bottom line - is there some sort of synchronization done under the hood by ZMQ when the publisher first sends a message with the topic on which the subscriber subscribed? or is this all handled between the broker and the subscriber? Thanks, Tomer On Mon, May 28, 2018 at 12:23 PM, Gyorgy Szekely wrote: > Hi Tomer > As far as I know the message from the publisher will reach the broker. > According to the docs, the PUB socket drops messages in mute-state (HWM > reached), and it's not the case here. The message will be sent as soon as > the connection is established, and the socket termination blocks until the > send is complete. Unless you set linger to zero. > > The slow joiner problem means that subscriptions may not be active by the > time the publisher send the message. Either because the subscriber is not > yet running, or because the subscribe calls themselves are asynchronous (by > the time setsockopt(SUNSCRIBE) returns the broker is not yet aware of > this). The zmq guide shows mitigations for this problem in the Advanced > Publish Subscribe chapter. > > Regards, > Gyorgy > > On Mon, May 28, 2018 at 11:06 AM, Tomer Eliyahu > wrote: > >> Hi, >> >> >> >> I know this topic was probably discussed before, I couldn't find a proper >> solution, so I implemented something a bit different. I'm not sure if this >> solves all pitfalls, i'll be greatfull for comments. >> >> >> >> We have a system with a XPUB-XSUB broker running as a separate process in >> the system (binds frontend to ipc:///tmp/publishers and backend to >> ipc:///tmp/subscribers). >> >> >> >> Clients of the broker have both SUB socket for receiving messages, and a >> PUB socket for sending messages. When a client boots, it connects both its >> PUB and SUB sockets to the broker's endpoints, and subscribes to the topic >> of interest. >> >> >> For the sake of simplicity, lets assume there we have only the broker, a >> publisher and a subscriber processes in the system: >> >> We make sure that the broker process starts first, then a subscriber >> which connects and subscribes to the topic, and only then start the >> publisher. The publisher then sends a single message and terminates. >> >> Obviously, the message is lost due to the slow joiner syndrome - I assume >> the reason for that is because the publisher process zmq_connect() call is >> asynchronous, therefore the connect is not actually complete by the time we >> send the message. >> >> >> >> I thought of a possible solution for this - basically we want to >> synchronize the connect operation done by the publisher. Having both PUB >> and SUB socket, we can simply send a dummy message from PUB to SUB on the >> same publisher process until the first message is receieved, and then it is >> guarantied that the connect is done and consecutive messages (now to "real" >> topics with actual subscribers) will not be lost. >> >> >> >> The only part i'm not sure about is the subscriber side - assuming the >> subscriber boots, connects and subscribes _before_ we start the publisher - >> is it guarantied that no message will be lost (assuming ofcourse the >> subscriber doesn't crash / unsubscribe / etc.) ? >> >> >> >> Thanks, >> >> Tomer >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Slow joiner syndrome solution for XSUB/XPUB broker based system
Hi, I know this topic was probably discussed before, I couldn't find a proper solution, so I implemented something a bit different. I'm not sure if this solves all pitfalls, i'll be greatfull for comments. We have a system with a XPUB-XSUB broker running as a separate process in the system (binds frontend to ipc:///tmp/publishers and backend to ipc:///tmp/subscribers). Clients of the broker have both SUB socket for receiving messages, and a PUB socket for sending messages. When a client boots, it connects both its PUB and SUB sockets to the broker's endpoints, and subscribes to the topic of interest. For the sake of simplicity, lets assume there we have only the broker, a publisher and a subscriber processes in the system: We make sure that the broker process starts first, then a subscriber which connects and subscribes to the topic, and only then start the publisher. The publisher then sends a single message and terminates. Obviously, the message is lost due to the slow joiner syndrome - I assume the reason for that is because the publisher process zmq_connect() call is asynchronous, therefore the connect is not actually complete by the time we send the message. I thought of a possible solution for this - basically we want to synchronize the connect operation done by the publisher. Having both PUB and SUB socket, we can simply send a dummy message from PUB to SUB on the same publisher process until the first message is receieved, and then it is guarantied that the connect is done and consecutive messages (now to "real" topics with actual subscribers) will not be lost. The only part i'm not sure about is the subscriber side - assuming the subscriber boots, connects and subscribes _before_ we start the publisher - is it guarantied that no message will be lost (assuming ofcourse the subscriber doesn't crash / unsubscribe / etc.) ? Thanks, Tomer ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] cppzmq revival and RFC on design goals and supported platforms
I suggest to add support for building cppzmq without enforcing the build of libzmq via cmake. In openwrt for example, there is cmake support but zmq is built with autotools, which majes integrating cppzmq cumbersome so we eventually had to patch cppzmq to a previous version and strip some cmake features which were recently added. On Wed, May 23, 2018, 19:07 wrote: > Hi, > > Pawel Kurdybacha (kurdybacha) and me (sigiesec) have recently started to > "revive" cppzmq (https://github.com/zeromq/cppzmq), the light-weight C++ > wrapper around libzmq. We added CI for Windows/MSVC, Linux and MacOS, > implemented tests, cleaned up the CMake infrastructure, formatted the > source code consistently and added some overview documentation. > > If you are using cppzmq or are interested in using it, we encourage you to > have a look at the recent changes. > > One particular point we would like to seek feedback on are the design > goals, which have recently been documented for the first time. I tried to > extrapolate them from the actual design, and from the reasons we chose to > use cppzmq in comparison to other alternatives. These are part of the > https://github.com/zeromq/cppzmq/blob/master/README.md file: > > * cppzmq maps the libzmq C API to C++ concepts. In particular: >* it is type-safe (the libzmq C API exposes various class-like concepts > as void*) >* it provides exception-based error handling (the libzmq C API provides > errno-based error handling) >* it provides RAII-style classes that automate resource management (the > libzmq C API requires the user to take care to free resources explicitly) > * cppzmq is a light-weight, header-only binding. You only need to include > the header file zmq.hpp (and maybe zmq_addon.hpp) to use it. > * zmq.hpp is meant to contain direct mappings of the abstractions provided > by the libzmq C API, while zmq_addon.hpp provides additional higher-level > abstractions. > > We would like to here from you if you agree with these design goals. If > you have any opposing views, proposals for improvement or extension of the > design goals, please share them on the mailing list or by sending a PR. > > Another part of the README is a section on the supported platforms. Please > review this section, in particular if you do not use MacOS, Linux or > Windows/MSVC with a recent compiler. If you successfully use a different > platform, please send a PR to include this in the list of "Additional > platforms that are known to work". Support for non-C++11 compilers is > already partial only, and might be removed completely, unless there are > users that still require such support. > > Of course, you are also invited to contribute extensions, new features, > cleanup, further tests, etc. to cppzmq. > > Best regards > Simon > > -- > i.A. Simon Giesecke > BTC Business Technology Consulting AG > Kurfürstendamm 33 > 10719 Berlin > E-Mail: simon.giese...@btc-ag.com > > Rechtliche Hinweise: > www.btc-ag.com/impressum.htm > Handelsregister: Amtsgericht Oldenburg HRB 4717 > Aufsichtsratsvorsitzender: Michael Heidkamp > Vorstand: Dr. Jörg Ritter, Dirk Thole > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] FBZMQ
I know for a fact that Facebook is using it, isn't that enough?:) On Wed, May 9, 2018, 21:25 E.W.Z. wrote: > Not a problem, running in a container, so, I can bring with me any > dependency I need. > > It looks promising, but lacks documentation, samples are quite simple and > do not reflect real life cases. My main concern, is it mature enough? > Didn’t find any reference to anyone using it. > > In any case, the code looks solid, will give it a try > > > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > > > *From: *Tomer Eliyahu > *Sent: *Wednesday, May 9, 2018 9:20 PM > *To: *ZeroMQ development list > *Subject: *Re: [zeromq-dev] FBZMQ > > > > It sounds really promising and I considered using it for our project, > mainly due to the thrift support it provides. Unfortunately it requires > c++14 and we can't do more than c++11 (I was told that our customers > wouldn't like to add this dependency - embedded target), so take that into > account.. > > > > On Wed, May 9, 2018, 19:42 Harald Achitz wrote: > > this is a very interesting link. > > what a luck that I had some time to read on this list today. > > thanks for mention this! > > > > /Harald > > > > > > Am Mi., 9. Mai 2018 um 07:26 Uhr schrieb Ernest Zed < > kreuzerkr...@gmail.com>: > > Hi, > > Does anyone have an experience with facebook's C++ binding to 0MQ aka > fbzmq? > > > > Sincerely, > > E. > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] FBZMQ
It sounds really promising and I considered using it for our project, mainly due to the thrift support it provides. Unfortunately it requires c++14 and we can't do more than c++11 (I was told that our customers wouldn't like to add this dependency - embedded target), so take that into account.. On Wed, May 9, 2018, 19:42 Harald Achitz wrote: > this is a very interesting link. > what a luck that I had some time to read on this list today. > thanks for mention this! > > /Harald > > > Am Mi., 9. Mai 2018 um 07:26 Uhr schrieb Ernest Zed < > kreuzerkr...@gmail.com>: > >> Hi, >> Does anyone have an experience with facebook's C++ binding to 0MQ aka >> fbzmq? >> >> Sincerely, >> E. >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
