[zeromq-dev] Cleaning up file descriptors for dead router peers

2015-06-23 Thread Marcin Romaszewicz
Hi All, I've got an issue with ZMQ_ROUTER sockets which I'm having a hard time working around, and I'd love some advice, but I suspect the answer is that what I want to do isn't possible. Say I have a router socket listening on a port, and I have peers connecting and disconnecting randomly over T

Re: [zeromq-dev] Cleaning up file descriptors for dead router peers

2015-06-24 Thread Marcin Romaszewicz
ote: > Do you think there's any way to reproduce this in the lab, e.g. > killing a peer before it can shut down TCP properly? > > On Tue, Jun 23, 2015 at 10:08 PM, Marcin Romaszewicz > wrote: > > Hi All, > > > > I've got an issue with ZMQ_ROUTER sockets whi

Re: [zeromq-dev] Cleaning up file descriptors for dead router peers

2015-06-24 Thread Marcin Romaszewicz
be fun to see if this solves your > problem. (In theory it should...) > > https://github.com/zeromq/libzmq/pull/1448 > > > On Wed, Jun 24, 2015 at 6:48 PM, Marcin Romaszewicz > wrote: > > Yes, you can easily reproduce this by pulling a network cable or shutting > > the

Re: [zeromq-dev] Cleaning up file descriptors for dead router peers

2015-06-25 Thread Marcin Romaszewicz
x27;t handling this properly. > > > On Wed, Jun 24, 2015 at 8:16 PM, Marcin Romaszewicz > wrote: > > Thanks, this probably would solve our problem, however, I'm reluctant to > > deploy the bleeding edge from your git repo into our production systems, > > even if

Re: [zeromq-dev] Cleaning up file descriptors for dead router peers

2015-06-26 Thread Marcin Romaszewicz
, the router can't ping the peers, and the peers can't ping the router. The file descriptors and connections remain open forever on both sides. Furthermore, when you undo the iptables block, the connections never come back. On Thu, Jun 25, 2015 at 3:35 PM, Marcin Romaszewicz wrote: &

Re: [zeromq-dev] Cleaning up file descriptors for dead router

2015-06-29 Thread Marcin Romaszewicz
fine and the peers to never close their sockets. > > Jonathan > > On Fri, Jun 26, 2015 at 4:58 PM, Marcin Romaszewicz > wrote: > >> Hi All, >> I've gota trivial bit of code to reproduce this issue on a single host >> using iptables to simulate network parti

Re: [zeromq-dev] Cleaning up file descriptors for dead router peers

2015-06-29 Thread Marcin Romaszewicz
ams' heartbeat fix does indeed fix this issue. What would be really nice is some sort of API call to tell a router socket to close a peer. On Fri, Jun 26, 2015 at 4:58 PM, Marcin Romaszewicz wrote: > Hi All, > > I've gota trivial bit of code to reproduce this issue on a

[zeromq-dev] zmq_msg_send blocks reading from zmq::mailbox_t

2015-07-01 Thread Marcin Romaszewicz
Hi Guys, We had our prod systems running on 3.2.4, and to work around some file descriptor leaks, I'm testing 4.1.2 before back-porting jbreams' heartbeats patch, and testing unmodified 4.1.2 is failing for me in production. I've got ZMQ_ROUTER sockets wedged in zmq_msg_send, spinning in a while

Re: [zeromq-dev] zmq_msg_send blocks reading from zmq::mailbox_t

2015-07-02 Thread Marcin Romaszewicz
munication has already happened. I'll post the minimal test case here if I can come up with one. > On Thu, Jul 2, 2015 at 2:12 AM, Marcin Romaszewicz > wrote: > > Hi Guys, > > > > We had our prod systems running on 3.2.4, and to work around some file > > descri

Re: [zeromq-dev] zmq_msg_send blocks reading from zmq::mailbox_t

2015-07-02 Thread Marcin Romaszewicz
2015 at 8:19 PM, Marcin Romaszewicz > wrote: > > > > > > On Thu, Jul 2, 2015 at 11:16 AM, Pieter Hintjens wrote: > >> > >> Can you make a minimal test case? > >> > > > > I'm trying, yes, but so far, no luck. Our zmq-based message r

[zeromq-dev] control messages appearing in application message reads

2015-07-08 Thread Marcin Romaszewicz
Hi All, I've been working for a while on avoiding file descriptor leaks, and various tests were going well using jbreams heartbeat protocol, but I've hit a snag. I've got router sockets in servers running in AWS on zmq 4.2.0 (fresh from your git repo), which are accepting connections from a whole

Re: [zeromq-dev] control messages appearing in application message reads

2015-07-09 Thread Marcin Romaszewicz
se where the > PING messages properly get filtered out on older versions of zeromq? > > > On Wed, Jul 8, 2015 at 7:15 PM, Marcin Romaszewicz > wrote: > >> Hi All, >> >> I've been working for a while on avoiding file descriptor leaks, and >> various te

Re: [zeromq-dev] Setting an Android Device as a Subscriber

2015-07-21 Thread Marcin Romaszewicz
Subscribers by default (at least in the C++ version of ZMQ) do not subscribe to anything. You need to call zmq_setsockopt(ZMQ_SUBSCRIBE...) to establish a subscription. You probably have to do something similar on the subscriber socket in jeroMQ On Tue, Jul 21, 2015 at 2:21 PM, Areeb Mehmood < are

Re: [zeromq-dev] Setting an Android Device as a Subscriber

2015-07-21 Thread Marcin Romaszewicz
.setSocketOption command. The parameters > are as such: > setSocketOption(SocketBase s, int option, Object optval). Any guidance on > what parameters I should put in? I'm expecting a stream of strings from the > publisher > > On Tue, Jul 21, 2015 at 3:00 PM, Marcin Romaszewicz

Re: [zeromq-dev] Determine Connection state PUB/SUB

2015-08-13 Thread Marcin Romaszewicz
I've tried to figure that out as well, and there is no way to do it on the ZMQ level. Have Task1 emit heartbeats and if Task2 and Task3 aren't seeing heartbeats, it means that the subscriptions haven't propagated yet. There can be a sizeable delay between subscribing and actually getting data (I've

Re: [zeromq-dev] Pub-Sub Message Envelopes with Sender

2015-10-01 Thread Marcin Romaszewicz
No it's not possible, because pub-sub filtering only looks at the first message part. I do something similar in my own pub/sub system, and I just combine the first two parts. So, msg[0] = 'f1'.'f2' msg[1] = other stuff Then, my subscribers just subscribe to 'f1.f2' If you have a lot of combinat

Re: [zeromq-dev] zmq_msg_recv not blocking

2015-11-23 Thread Marcin Romaszewicz
That should be rc != -1. If you receive, say, 3 bytes, your error code will trigger. Those errno values could be from previous functions. On Mon, Nov 23, 2015 at 6:15 AM, Jeff Shanab wrote: > The Guide has an assert (rc == 0) that of course exited the program. > I replaced that with the if (rc !

Re: [zeromq-dev] PUSH socket behavior

2016-03-14 Thread Marcin Romaszewicz
Check the man page for zmq_getsockopt and zmq_poll, you can use the ZMQ_POLLOUT flag to determine if the socket can send something without blocking, and if it can't, you can drop it yourself. -- Marcin On Mon, Mar 14, 2016 at 11:59 AM, Diego Fons wrote: > Hi everyone, > > Is it possible to chan

Re: [zeromq-dev] Resource temporarily unavailable") at src/err.cpp:83

2016-08-12 Thread Marcin Romaszewicz
You are running out of file descriptors, most likely. Each outgoing connection requires a file descriptor, and 100k is a lot of sockets. Are you trying to make 100k ZMQ sockets? If you are, just limit the number of outgoing sockets, and it should work. Only send 1000 at a time, for example. On Fr

Re: [zeromq-dev] sending from multiple threads

2016-12-01 Thread Marcin Romaszewicz
On Thu, Dec 1, 2016 at 11:30 AM, Lineker Tomazeli wrote: > Hi Luca, > > Thanks for answering. See my comments below. > > Lineker Tomazeli > tomazeli.net > > On Thu, Dec 1, 2016 at 12:19 PM, Luca Boccassi > wrote: > >> On Thu, 2016-12-01 at 10:23 -0500, Lineker Tomazeli wrote: >> > Hi guys, >> >

Re: [zeromq-dev] How to get transactions from bitcoind?

2017-01-12 Thread Marcin Romaszewicz
If it's blocked on socket.recv() it means you're not receiving anything, because it will wait until the first message arrives. If you want to receive stuff and print in a loop like the python code, you need to do it roughly like this (I'm just typing in from memory, so there might be some compiler

Re: [zeromq-dev] publish/subscribe and the "fast producer" problem

2017-03-03 Thread Marcin Romaszewicz
Yeah, you need a way to throttle back the publisher. Have the many nodes tell the publisher to slow down if they detect too high of a drop rate. Your publisher can then decide whether to throttle for the slowest client, or for the fastest client. Also, pub/sub may not be the right way to do this fo

Re: [zeromq-dev] Porting libzmq to C++17

2017-05-22 Thread Marcin Romaszewicz
Why? C++17 is syntactic sugar on top of older versions of C++. C++17 won't give you more performance or better code, just easier to read. There are still lots of old devices which only have C++98 support, which is probably the most widely adopted C++ standard. A C++17 wrapper API which is optional

Re: [zeromq-dev] Zeromq with Docker Swarm

2017-06-19 Thread Marcin Romaszewicz
Any load balancer is going to pick between your servers at random, as far as you're concerned. The way this is typically handled is that your server instances use some kind of shared state, like a shared database, filesystem, quorum, or any other form of shared state so that it doesn't matter which

Re: [zeromq-dev] Stateless service protocols

2017-06-26 Thread Marcin Romaszewicz
A "stateless" protocol would be one where your message has to contain all the information needed to handle it, so that the server contains no state. It sounds like a very large constraint. I work on a large load-balanced set of microservices, and they're all stateful, using shared state via a postg

Re: [zeromq-dev] Stateless service protocols

2017-07-05 Thread Marcin Romaszewicz
The traditional way to deal with this issue is to have your schedulers use shared state between themselves. It would work something like this 1) worker talks to scheduler, performs an operation. Scheduler hands back a request ID or something and writes it to the database. worker->scheduler.doS

Re: [zeromq-dev] Assertion failed: pfd.revents & POLLIN (src/signaler.cpp:243)

2017-09-29 Thread Marcin Romaszewicz
On Fri, Sep 29, 2017 at 2:38 AM, Bachmair Florian - flexSolution GmbH < florian.bachm...@flexsolution.eu> wrote: > Hi! > > When I stop my application in roughly 1 out of 20 tries I get this error: > > Assertion failed: pfd.revents & POLLIN (src/signaler.cpp:243) (here is > the corresponding line o