Hi,
I have been using ZMQ_PAIR recently for a one to one data feed over TCP. I
know the documentation doesn't explicitly say this should work (over TCP) but
for the most part it does. Last week I came across a situation where the bind
end of the connection had reconnection issues if no data
details again. So no
1 second delay.
You can use either method or both like me.
I also think there is a beacon service built into zeromq (or czmq?) that may
suit but I have never used it.
Cheers
James Harvey
On 25 Jun 2018, at 17:16, James Addison
mailto:add...@gmail.com>> wrote:
James -
again,
James
On Mon, Jun 25, 2018 at 9:38 AM James Harvey
mailto:jhar...@factset.com>> wrote:
Hi James,
The cachers in my setup, publish their discovery information every second to
the discovery proxies. I have maybe 100 catchers and the network overhead is
low compared with the
have more
discovery traffic (as mcast PUB/SUB have to filter on the SUB side) and you
need a mcast capable network.
James Harvey
From: zeromq-dev On Behalf Of Bill Torpey
Sent: 23 June 2018 21:29
To: ZeroMQ development list
Subject: Re: [zeromq-dev] zmq architecture/protocol planning
Hi
When you say “the system memory skyrockets” are you talking about the memory
used by the SUB process or PUB process?
I would expect the memory of the PUB process to increase as it will be queueing
messages.
> On 20 Jun 2018, at 21:02, Kevin Wang wrote:
>
> the system memory skyrockets,
. Maybe my Ubuntu is just messed up.
I assume since its CONFLATE is on, maybe the PUB doesn't queue it, but SUB
still does? In that case, I also would assume that replacing the old message
with the new one would free the memory held by the old message.
On 06/20/2018 01:38 PM, James Harvey wrote
Yes it does work over TCP and it does attempt to reconnect when you restart the
bind end.
In the background the zmq::tcp_connector attempts to make the reconnection. It
does all the right things and ends up in zmq::tcp_connecter_t::out_event and
calls the last line _socket->event_connected
the dead pipe much faster.
But in the end, PAIR is meant to be used between two endpoints, hence the name
- if one disappears and another arrives, it's no longer a pair, and as they
say, three's a crowd.
On Wed, 2018-06-20 at 14:33 +, James Harvey wrote:
> Yes it does work over TCP and it d
it. But until then,
maybe I should change that value to 1. What other penalties/trade offs are
there for changing it to 1?
> On Jun 21, 2018, at 4:39 AM, James Harvey wrote:
>
> Hi Kevin,
>
> I ran your test code and saw the same issue. I also created the tests with
> straigh
Hi,
Yes the message is atomic and you can assume all parts are available
immediately once the first part has arrived.
In general if I create anything that cares about blocking on the receive side I
always end up using zmq_poll() and handle it that way. Part of the reason being
it leaves it
development list
Subject: Re: [zeromq-dev] Abandon a multipart message?
On Wed, Aug 8, 2018 at 2:04 PM, James Harvey
mailto:jhar...@factset.com>> wrote:
You could send a zero length message with no (SND_MORE) flag to signify that’s
the last part.
I could, or I could send a part with a s
You could send a zero length message with no (SND_MORE) flag to signify that’s
the last part.
From: zeromq-dev On Behalf Of Al Grant
Sent: 08 August 2018 12:56
To: zeromq-dev@lists.zeromq.org
Subject: [zeromq-dev] Abandon a multipart message?
Is there a way to abandon a multipart message if
As a side note, having some method to call process_commands while idle
would also fix the memory usage issues encountered when using ZMQ_CONFLATE
and not reading from the socket.
https://github.com/zeromq/libzmq/issues/3171
I added documentaion to periodically call getsockopt with ZMQ_EVENTS but
ZMQ_PUB does not block which makes sense as you could end up with one slow
consumer slowing down everyone.
Maybe ZMQ_CONFLATE is what you need?
or If you only have one consumer try a different socket type that does
block.
On Wed, Aug 22, 2018 at 10:54 PM vincent freedom wrote:
> Not sure
t that doesn’t address the
> whole process_commands business).
>
> Last but not least, ZeroMQ is amazing stuff, and I don’t mean to sound
> ungrateful to or critical of the smart people who built and maintain it,
> but it’s part of my job to beat the stuffing out of any software that th
To be honest it's not a great test, the latency on each (essential) round
trip is going to kill the throughput measured in this way.
My point is this:
If you increase the size of the messages the throughput will increase as
less overall time (sender has to wait for less replies) spent waiting
Which side (send or recv) has the issue with the growing queue?
If it's the recv side and your "extremely slow" process is only calling
recv rarely then there may be no events to trigger the queue to check the
HWM.
Try calling zmq_getsockopt[3] with ZMQ_EVENT intermittently on the recv
socket to
ve ALL topics on a single iteration of the
> loop?
>
> On Thu, Sep 20, 2018 at 11:45 AM, James Harvey <
> jamesdillonhar...@gmail.com> wrote:
>
>> That’s actually just the libzmq wrapped in the cppzmq wrapper header (not
>> czmq). Just swap out each line for the C e
core C code? And not czmq, I am using capnproto-c with
> zmq. I could not get czmq to work with it.
>
> Thank you for your help
>
>
> On Wed, Sep 19, 2018 at 4:52 AM, James Harvey > wrote:
>
>> Also keep in mind the first byte of the message will be a 0/1 to gi
Also keep in mind the first byte of the message will be a 0/1 to give the
context.
So something like this:
zmq::message_t subscription;
pub_sock_->recv();
char* data = (char*)subscription.data();
// Subscriptions should start with 0x1 and unsubscriptions 0x0
if(*data != 0x1)
y the
data if there are no current subscriptions
Search up "late joiner" for possible solutions if that is a feature you
need to support.
On Wed, May 15, 2019 at 10:39 AM James Harvey
wrote:
> Hi Benjy,
>
> From your original example you are mixing binary for subscr
Hi Benjy,
>From your original example you are mixing binary for subscription subjects
and strings for publish subjects. Use the same for both.
The issue is not with the number of subscriptions, its the fact the
subjects you send do not match the subjects you subscribed to.
You std::string
ugh…
>
>
>
> *From:* James Harvey
> *Sent:* Wednesday, May 15, 2019 12:54 PM
> *To:* ZeroMQ development list
> *Cc:* Bekritsky, Benjamin
> *Subject:* Re: [zeromq-dev] max PUB/SUB topics
>
>
>
> [External Email]
>
> And regarding the c example where yo
The following needs to happen (in the background) before you call send().
1. The connect() establishes the tcp connection to the subscriber
2. The subscriber sends its subscription string to the publisher (filtering
is pub side on all protocols apart from PGM)
If send() happens before this the
Hi,
I do something similar in the “cloud”.
I have upstream load balancer connected to 3 proxies.
I have a downstream load balancer connected to 3 other proxies
All of the above are in a cloud scale set/group of 3 VM’s, each VM contains 1
upstream and 1 downstream proxy.
Using the magic of the
Hi
The system relies on quite a lot of cloud concepts, below I will use the azure
naming convention
When you say each VM contains 1 upstream and 1 downstream proxy - are these
running as two separate processes/applications?
* Yes two separate processes, the upstream proxy is slightly
I use epgm with zeromq and the c++ api all the time. I don't see the rate
issues you see.
Are you:
- Setting ZMQ_RATE before you call connect ?
- Are you setting ZMQ_RATE on both ends (pub/sub)
Other things to try:
- test using c/c++ rather than python to verify
Other things pgm related
Hi,
I have a rare/random bug that causes my ZMQ_SUB socket to fail for a
certain endpoint with no way to track/notify. Yes it's because a SUB
connects to a REQ socket but once you start to use zeromq for lots of
transient systems in a large company this kind of thing will happen
occasionally.
u’re running, but there have been some
> changes in that area not that long ago — check these out and see if they
> help:
>
> https://github.com/zeromq/libzmq/pull/3831
>
> https://github.com/zeromq/libzmq/pull/3960
>
> https://github.com/zeromq/libzmq/pull/4053
>
>
.
OR
Should we add a protocol error event to socket monitor so the calling
process can handle it by calling disconnect/connect
Just want some clarification so I work on the correct code.
Thanks
James
On Thu, May 13, 2021 at 4:48 PM James Harvey
wrote:
> Hi,
>
> I have a rare/r
27b0364be80de4451bf1a80a26c00d0ba9310f/src/transport.c#L1549
>
> I’ll try to take a look when I have some time, but not sure when that will
> be …
>
> Regards,
>
> Bill
>
> On May 21, 2021, at 10:04 AM, James Harvey
> wrote:
>
> Thanks Bill
>
&g
mq/zmtpdump) or wireshark to see what is really
> going on.
>
> Last but not least, you are likely better off creating an issue on GitHub
> for this.
>
> Regards,
>
> Bill
>
>
> On May 21, 2021, at 2:38 PM, James Harvey
> wrote:
>
> Hi Bill,
>
> I will check/
hat is really
> going on.
>
> Last but not least, you are likely better off creating an issue on GitHub
> for this.
>
> Regards,
>
> Bill
>
>
> On May 21, 2021, at 2:38 PM, James Harvey
> wrote:
>
> Hi Bill,
>
> I will check/reply to rest of poin
If you are only interested in synchronising flow with the first subscriber
you can use an XPUB socket. With XPUB you can read the actual subscription
message off the pub socket so you know for sure the subscriber is connected
(and the subscribe subject)
If you want to know when duplicate
34 matches
Mail list logo
