[zeromq-dev] imatix deleted?
Hi, Apologies if this is inappropriate for this list, but the imatix.com site (linked to from the zeromq home page) appears to have been deleted. Anyone know why? Cim ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] imatix deleted?
Hi Cim, Apologies if this is inappropriate for this list, but the imatix.com site (linked to from the zeromq home page) appears to have been deleted. Anyone know why? It was a glitch somewhere due to us moving DNS entries around. All works again now. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] imatix deleted?
On Mon, Sep 24, 2012 at 2:05 PM, Pieter Hintjens p...@imatix.com wrote: Hi Cim, Apologies if this is inappropriate for this list, but the imatix.com site (linked to from the zeromq home page) appears to have been deleted. Anyone know why? It was a glitch somewhere due to us moving DNS entries around. All works again now. It smells like wikidot migration :-) http://blog.wikidot.com/blog:wikidot-moving -- Benjamin Henrion bhenrion at ffii.org FFII Brussels - +32-484-566109 - +32-2-3500762 In July 2005, after several failed attempts to legalise software patents in Europe, the patent establishment changed its strategy. Instead of explicitly seeking to sanction the patentability of software, they are now seeking to create a central European patent court, which would establish and enforce patentability rules in their favor, without any possibility of correction by competing courts or democratically elected legislators. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: !(msg_-flags () msg_t::more)(session_base.cpp:157)
Ian Barber ian.barber at gmail.com writes: On Fri, Sep 21, 2012 at 6:15 PM, RohanB rohanb at cs.uchicago.edu wrote: Its going to be really hard to reproduce this one. It works perfectly on one box and not the other. Same build. Here's the difference though. The box on which it works is 64 bit SuSe. The one where it doesn't is a 32 bit SuSe. By the way, the publisher works perfectly on both boxes. Its just the subscriber that has issues. How consistently does it fail when setup in the failing configuration? If it's reliably enough it would be interesting perhaps to get a packet capture between the two (possibly one the other away as well to look for differences). There were some issues a while ago where the identity could get lost if there were failures of a certain type - I know those were resolved, but it would be interesting to see if there issue is in the handling of the message or the sending. Ian Hi Ian Thanks for the follow up. Yes, it consistently fails when setup on the failing box. Note again that the publisher always, its just the subscriber that just won't come up. I doubt we need a packet capture here because the entire thing happens before any package exchange takes place. What I mean is there is no publisher running when I bring up the subscriber on the failing box. Even then its aborts with the assertion. You see what I mean? I am pretty certain I need some configuration on the failing box that the other box has and I just can't figure out what. Could you give me some info on the earlier issue you mentioned and what the assert means? I would be happy to help resolve this issue in any way I can. Rohan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: !(msg_-flags () msg_t::more)(session_base.cpp:157)
On Mon, Sep 24, 2012 at 3:07 PM, RohanB roh...@cs.uchicago.edu wrote: the entire thing happens before any package exchange takes place. What I mean is there is no publisher running when I bring up the subscriber on the failing box. Even then its aborts with the assertion. You see what I mean? Ah, that's interesting. Could you post up a code sample of a stand-along subscriber that works on one box and not the other? I am pretty certain I need some configuration on the failing box that the other box has and I just can't figure out what. Could you give me some info on the earlier issue you mentioned and what the assert means? It's just attempting to pick the identity out of the message iirc, which potentially means ZMQ is receiving message data when it shouldn't be - certainly if no one has actually started sending! Ian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: !(msg_-flags () msg_t::more)(session_base.cpp:157)
Ian Barber ian.barber at gmail.com writes:
On Mon, Sep 24, 2012 at 3:07 PM, RohanB rohanb at cs.uchicago.edu wrote:
the entire thing
happens before any package exchange takes place. What I mean is there is no
publisher running when I bring up the subscriber on the failing box. Even
then
its aborts with the assertion. You see what I mean?
Ah, that's interesting. Could you post up a code sample of a
stand-along subscriber that works on one box and not the other?
I am pretty certain I need some configuration on the failing box that the
other
box has and I just can't figure out what. Could you give me some info on the
earlier issue you mentioned and what the assert means?
It's just attempting to pick the identity out of the message iirc,
which potentially means ZMQ is receiving message data when it
shouldn't be - certainly if no one has actually started sending!
Ian
ahh.. I see what you mean. Here's the snippet Pretty straightforward...
Same code works on 1 box and not the other. Both boxes are enabled to
receive data on that multicast address and there is no other publisher
currently
running.
int main (int argc, char* argv[])
{
if (argc 2)
{
std::cout \nUsage: pgm_sub [topic_name]\n(for wildcard: pgm_sub
all)\n std::endl;;
return 0;
}
zmq::context_t context (1);
zmq::socket_t subscriber(context, ZMQ_SUB);
subscriber.connect(epgm://10.34.253.16;239.196.1.1:16000);
const char* topic = strcmp(argv[1], all) == 0 ? \x00 : argv[1];
subscriber.setsockopt(ZMQ_SUBSCRIBE, topic, strlen(topic));
int rate = 1;
subscriber.setsockopt(ZMQ_RATE, rate, sizeof(rate));
while (1)
{
zmq::message_t update;
subscriber.recv(update);
char* msg_buffer = reinterpret_castchar*(update.data());
std::string topic_name(msg_buffer);
books::Price priceUpdate;
priceUpdate.ParseFromArray(msg_buffer+28,450);
print_price(priceUpdate);
}
return 0;
}
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Visibility into pipes of a socket
Thanks for the response. Comments inline On 9/21/12 12:05 AM, Pieter Hintjens wrote: On Thu, Sep 20, 2012 at 9:40 PM, Stuart Brandt s...@compuserve.com wrote: All thoughts welcome! Very roughly, trying to do this kind of thing is why it takes people months to write even basic protocols over TCP. No layering. I'm a little puzzled on this statement. The API for 3.x has peer address transiting both into and out of the 0MQ library layer. Exposing the transport origin of a received message using the same philosophy as zmq_ctx_set_monitor (an after-the-fact thing intended for monitoring and operational concerns) doesn't strike me as a layering violation given 0MQ's existing APIs. Logging IP addresses in the server? Absolutely! Operational auditing, monitoring, and troubleshooting is a real PITA without it. Peer address info, or at least peer IP, is pretty much standard monitoring info in most distributed apps I've seen. Measuring latency of servers? Yes. Again this is one of those operational kinds of things. Measuring server latency is a valuable tool for identifying issues that need to be raised to operations staff. Most of what you want to achieve can be done easily above 0MQ by ignoring the physical network and talking application to application. Heartbeats from servers to clients. Application-level identifiers. Agreed...but only if you assume a well behaved app-level that never runs into version mismatch problems, or questionable/rogue peers, or word boundary alignment problems, or any of a variety of application problems that could pollute an app-level identifier. There are a lot of great lessons in The Guide, and in this case the part on reliability that cites application code as the worst offender when it comes to reliability seems to apply. I'm not looking to guard against a hacker working at the TCP or even ZMTP layer here. Protection against that is well beyond the scope of 0MQ. I am, however, looking to make sure that when some new client code makes it into production with a buffer overwrite bug that my server can log/alarm with enough specifics to be useful. Given the choice between 2012-09-24 14:25:51,644 ERROR - Unrecognized request from peer 192.0.0.1:12234 and 2012-09-24 14:25:51,644 ERROR - Unrecognized request from peer #^2ffwi23r098vnasdf0 which would you prefer to use for quickly finding the client instance that went bad? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] REQ/REP newbie question
I think this question is so common, that it should belong in the general ZeroMQ FAQ page on the website; as I am sure the guide already covers this topic. :) Such a FAQ entry could be constructed from the various constructive replies provided on this forum. Just an idea. - HP On 01/09/2012 02:51, Jeremy Selan wrote: Thanks for the pointer. Dealer/router looks like exactly what I'd like to use. Thanks! On Fri, Aug 31, 2012 at 12:46 PM, Amr Ali amr.ali...@gmail.com wrote: Have you tried to use DEALER/ROUTER pattern (used to be called XREQ/XREP)? I'd like to think of DEALER/ROUTER as the primitive for REQ/REP as it doesn't constrain you having to reply to every request but of course you'll have to manage socket IDs yourself to know whom to reply to only if needed. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_poll versus zloop reactor: different semantics - how to resolve?
On Mon, Sep 24, 2012 at 2:00 PM, Michael Haberler mai...@mah.priv.at wrote: I have an application where a named pipe, and a sysfs device is monitored. I did the prototype with libzmq - which worked fine - and the switched to czmq, which broke things for me. the issue is: czmq is too agresssive in shutting down a poll handler if a POLLERR is returned. Could be. I dont like it, but am unsure what to do; fiddling with the inner loop of the reactor I dont feel confident about yet. It's a little involved but not so complex so my advice would be to look at the internals, propose a patch. You do want to provide a test case so people can see the before/after results. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] obtain generated identity of socket
Hi, I'm using ROUTER-ROUTER connections for one-way communication. How can the sending peers identify the destination peers? Currently I'm generating peer ids and telling the senders out of band (using a different socket), but the destination ROUTER sockets are not yet aware of these ids, so messages don't get delivered. I think I could use ZMQ_IDENTITY to set my generated ids on the destination ROUTER sockets, yes? The sending peers use the envelope format, where the first part of each message is the destination peer id. I recall reading somewhere that setting ZMQ_IDENTITY was being discouraged, but maybe that was only in the context of durability? Is there a way for me to create a ROUTER socket and then ask it for its generated id, instead of me making one? The docs seem to indicate that by default the socket would have no identity, so I'm not sure if reading the default ZMQ_IDENTITY value is what I want. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] REQ/REP newbie question
On Mon, Sep 24, 2012 at 2:39 PM, hp0101070 hp010...@gmail.com wrote: I think this question is so common, that it should belong in the general ZeroMQ FAQ page on the website; as I am sure the guide already covers this topic. :) Such a FAQ entry could be constructed from the various constructive replies provided on this forum. Go for it... it's a wiki :) -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Compatibility between 2.0.10 and 2.2.0 and 3.2
Hi There, We have been on 2.0.10 for the past year and a half and are planning to upgrade to 2.2.0 first and then move to 3.2 shortly after that. From a wireformat perspective, are 2.0.10 and 2.2.0 compatible? My understanding is that, yes, they both use the same ZMTP/1.0. And that 3.2 will use an improved ZMTP/2.0 protocol. Is my understanding correct? Any additional tips or suggestions on running 2.0.10 and 2.2.0 on the same network? Thanks, Tom ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] obtain generated identity of socket
On Mon, Sep 24, 2012 at 2:42 PM, Justin Karneges jus...@affinix.com wrote: I'm using ROUTER-ROUTER connections for one-way communication. How can the sending peers identify the destination peers? The only sane way I found was to use the server endpoints as agreed identities, i.e. if you're binding to 192.168.55.131:, then use that as your identity when connecting. This lets the other router talk to you. But mostly router-to-router seems like a bad pattern. I'd rather do two connections, one dealer-router in each direction, so each peer plays server and client cleanly. Just simpler to understand. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Compatibility between 2.0.10 and 2.2.0 and 3.2
On Mon, Sep 24, 2012 at 3:34 PM, Tom Wilberding t...@wilberding.com wrote: We have been on 2.0.10 for the past year and a half and are planning to upgrade to 2.2.0 first and then move to 3.2 shortly after that. Is my understanding correct? Any additional tips or suggestions on running 2.0.10 and 2.2.0 on the same network? I think they share the same wire protocol. The API did change. Going from 2.0.10 to 2.2.0 to 3.2 seems like a good plan but I'd test it, and if you hit any issues, raise them with us. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] obtain generated identity of socket
On Monday, September 24, 2012 04:24:06 PM Pieter Hintjens wrote: On Mon, Sep 24, 2012 at 2:42 PM, Justin Karneges jus...@affinix.com wrote: I'm using ROUTER-ROUTER connections for one-way communication. How can the sending peers identify the destination peers? The only sane way I found was to use the server endpoints as agreed identities, i.e. if you're binding to 192.168.55.131:, then use that as your identity when connecting. This lets the other router talk to you. Makes sense. I've currently made it work by generating a UUID (in my app code) and using that as the identity. This identity is sent using a separate socket to the client. But mostly router-to-router seems like a bad pattern. I'd rather do two connections, one dealer-router in each direction, so each peer plays server and client cleanly. Just simpler to understand. Here's my use-case; maybe you can critique it: I want to send a stream of data to a server instance (of which there may be multiple), and I want to ensure all the messages for a given stream go to the same server. The server uses 3 sockets: in: PULL socket, for retrieving start requests out: PUB socket, for sending notifications (ignore why this is PUB, for now) in_stream: ROUTER socket, for retrieving append messages. this socket is given an explicit identity. Protocol flow goes like this: 1) client pushes a start request that gets picked up by one of the server instances 2) server pubs a clear to send response that includes a reply address field, containing the value of the in_stream identity 3) client sends a series of messages to in_stream, addressed using the reply address In other words, PUSH deals to an arbitrary server, but once the server acks the request, the client locks on to that server and sends the remainder of the stream specifically to that server. For flow control, the server regularly grants credits to the client via the pub socket. The client writing to in_stream should use a ROUTER socket for itself, since it is using the enveloped addressing that requires routing. The server, on the other hand, could be DEALER or ROUTER, I suppose. Both terms seem misleading for a socket that never gets written to, but ROUTER seemed slightly less so as it is the most raw. Maybe it should be possible to connect ROUTER to PULL? :) Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Compatibility between 2.0.10 and 2.2.0 and 3.2
On Sep 24, 2012, at 3:34 PM, Tom Wilberding wrote: Hi There, We have been on 2.0.10 for the past year and a half and are planning to upgrade to 2.2.0 first and then move to 3.2 shortly after that. From a wireformat perspective, are 2.0.10 and 2.2.0 compatible? My understanding is that, yes, they both use the same ZMTP/1.0. And that 3.2 will use an improved ZMTP/2.0 protocol. Is my understanding correct? Any additional tips or suggestions on running 2.0.10 and 2.2.0 on the same network? If you are expending effort to upgrade from 2.0.10, then please go to 3.2. There is no advantage in doing an interim upgrade to 2.2.0 and then moving to 3.2. The primary effort is in testing; do it on 3.2. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] obtain generated identity of socket
Hi Justin, On Tue, Sep 25, 2012 at 1:08 AM, Justin Karneges jus...@affinix.com wrote: Protocol flow goes like this: 1) client pushes a start request that gets picked up by one of the server instances 2) server pubs a clear to send response that includes a reply address field, containing the value of the in_stream identity 3) client sends a series of messages to in_stream, addressed using the reply address As your transfers seems big, I think it's ok to open separate (push) socket for each transfer. (Or just have a separate socket connected to each server). I.e. in the clear to send message server gives and address where to connect to for sending message. Creating sockets for each transfer is antipattern if you are doing it hundred times per second, but for your use case it may be cleaner than router to router connection. -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] obtain generated identity of socket
On Tuesday, September 25, 2012 08:18:03 AM Paul Colomiets wrote: Hi Justin, On Tue, Sep 25, 2012 at 1:08 AM, Justin Karneges jus...@affinix.com wrote: Protocol flow goes like this: 1) client pushes a start request that gets picked up by one of the server instances 2) server pubs a clear to send response that includes a reply address field, containing the value of the in_stream identity 3) client sends a series of messages to in_stream, addressed using the reply address As your transfers seems big, I think it's ok to open separate (push) socket for each transfer. (Or just have a separate socket connected to each server). I.e. in the clear to send message server gives and address where to connect to for sending message. Creating sockets for each transfer is antipattern if you are doing it hundred times per second, but for your use case it may be cleaner than router to router connection. I do need this to support hundreds-of-times-per-second capacity, and the stream sizes may vary wildly (could be one message, could be 200MB of messages). Your idea of the client having separate PUSH sockets to each server could work as a substitute for the single ROUTER socket. But I do think ROUTER is logical, at least on the client side. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
