Re: [zeromq-dev] Using Strawhouse security pattern with ZeroMQ
czmqpp wraps CZMQ so provides the same classes. You use zauth to do the authentication. On Mon, Dec 15, 2014 at 11:40 PM, Check Peck comptechge...@gmail.com wrote: Hi Pieter, Is there any example for Strawhouse security pattern using C++ czmq wrapper? This link http://hintjens.com/blog:49 only talks about c way of doing it. I found github repository https://github.com/zeromq/czmqpp which looks like it's a C++ wrapper for czmq but not able to find any example how to use Strawhouse security pattern. On Mon, Dec 15, 2014 at 1:21 PM, Check Peck comptechge...@gmail.com wrote: Ok got it. I have another question on zauth which I have asked separately in another question with the subject name. How to use ZeroMQ context with zauth? See if you can help me out. On Mon, Dec 15, 2014 at 1:17 PM, Pieter Hintjens p...@imatix.com wrote: You can't white/blacklist on domain names without a lot more work. The zauth class uses the IP address as provided by the network. On Mon, Dec 15, 2014 at 9:57 PM, Check Peck comptechge...@gmail.com wrote: Thanks Pieter, Yes it worked fine after I remove older version of libzmq. One question I have on Strawhouse pattern is - Does it always work with IP Address? I cannot use hostname to white list it? If I try to replace 127.0.0.1 with localhost or the actual machine name, then it doesn't work. zauth_allow (auth, 127.0.0.1); // this works fine zauth_allow (auth, localhost); // this doesn't works zauth_allow (auth, machineA.dev.com); // this doesn't works // The Strawhouse Pattern // // We allow or deny clients according to their IP address. It may keep // spammers and idiots away, but won't stop a real attacker for more // than a heartbeat. #include czmq.h int main (void) { // Create context zctx_t *ctx = zctx_new (); // Start an authentication engine for this context. This engine // allows or denies incoming connections (talking to the libzmq // core over a protocol called ZAP). zauth_t *auth = zauth_new (ctx); // Get some indication of what the authenticator is deciding zauth_set_verbose (auth, true); // Whitelist our address; any other address will be rejected zauth_allow (auth, 127.0.0.1); // Create and bind server socket void *server = zsocket_new (ctx, ZMQ_PUSH); zsocket_set_zap_domain (server, global); zsocket_bind (server, tcp://*:9000); // Create and connect client socket void *client = zsocket_new (ctx, ZMQ_PULL); zsocket_connect (client, tcp://127.0.0.1:9000); // Send a single message from server to client zstr_send (server, Hello); char *message = zstr_recv (client); assert (streq (message, Hello)); free (message); puts (Strawhouse test OK); zauth_destroy (auth); zctx_destroy (ctx); return 0; } On Sat, Dec 13, 2014 at 1:04 AM, Pieter Hintjens p...@imatix.com wrote: You presumably have two versions of libzmq installed on your system, and gcc is complaining they both have the same symbols. I'd recommend removing the older version. On Sat, Dec 13, 2014 at 1:29 AM, Check Peck comptechge...@gmail.com wrote: I am trying to use Strawhouse security pattern in my zero-mq development. I was following this wiki http://hintjens.com/blog:49 and when I try to run below simple program to make sure I have everything installed, I got an error - #include czmq.h int main (void) { zctx_t *ctx = zctx_new (); void *publisher = zsocket_new (ctx, ZMQ_PUB); zsocket_set_curve_server (publisher, true); puts (Hello, Curve!); zctx_destroy (ctx); return 0; } I tried to compile it like this - gcc -o hello hello.c -lczmq -lzmq -lsodium And the error I got - /usr/bin/ld: warning: libzmq.so.4, needed by /usr/local/lib/libczmq.so, may conflict with libzmq.so.3 Does anyone know what does this mean and what wrong I am doing? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How to use ZeroMQ context with zauth?
You can't cast a libzmq context pointer to a CZMQ class reference. These are two different types. The shadow_ctx method does a conversion. The use of contexts is deprecated in CZMQ/3.x. On Tue, Dec 16, 2014 at 1:30 AM, Check Peck comptechge...@gmail.com wrote: Thanks a lot. Yes it worked after that change. Can you explain little bit what does that line do and what it means in general? On Mon, Dec 15, 2014 at 4:26 PM, KIU Shueng Chuan nixch...@gmail.com wrote: zctx_t *ctx_ptr = zctx_shadow_zmq_ctx ((void *)context); should do what you want. On 16 Dec 2014 06:29, Check Peck comptechge...@gmail.com wrote: I need to use ZeroMQ context from C++ binding to C way of ZeroMQ context so when I do it like this, I get a Segmentation fault zctx_t* ctx_ptr = static_castzctx_t*((void*)context); m_auth = zauth_new (ctx_ptr); I am using Strawhouse security pattern by following this link - http://hintjens.com/blog:49 On Mon, Dec 15, 2014 at 1:20 PM, Check Peck comptechge...@gmail.com wrote: I am using C++ bindings for ZeroMQ and I have declared context as - zmq::context_t context zauth_t *m_auth; Now I am trying to use Strawhouse security pattern in my application and for which I need to czmq which is a C binding not C++ binding. Now when I try to initialize zauth like below, it always fail during compilation and I am not sure what's wrong? m_auth = zauth_new (*context); Below is the compilation error - In file included from /usr/local/include/boost/detail/container_fwd.hpp:98:0, from /usr/local/include/boost/functional/hash/extensions.hpp:17, from /usr/local/include/boost/functional/hash/hash.hpp:529, from /usr/local/include/boost/functional/hash.hpp:6, from /usr/local/include/boost/regex/v4/basic_regex.hpp:23, from /usr/local/include/boost/regex/v4/regex.hpp:67, from /usr/local/include/boost/regex.hpp:31, from /home/david/ZeroMQTester/test_queue.cpp:7: /usr/include/c++/4.7/complex:379:5: note: templateclass _Tp std::complex_Tp std::operator*(const std::complex_Tp, const std::complex_Tp) /usr/include/c++/4.7/complex:379:5: note: template argument deduction/substitution failed: /home/david/ZeroMQTester/test_queue.cpp:126:26: note: âzmq::context_tâ is not derived from âconst std::complex_Tpâ ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_poll performance question
Hi, On 12/12/2014 5:00 PM, Arnaud Kapp wrote: Hey, I did not check for performance improvement, I simply tested that my program continued to work. The thing is, my poller only has 1 zmq socket and 1 file descriptor (that relies on POLLPRI -- its a GPIO pin). So in my case I could'nt really check for perf improvement. Also I am running this on a Raspberry Pi. Of course with only two descriptors the performance improvement will be negligible. I used it for about 1hour before sending the previous mail. Gonna work with this patch applied next week, making sure it keeps working. Thank you very much for testing it, at least it doesn't break anything at first glance. Cheers, Francis On Thu, Dec 11, 2014 at 2:39 PM, Francis Le Bourse zno-reply-francis.lebou...@sfr-sh.fr wrote: Hello, On 12/10/2014 6:54 PM, Arnaud Kapp wrote: Hello, Sorry it took me a while, but I finally go to test your patch. My setup that use POLLPRI seems to work properly with your patch applied :). Good. Do you see a performance improvement ? How long have you been using it ? Did you submit a PR to get it merged into libzmq master yet? No, not yet. I was waiting for feedback before. And I had another issue with a memory hog in libzmq. Cheers, Francis On Fri, Nov 28, 2014 at 11:35 AM, Francis Le Bourse zno-reply-francis.lebou...@sfr-sh.fr wrote: On 11/24/2014 8:08 PM, Arnaud Kapp wrote: Currently, the patch is written for 3.2.4. I'll wait to put it in libzmq master The first patch for 3.2.4 had an issue in zmq_poll(), I had tried a little too aggressive optimization by bypassing the first_pass processing. It is fixed in the current one. The patch for the current head is clean. Cheers, Francis Oh okay. This is the commit that added the flag: https://github.com/zeromq/libzmq/commit/779c37abc433cb6595ddeedaf86b280317656bdd libzmq was 4.1 at the time I believe. I'll probably look at it this week-end then :) On Mon, Nov 24, 2014 at 12:10 PM, Francis Le Bourse zno-reply-francis.lebou...@sfr-sh.fr wrote: Hi, On 11/24/2014 11:35 AM, Arnaud Kapp wrote: Hello, I recently added support for POLLPRI flag. It looks like it's not handled in your patch No, it isn't handled. In which version do you have added this flag ? Currently, the patch is written for 3.2.4. I'll wait to put it in libzmq master. and that it needs custom support. Since there is no test related to this flags you wouldn't notice. I can give it a look if you want. That would be nice. Cheers, Francis On Sat, Nov 22, 2014 at 2:16 PM, Pieter Hintjens p...@imatix.com wrote: I suggest you send the patch to libzmq master, and ensure all test cases pass. Then we can get this into the next version. On Fri, Nov 21, 2014 at 2:50 PM, Francis Le Bourse zno-reply-francis.lebou...@sfr-sh.fr wrote: Hi, On 11/6/2014 3:18 PM, Pieter Hintjens wrote: Oh, ok. Sounds like you have a good candidate for some before/after measurement and optimization. Are you going to try to make a patch for this? I have a patch candidate for this optimization, the performance improvement is very good and it doesn't seem to introduce any new instability. What is modified: - zmq_poll(), there is only one poll() now, - and epoll() from epoll.cpp Other calls to poll() and select() are left unmodified. I woulld be happy to have any feedback. Cheers, Francis On Thu, Nov 6, 2014 at 2:09 PM, Francis Le Bourse zno-reply-francis.lebou...@sfr-sh.fr wrote: On 11/6/2014 11:47 AM, Pieter Hintjens wrote: A simple optimization is, when you are polling sockets for input, to continue reading from an active socket using a non-blocking read. So you process all waiting messages on a socket and then only switch back to poll when needed. Thank you for you quick reply. Yes, but the question was more about the zmq_poll() internals. For 600+ file descriptors, zmq_poll() calls poll() a huge number of times for only a few that will trigger a POLLIN and the relevant information is already known / present in the pollfds array. The performance hit is there. Cheers, Francis On Thu, Nov 6, 2014 at 11:28 AM, Francis Le Bourse zno-reply-francis.lebou...@sfr-sh.fr wrote: Hi, I am looking at a performance issue in zmq, when the number of zsockets / file descriptors becomes large. The relevant calls are: poll+0x57 zmq_poll+0x2e3 zloop_start+0x1e8 main+0xb40 __libc_start_main+0xfd immediately followed by a loop of poll+0x57 zmq::signaler_t::wait(int)+0x33 zmq::mailbox_t::recv(zmq::command_t*, int)+0x78 zmq::socket_base_t::process_commands(int, bool)+0xbe zmq::socket_base_t::getsockopt(int, void*, unsigned long*)+0x135 zmq_getsockopt+0x75 zmq_poll+0x3da zloop_start+0x1e8 main+0xb40 __libc_start_main+0xfd The code in the loop is executed
[zeromq-dev] Router socket reconnection failure
Hi all, I'm experimenting with a router-router setup and I'm getting a strange issue when peers reconnect. Basically, I have three nodes, which I'll call D, P1 and P2. The idea is that D has a known TCP endpoint and socket identity. P1 and P2 connect to D, register their TCP endpoint and identify and then discover each other through D (the directory). At this point, one of them connects to the other and they become peers. Through heartbeating, they can successfully detect connections and disconnections of the other peer. Because the topology is dynamic and volatile, peers explicitly disconnect when they detect that one of their peers is unresponsive. So far, my prototype implementations of programs for D and P* are working as intended. The issue I'm having is with this sequence: - P1 and P2 discover each other through D; - P1 connects to P2 and P2 waits for a connection from P1 (direction is determined by lexicographical ordering of identities, which both peers have prior to connecting); - Peers exchange heartbeats for a while; - I forcibly crash P2; - P1 eventually detects that P2 is unresponsive and explicitly disconnects; - after this happens, I restart P2; - P1 and P2 discover each other through D again; - P1 tries to connect to P2 and P2 expects a connection from P1; - both peers send heartbeats, but neither peer receives the other's messages and it appears the connection is never established. Also note that after this has happened, context termination hangs despite closing the (only) socket and setting the linger to 1 second. If I crash P1 instead of P2, the reconnection is successful. Also, if after the error sequence above I crash P1, peers reconnect successfully. As far as I can tell, the problem seems to be that a sequence of zmq_connect(), zmq_disconnect() and zmq_connect() on the same router socket and with the same endpoint corrupts the router socket. Has anyone encountered this issue before? I'm using ZMQ 4.1.0 via the PyZMQ bindings. I may be able to work out a minimalist repro if necessary. Thanks, André ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] zeromq4-1 build errors using Visual Studio solution.
I've cloned https://github.com/zeromq/zeromq4-1 and copied it to the location C:\zeromq4-1 on my machine. I've then opened the Visual Studio 2013 solution and tried to build zeromq using either the 'StaticRelease' or 'DynamicRelease' settings. Either way I get errors, such as listed below. Does anyone please know how to fix this? Thanks, Riskybiz. 1-- Build started: Project: libzmq, Configuration: ReleaseLIB Win32 -- 1 ConfigurationType : StaticLibrary 1 Configuration : ReleaseLIB 1 PlatformToolset : v120 1 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\libzmq\..\..\..\..\bin\Win32\Release\v120\st atic\libzmq.lib 1 Option-openpgm: 1 Option-sodium : true 1 Option-gssapi : 1 Linkage-libsodium : static 1 address.cpp 1 clock.cpp 1 ctx.cpp 1 curve_client.cpp 1 curve_server.cpp 1 dealer.cpp 1 devpoll.cpp 1 dist.cpp 1 epoll.cpp 1 err.cpp 1 fq.cpp 1 gssapi_client.cpp 1 gssapi_mechanism_base.cpp 1 gssapi_server.cpp 1 io_object.cpp 1 io_thread.cpp 1 ip.cpp 1 ipc_address.cpp 1 ipc_connecter.cpp 1 ipc_listener.cpp 1 kqueue.cpp 1 lb.cpp 1 mailbox.cpp 1 mechanism.cpp 1 metadata.cpp 1 msg.cpp 1 mtrie.cpp 1 null_mechanism.cpp 1 object.cpp 1 options.cpp 1 own.cpp 1 pair.cpp 1 pgm_receiver.cpp 1 pgm_sender.cpp 1 pgm_socket.cpp 1 pipe.cpp 1 plain_client.cpp 1 plain_server.cpp 1 poll.cpp 1 poller_base.cpp 1 precompiled.cpp 1 proxy.cpp 1 pub.cpp 1 pull.cpp 1 push.cpp 1 random.cpp 1 raw_decoder.cpp 1 raw_encoder.cpp 1 reaper.cpp 1 rep.cpp 1 req.cpp 1 router.cpp 1 select.cpp 1 session_base.cpp 1 signaler.cpp 1 socket_base.cpp 1 socks.cpp 1 socks_connecter.cpp 1 stream.cpp 1 stream_engine.cpp 1 sub.cpp 1 tcp.cpp 1 tcp_address.cpp 1 tcp_connecter.cpp 1 tcp_listener.cpp 1 thread.cpp 1 trie.cpp 1 v1_decoder.cpp 1 v1_encoder.cpp 1 v2_decoder.cpp 1 v2_encoder.cpp 1 xpub.cpp 1 xsub.cpp 1 zmq.cpp 1 zmq_utils.cpp 1..\..\..\..\src\zmq.cpp(631): warning C4244: 'return' : conversion from 'int64_t' to 'int', possible loss of data 1 libzmq.vcxproj - C:\zeromq4-1\builds\msvc\vs2013\libzmq\..\..\..\..\bin\Win32\Release\v120\st atic\libzmq.lib 2-- Build started: Project: inproc_thr, Configuration: ReleaseSEXE Win32 -- 3-- Build started: Project: inproc_lat, Configuration: ReleaseSEXE Win32 -- 4-- Build started: Project: remote_thr, Configuration: ReleaseSEXE Win32 -- 5-- Build started: Project: remote_lat, Configuration: ReleaseSEXE Win32 -- 2 ConfigurationType : Application 2 Configuration : ReleaseSEXE 2 PlatformToolset : v120 2 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\inproc_thr\..\..\..\..\bin\Win32\Release\v12 0\static\inproc_thr.exe 2 Linkage-libzmq: static 2 Linkage-libsodium : static 2 inproc_thr.cpp 3 ConfigurationType : Application 3 Configuration : ReleaseSEXE 3 PlatformToolset : v120 3 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\inproc_lat\..\..\..\..\bin\Win32\Release\v12 0\static\inproc_lat.exe 3 Linkage-libzmq: static 3 Linkage-libsodium : static 3 inproc_lat.cpp 4 ConfigurationType : Application 4 Configuration : ReleaseSEXE 4 PlatformToolset : v120 4 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\remote_thr\..\..\..\..\bin\Win32\Release\v12 0\static\remote_thr.exe 4 Linkage-libzmq: static 4 Linkage-libsodium : static 4 remote_thr.cpp 5 ConfigurationType : Application 5 Configuration : ReleaseSEXE 5 PlatformToolset : v120 5 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\remote_lat\..\..\..\..\bin\Win32\Release\v12 0\static\remote_lat.exe 5 Linkage-libzmq: static 5 Linkage-libsodium : static 5 remote_lat.cpp 2LINK : fatal error LNK1181: cannot open input file 'libzmq.lib' 3LINK : fatal error LNK1181: cannot open input file 'libzmq.lib' 6-- Build started: Project: local_thr, Configuration: ReleaseSEXE Win32 -- 6 ConfigurationType : Application 6 Configuration : ReleaseSEXE 6 PlatformToolset : v120 6 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\local_thr\..\..\..\..\bin\Win32\Release\v120 \static\local_thr.exe 6 Linkage-libzmq: static 6 Linkage-libsodium : static 6 local_thr.cpp 7-- Build started: Project: local_lat, Configuration: ReleaseSEXE Win32 -- 4LINK : fatal error LNK1181: cannot open input file 'libzmq.lib' 7 ConfigurationType : Application 7 Configuration : ReleaseSEXE 7 PlatformToolset : v120 7 TargetPath: C:\zeromq4-1\builds\msvc\vs2013\local_lat\..\..\..\..\bin\Win32\Release\v120 \static\local_lat.exe 7 Linkage-libzmq: static 7 Linkage-libsodium : static 7 local_lat.cpp 6LINK : fatal error LNK1181: cannot open input file 'libzmq.lib' 5LINK : fatal error LNK1181: cannot open input file 'libzmq.lib' 7LINK : fatal error LNK1181: cannot open input file
Re: [zeromq-dev] Router socket reconnection failure
Hi Andre, On Tue, Dec 16, 2014, at 07:14 AM, Andre Caron wrote: The issue I'm having is with this sequence: - P1 and P2 discover each other through D; - P1 connects to P2 and P2 waits for a connection from P1 (direction is determined by lexicographical ordering of identities, which both peers have prior to connecting); - Peers exchange heartbeats for a while; - I forcibly crash P2; - P1 eventually detects that P2 is unresponsive and explicitly disconnects; - after this happens, I restart P2; - P1 and P2 discover each other through D again; - P1 tries to connect to P2 and P2 expects a connection from P1; - both peers send heartbeats, but neither peer receives the other's messages and it appears the connection is never established. Also note that after this has happened, context termination hangs despite closing the (only) socket and setting the linger to 1 second. If I crash P1 instead of P2, the reconnection is successful. Also, if after the error sequence above I crash P1, peers reconnect successfully. This is a known issue, and I reported it earlier this year: http://lists.zeromq.org/pipermail/zeromq-dev/2014-February/025202.html I believe the problem is that once a connector queue learns the ID of a remote address, this binding sticks for life. The reason that you can restart P1 and things work is because connectors maintain queues even if there are no connections, but binders don't. Unfortunately I haven't had time yet to look at a fix. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Router socket reconnection failure
Hi Justin, Thanks for the info :-) Just read that thread, but the case seems slightly different: all my nodes use a persistent identity, which I set immediately after creating the socket and thus before any bind or connect operation. However, I just tried having P2 restart with a new identity and I get the same problem. I'm really confused by the answers from Laurent near the end of the thread. It seems to me like the whole point of the identity socket option is to send the string to the peer so that it can resume a session across multiple TCP connections and/or process executions. It also seems to me like if it doesn't work in this scenario, then the identity's only purpose would be for debugging purposes. In addition, nothing I've seen so far explains the fact that this scenario causes zmq_term() to hang forever despite closing all sockets and setting a non-zero linger value, which is clearly a bug. I tried playing around with my code a bit more. Using ZMQ 4.0.5, I get the error. If I switch to ZMQ 4.1.0, the peers reconnect, but I zmq_term() still hangs as soon as P1 reconnects to P2. I don't know what was fixed between those two releases, but something almost fixed the problem! If it's of any help, setting the ZMQ_ROUTER_HANDOVER option to 1 doesn't prevent zmq_term() from hanging. This option doesn't exist in 4.0 releases, so I can't try it out there. André On Tue, Dec 16, 2014 at 5:17 PM, Justin Karneges jus...@affinix.com wrote: Hi Andre, On Tue, Dec 16, 2014, at 07:14 AM, Andre Caron wrote: The issue I'm having is with this sequence: - P1 and P2 discover each other through D; - P1 connects to P2 and P2 waits for a connection from P1 (direction is determined by lexicographical ordering of identities, which both peers have prior to connecting); - Peers exchange heartbeats for a while; - I forcibly crash P2; - P1 eventually detects that P2 is unresponsive and explicitly disconnects; - after this happens, I restart P2; - P1 and P2 discover each other through D again; - P1 tries to connect to P2 and P2 expects a connection from P1; - both peers send heartbeats, but neither peer receives the other's messages and it appears the connection is never established. Also note that after this has happened, context termination hangs despite closing the (only) socket and setting the linger to 1 second. If I crash P1 instead of P2, the reconnection is successful. Also, if after the error sequence above I crash P1, peers reconnect successfully. This is a known issue, and I reported it earlier this year: http://lists.zeromq.org/pipermail/zeromq-dev/2014-February/025202.html I believe the problem is that once a connector queue learns the ID of a remote address, this binding sticks for life. The reason that you can restart P1 and things work is because connectors maintain queues even if there are no connections, but binders don't. Unfortunately I haven't had time yet to look at a fix. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev