Re: [zeromq-dev] HWM behaviour blocking
Peter, I am using mdclient2 (DEALER/ROUTER) and it is excellent, I don't have to manage IP, session,... However without ACK from the worker, the sender wouldn't know if message is received or lost during transit. I want to deploy pattern over the internet, therefore deliver confirmation is important. My implementation requires worker to ACK for each message it received. The sender waits to receive ACK before sending the next message. I want to improve the performance by sending a batch of messages, then check ACKs, if any ACK is missed, resend missing messages. Cheers, Viet Viet Hoang | Chief Executive | Quant Edge Corp mobile: +849 8822 3399 email: viet.ho...@quant-edge.com skype: viet.hoang.qe Floor 1A, Viet Hai Building, Block C2H Duy Tan Street, Cau Giay Dist Hanoi, Vietnam tel: +84 (4) 8587 6467 fax: +84 (4) 3795 9142 web: www.quant-edge.com - This message (including attachments, if any) is confidential, may be privileged and is intended for the above-named recipient(s) only. If you have received this message in error, please notify me by return email and delete this message from your system. Any unauthorized use or disclosure of this message is strictly prohibited. Quant Edge Corp assumes no responsibility for errors, inaccuracies or omissions in these materials, and does not warrant the accuracy or completeness of the information contained within these materials. Quant Edge Corp shall not be liable for any special, indirect, incidental, or consequential damages, including without limitation losses, lost revenues, or lost profits that may result from these materials. On May14, 2012, at 11:11 PM, Pieter Hintjens wrote: Viet, There is actually a variant of the MDP client that works asynchronously. You still want workers to be synchronous, but clients can stream multiple requests and get replies. See the mdpclient2 example. -Pieter On Mon, May 14, 2012 at 10:36 AM, Viet Hoang (Quant Edge) viet.ho...@quant-edge.com wrote: The Major Domo pattern may suits this well. Receiver to ACK for each message. If you don't receive, just resend. However you sacrifice the beauty of ZeroMQ - SPEED. We applied Major Domo into our demo platform and so far so good (better than our old C# raw socket implementation). What we will do is to have async ACK to improve performance. On May 11, 2012, at 6:44 AM, Michel Pelletier wrote: On Thu, May 10, 2012 at 1:53 PM, Pieter Hintjens p...@imatix.com wrote: On Thu, May 10, 2012 at 3:44 PM, Paul Colomiets p...@colomiets.name wrote: Can you be more specific, why setting HWM to 1 is a bad thing? Do you mean, that it smells bad to set HWM to 1 for reliability? Or do you think that setting it will have other consequences? (low performance?) it's bad because you're trying to force a synchronous model on an asynchronous system, and doing it at the wrong level. If you really want synchronization you MUST get some upstream data from the receiver. Just throttling the sender cannot work reliably. Agreed. Here's my take on what trips a lot of people up with 0mq: we are used to controlling how and when something is sent at the point that we call send(), or at least knowing in advance what will happen if we try, but in an async model you have to let that go. send() is going to return immediately (if you haven't hit a blocking case) and your message is now on its own, free as a bird, to live in various queues and buffers before it ends up at its destination. You have no control or visibility of its fate after you send it unless your receiver acknowledges it, or acknowledges it didn't receive it after a period of time (nack). The blocking case isn't really an exception, you sent when your application wasn't ready to receive, either because your buffers were full or your receivers weren't ready. Senders and receivers should synchronize this application level state with each other, possibly via some out-of-band channel, either by indicating they are ready, or connected, or that they are busy and can't do anymore, or by exchanging some kind of flow control information so that the sender doesn't fill the buffers because the receiver can't keep up. To use an analogy, all 0mq provides are the pipes. The pipes can't tell you that the tap is running and the sink is overflowing or the drain is clogged. If you want to have a reservoir at some point to regulate flow, an inline device can store a certain capacity of messages. If your pipe is delivering to a downstream reservoir which is near full capacity, someone at the downstream end needs to pickup the phone (yet another pipe of sorts) and tell the upstream to turn down the flow. If that doesn't happen, the reservoir is full, and the flow stops (blocked) or maybe spills over (discards) depending on the design of the *application*, not the pipe. In either case it's not the pipe's fault, it did its job, it
Re: [zeromq-dev] internet deployment
It doesn't happen very often, and it only happens to Windows XP (so far). The scenario is we have the trading terminal running on client's computers. During network lost, the terminal using Major Domo pattern trying to reconnect (connect, timeout, destroy) to a list of gateways. Terminals running on Windows 7 are OK, as soon as network is back, trading resumes. Instants on Windows XP have problem. After 10-20 times of trying to reconnect, the terminal crashes. We build transparent module to separate the network layer and business layer, therefore we could localize the problem is with the libzmq 3.1. However without console screen, we could not trace to the exact assertion. I will try to reproduce the issue by running a small tests, connect/destroy sockets, on Windows XP. I will let you guys know later if I could reproduce. Cheers, Viet Viet Hoang | Chief Executive | Quant Edge Corp mobile: +849 8822 3399 email: viet.ho...@quant-edge.com skype: viet.hoang.qe Floor 1A, Viet Hai Building, Block C2H Duy Tan Street, Cau Giay Dist Hanoi, Vietnam tel: +84 (4) 8587 6467 fax: +84 (4) 3795 9142 web: www.quant-edge.com - This message (including attachments, if any) is confidential, may be privileged and is intended for the above-named recipient(s) only. If you have received this message in error, please notify me by return email and delete this message from your system. Any unauthorized use or disclosure of this message is strictly prohibited. Quant Edge Corp assumes no responsibility for errors, inaccuracies or omissions in these materials, and does not warrant the accuracy or completeness of the information contained within these materials. Quant Edge Corp shall not be liable for any special, indirect, incidental, or consequential damages, including without limitation losses, lost revenues, or lost profits that may result from these materials. On May14, 2012, at 11:13 PM, Pieter Hintjens wrote: On Mon, May 14, 2012 at 10:42 AM, Viet Hoang (Quant Edge) viet.ho...@quant-edge.com wrote: When will assertions be removed? If I just remove the assertions myself, will that compromise anything in the lib? It is better to discard message rather than crashing the program. We have zeromq working good with our demo system, and we want to move it to our production, running on the internet. You do not want to remove the assertions. As Chuck says, if you are getting assertions, these mean the application has errors that need to be corrected (the assertions highlight the errors so if you remove the assertion, the error will still be there and your code may not crash immediately but it will crash later with worse results). If you can get reproducible assertions, make minimal test cases, and help us fix the underlying problems. Thanks Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] uuid again
A short question: how libzqm is linked to libuuid (dynamically or statically)? Because I'm trying to build and run libzmq on Android. On last iteration I receive the following error when try to load this library: cannot locate 'uuid_generate'... signature.asc Description: OpenPGP digital signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] HWM behaviour blocking
On Tue, May 15, 2012 at 2:20 AM, Viet Hoang (Quant Edge) viet.ho...@quant-edge.com wrote: My implementation requires worker to ACK for each message it received. The sender waits to receive ACK before sending the next message. I want to improve the performance by sending a batch of messages, then check ACKs, if any ACK is missed, resend missing messages. If you want to send a batch to a single worker, then a batch is just a larger message. I.e. you can do this fully at the application level. If you want to break a batch over multiple workers, then you would have to modify the MDP protocol. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] uuid again
Have you used the instructions here: http://www.zeromq.org/build:android? On Tue, May 15, 2012 at 3:23 AM, Boris Gulay bo...@boressoft.ru wrote: A short question: how libzqm is linked to libuuid (dynamically or statically)? Because I'm trying to build and run libzmq on Android. On last iteration I receive the following error when try to load this library: cannot locate 'uuid_generate'... ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] uuid again
15.05.2012 14:51, Pieter Hintjens написал: Have you used the instructions here: http://www.zeromq.org/build:android? Yes. They are a bit outdated. When I finish with my task I will submit updated instructions. And I will write how to build all necessary libraries (uuid, zmq, jzmq) in one file. I see no reason to use them in separate files because each .apk has it's own set of libraries. It means when you load jzmq from your java app all this libraries will be loaded anyway. On Tue, May 15, 2012 at 3:23 AM, Boris Gulay bo...@boressoft.ru wrote: A short question: how libzqm is linked to libuuid (dynamically or statically)? Because I'm trying to build and run libzmq on Android. On last iteration I receive the following error when try to load this library: cannot locate 'uuid_generate'... ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev signature.asc Description: OpenPGP digital signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Compilation warning on GCC 4.4
When compiling on GCC 4.4+ the following warning message is issued: ctx.hpp:56: note: the mangling of 'va_list' has changed in GCC 4.4 Is everything OK? signature.asc Description: OpenPGP digital signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Compilation warning on GCC 4.4
On Tue, May 15, 2012 at 10:15 AM, Boris Gulay bo...@boressoft.ru wrote: When compiling on GCC 4.4+ the following warning message is issued: ctx.hpp:56: note: the mangling of 'va_list' has changed in GCC 4.4 Is everything OK? I see some references on google but no idea what the impact is... -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] context as global/static variable
Just out of curiousity, nobody ever encountered such problem or nobody
tried to define zmq::context_t as global variable (in c++)?
On Wed, May 9, 2012 at 3:23 PM, Daniel Krikun krikun.dan...@gmail.comwrote:
Further, I suspected there maybe a problem due to the singleton not being
thread-safe. Thus, I modified the code to something like this:
boost::mutex mutex_;
zmq::context_t* p_ = 0;
zmq::context_t get_global_context()
{
boost::mutex::scoped_lock(mutex_);
if(p_ == 0)
p_ = new zmq::context_t(1);
return *p_;
}
Well, that worked fine, but apparently, context_t destructor is never
invoked (am I correct?). If, for example, I change p_ to some smart
pointer, like boost::shared_ptr, which will invoke ~context_t at program
shutdown, then the same problem appears.
I guess, there is some problem in calling ~context_t at program exit.
Ideas?
On Wed, May 9, 2012 at 12:39 PM, Daniel Krikun krikun.dan...@gmail.comwrote:
Hello,
I have some c++ code, that uses zeromq. In the code, I have a singleton
wrapping zmq::context_t, like this:
struct ctx_singleton
{
zmq::context_t instance()
{
static zmq::context_t _ctx;
return _ctx;
}
};
At application shutdown, I get an assertion failure:
Successful WSASTARTUP not yet perfromed (..\..\..\src\signaler.cpp:124)
I suspected there was a problem w/ zmq::context_t having global storage,
so I just take some zeromq sample and moved context definition from main
into global scope, and yes, I yields the same failure. I looked at the
stack trace, it seems that some socket/synch. mechanism is already disposed
when zmq_term is invoked and thus the failure.
I really would like to put zmq::context_t instance, just to make things
easier, and not to pass the instance to all threads. Is there a possibility
to do so?
I'm using zeromq-2.1.11, visual studio 2008 on Windows XP sp3, 32-bit.
Thanks,
--
Daniel Krikun
--
Daniel Krikun
--
Daniel Krikun
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
