Re: [zeromq-dev] Bye bye Pieter

[Charles Remes]

Goodbye, Pieter.


> On Oct 4, 2016, at 11:02, Oleksii Zamiatin  wrote:
> 
> RIP Pieter
> 
>> 4 окт. 2016 г., в 19:00, Osiris Pedroso > > написал(а):
>> 
>> RIP Pieter.
>> 
>> On Tue, Oct 4, 2016 at 10:13 AM YANG Fan > > wrote:
>> RIP Pieter
>> 
>> On Tue, Oct 4, 2016 at 10:35 PM Godefroid Chapelle > > wrote:
>> On 04/10/16 16:29, Benjamin Henrion wrote:
>> > Dear Pieter,
>> >
>> > I wanted to say a last goodbye at noon at the hospital, but you told
>> > me to come at 3pm.
>> >
>> > Now that you are gone, I can't stop crying.
>> >
>> > --
>> > Benjamin Henrion http://ffii.org/>>
>> 
>> Check http://stallmanism.wikidot.com/ , it 
>> might make you smile !
>> 
>> --
>> Godefroid
>> 
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org 
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
>> 
>> -- 
>> --
>> YANG Fan
>> PhD Candidate, The Chinese University of Hong Kong
>> (852) 59333775  |  http://www.husky-project.com 
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org 
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org 
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Mailing List

[Charles Remes]

All,

Please stop spamming the list. I answered the original poster privately and 
solved their problem. No need to clutter the list with more responses.



> On May 31, 2016, at 08:19, jean rouquet  wrote:
> 
> or maybe to this link http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
> 
> 
> From: outrun...@live.fr
> To: zeromq-dev@lists.zeromq.org
> Date: Tue, 31 May 2016 15:17:12 +0200
> Subject: Re: [zeromq-dev] Mailing List
> 
> Nooo
> 
> From: anthea4...@gmail.com
> Date: Tue, 31 May 2016 13:58:24 +0100
> To: zeromq-dev@lists.zeromq.org
> Subject: [zeromq-dev] Mailing List
> 
> Apologies but I think I'm on this mailing list by accident can I please be 
> removed?
> 
> Thanks,
> 
> -- 
> Anthea Middleton
> 
> Fractalise Web Design 
> Anthea's Thoughts on Nice Things (blog) 
> 
> ___ zeromq-dev mailing list 
> zeromq-dev@lists.zeromq.org 
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> ___ zeromq-dev mailing list 
> zeromq-dev@lists.zeromq.org 
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org 
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
> 
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] CZTop, a new CZMQ binding for Ruby

[Charles Remes]

This is great!!


> On Jan 18, 2016, at 13:09, paddor  wrote:
> 
> Hi
> 
> I've recently been working on CZTop [1], a new CZMQ binding for Ruby. It is 
> based on the generated low-level FFI binding found in CZMQ's repository, 
> which I've packaged into a gem [2].
> 
> It's supposed to be simple, comprehensive, feel like Ruby, provide a high 
> documentation quality, and make the use of security mechanisms (like CURVE) 
> dead easy.
> 
> I consider it still in an early development stage, but it covers all of 
> CZMQ's core classes and the test suite passes on MRI 2.3, 2.24, 2.1.8, 
> Rubinius, and JRuby 9000 (at least on my machine and Travis CI).
> 
> Since it depends on quite a few recent bug fixes and improvements in ZMQ and 
> CZMQ, I haven't released it yet. You'll have to compile both libraries from 
> master. However, if you're interested, I encourage you to give it a try and 
> tell me what you think. Any feedback is appreciated.
> 
> You can find more information on Github.
> 
> Regards,
> Patrik Wenger
> 
> [1] https://github.com/paddor/cztop
> [2] https://github.com/paddor/czmq-ffi-gen
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Zero-copy and INPROC transport

[Charles Remes]

Look at this man page:

http://api.zeromq.org/3-2:zmq-msg-init-data 


Zeromq does zero copy.

However, before you go and write a bunch of complex code, just use the normal 
facilities first and benchmark it. You may not need to do this extra work.


> On Jan 16, 2016, at 10:13, Wenzel Jakob  wrote:
> 
> Hello all,
> 
> I’m interested in using ZeroMQ to build a multithreaded application which 
> uses INPROC transport to propagate messages between threads.
> 
> My plan was to send messages using zmq_sendmsg (created using 
> zmq_msg_init_data with a custom deallocator), and to receive messages, I was 
> planning to use zmq_recvmsg.
> 
> I’m concerned if INPROC will perform unnecessary copies of messages, which 
> can be very large in my application (i.e. hundreds of megabytes). In 
> principle, it should just be possible to do it without copies by just moving 
> zmq_msg_t* from the inproc socket of one thread to that of the other — 
> however, I am not familiar enough with the implementation of ZeroMQ to know 
> if that is indeed what happens. Can anyone here advise?
> 
> Thank you in advance for your help!
> 
> Best regards,
> Wenzel
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Assertion failure in libzm msg.cpp:296 when reading from ZMQ_STREAM socket

[Charles Remes]

Yes, you should build libzmq master and run your application with it. If it 
solves your problem, all is well.


> On Nov 17, 2015, at 08:45, Auer, Jens <jens.a...@cgi.com> wrote:
> 
> Hi,
> 
> have there been any fixes relating to this in the current master? I could 
> check with the code from the repository.
> 
> Cheers,
>   Jens
> 
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: +49 6151 36860 154
> jens.a...@cgi.com <mailto:jens.a...@cgi.com>
> Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter 
> de.cgi.com/pflichtangaben <http://de.cgi.com/pflichtangaben>.
>  
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI 
> Group Inc. and its affiliates may be contained in this message. If you are 
> not a recipient indicated or intended in this message (or responsible for 
> delivery of this message to such person), or you think for any reason that 
> this message may have been addressed to you in error, you may not use or copy 
> or deliver this message to anyone else. In such case, you should destroy this 
> message and are asked to notify the sender by reply e-mail.
> Von: zeromq-dev-boun...@lists.zeromq.org 
> [zeromq-dev-boun...@lists.zeromq.org]" im Auftrag von "Charles Remes 
> [li...@chuckremes.com]
> Gesendet: Dienstag, 17. November 2015 15:43
> An: ZeroMQ development list
> Betreff: Re: [zeromq-dev] Assertion failure in libzm msg.cpp:296 when reading 
> from ZMQ_STREAM socket
> 
> Pieter, 
> 
> You were probably checking the source in libzmq master repository. Line 296 
> is blank in the msg.cpp source file.
> 
> However, in the zeromq-4-1 repository there is definitely an assertion on 
> that line. Since the OP said they were using 4.1.3 then that’s the correct 
> repository to check.
> 
> https://github.com/zeromq/zeromq4-1/blob/master/src/msg.cpp#L296 
> <https://github.com/zeromq/zeromq4-1/blob/master/src/msg.cpp#L296>
> 
> cr
> 
> 
>> On Nov 17, 2015, at 08:29, Pieter Hintjens <p...@imatix.com 
>> <mailto:p...@imatix.com>> wrote:
>> 
>> I don't see the assertion failure in msg.cpp:296...
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org <mailto:zeromq-dev@lists.zeromq.org>
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org <mailto:zeromq-dev@lists.zeromq.org>
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
> <http://lists.zeromq.org/mailman/listinfo/zeromq-dev>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Assertion failure in libzm msg.cpp:296 when reading from ZMQ_STREAM socket

[Charles Remes]

Pieter, 

You were probably checking the source in libzmq master repository. Line 296 is 
blank in the msg.cpp source file.

However, in the zeromq-4-1 repository there is definitely an assertion on that 
line. Since the OP said they were using 4.1.3 then that’s the correct 
repository to check.

https://github.com/zeromq/zeromq4-1/blob/master/src/msg.cpp#L296 


cr


> On Nov 17, 2015, at 08:29, Pieter Hintjens  wrote:
> 
> I don't see the assertion failure in msg.cpp:296...
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] how to use ZeroMQ socket in multithread

[Charles Remes]

Both of those techniques are the recommended ways to use zeromq sockets in a 
multi-threaded application. I’m sorry that you don’t like those techniques, but 
they work very well.

The zmq_poll function is *not* thread-safe, so be very careful with it.


 On May 18, 2015, at 23:46, 黄颖志 nohuan...@gmail.com wrote:
 
 Yeah, the zguide says if in multithread , i should create a inporc socket per 
 thread. but i don't want to create too much socket. Another method is to make 
 sure each thread executes a full memory barrier before accessing the socket, 
 in my opinion that is a hard work.
 
 2015-05-15 21:21 GMT+08:00 Charles Remes li...@chuckremes.com 
 mailto:li...@chuckremes.com:
 The FAQ available here http://zeromq.org/area:faq 
 http://zeromq.org/area:faq explains how to use zeromq in a multi-threaded 
 environment.
 
 The zguide also has good information on this topic.
 
 cr
 
 On May 15, 2015, at 08:18, 张凯 zhangkai@163.com 
 mailto:zhangkai@163.com wrote:
 
 It seems zeromq can not used in multithread.But you can adjust your code 
 with zmq_poll.
 
 At 2015-05-14 20:50:23, 黄颖志 nohuan...@gmail.com 
 mailto:nohuan...@gmail.com wrote:
 Hi every body,
   Lastly, my J2EE project use ZeroMQ as a message system , In that project,  
 I want to encapsulate ZeroMQ socket as a high level API, so others could use 
 that socket send messages if thay want . But the guide document says that 
 ZeroMQ sockets are not thread safe.  How can I solve that problem. I don't 
 want to create a new socket when sending some message through ZeroMQ every 
 time. 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org mailto:zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org mailto:zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] how to use ZeroMQ socket in multithread

[Charles Remes]

The FAQ available here http://zeromq.org/area:faq http://zeromq.org/area:faq 
explains how to use zeromq in a multi-threaded environment.

The zguide also has good information on this topic.

cr

 On May 15, 2015, at 08:18, 张凯 zhangkai@163.com wrote:
 
 It seems zeromq can not used in multithread.But you can adjust your code with 
 zmq_poll.
 
 At 2015-05-14 20:50:23, 黄颖志 nohuan...@gmail.com wrote:
 Hi every body,
   Lastly, my J2EE project use ZeroMQ as a message system , In that project,  
 I want to encapsulate ZeroMQ socket as a high level API, so others could use 
 that socket send messages if thay want . But the guide document says that 
 ZeroMQ sockets are not thread safe.  How can I solve that problem. I don't 
 want to create a new socket when sending some message through ZeroMQ every 
 time. 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Notes from a hackathon

[Charles Remes]

I agree with this. The final paragraph is key.

cr

 On Feb 6, 2015, at 9:41 AM, Thomas Rodgers rodg...@twrodgers.com wrote:
 
 Having thought about this for a couple of more days, I want to at least take 
 a stab at arguing against threadsafe sockets -
 
 libzmq's thread safety guarantees, to me anyway, are very clear, unsurprising 
 and non-controversial - I cannot share a socket with another thread without a 
 full fence.
 
 The kinds of systems I generally build have very strict requirements on 
 overall latency, to the point that most of my networking IO is done through 
 kernel-bypass libraries and NICs that support this, for raw TCP and UDP 
 multicast. The latency sensitive code that does IO is in it's own thread, 
 with exclusive access to the NICs which are accessed via kernel bypass. 
 Coordination with other threads in the same process is done via inproc pair 
 sockets. Pair sockets + very small messages (small enough that libzmq does 
 not need to perform allocation) provide a very nice interface to a lock free 
 queue with low overhead using a single atomic CAS operation. Atomic 
 operations are cheap, but they are not free (~30 clocks on x86). Adding a 
 mutex, even one that is never contended, to the socket will essentially 
 triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the 
 message on the pipe, one atomic CAS to release the mutex). I would like to 
 have the option to avoid this.
 
 If a wrapper wants thread safe sockets to enable certain use-cases that may 
 be more idiomatic for the language in question, it can provide the full 
 fence. AZMQ https://github.com/zeromq/azmq does exactly this by default, 
 but you have the option to opt out of it. It does this because Boost Asio by 
 default allows it's sockets to be used from multiple threads for async IO and 
 I need to guard more than just exclusive access to the ZeroMQ socket a the 
 fence in this case. Putting a mutex inside of the libzmq socket, essentially 
 doubles the overhead for no gain in useful functionality and runs completely 
 counter to one of C and C++'s overarching principles: don't pay for what you 
 don't use.
 
 If a class of apps really demands short lived exclusive access to a socket, 
 provide a pool abstraction. The pool is thread safe, obtain a socket, perform 
 I/O in a single thread, return the socket to the pool.
 
 On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier pelletier.mic...@gmail.com 
 mailto:pelletier.mic...@gmail.com wrote:
 
 
 On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens p...@imatix.com 
 mailto:p...@imatix.com wrote:
 The discussion about thread safety was quite short iirc, though that
 contributor did discuss other things... at length. I merged his
 thread safe socket change rapidly, then we reverted it after a few
 days, and he disappeared. It was rather brute force and I suspect did
 not work at all, it simply wrapped all accesses to the socket
 structure in mutexes. No discussion at the time of multipart data and
 atomic send/recv.
 
 My memory of the conversation at the time is pretty dim, I agree the changes 
 were ugly and untested and the contributor was difficult to reason with and 
 seemed to want to make the changes based on no real need at all.
  
 
 As for socket safety, I've no strong opinion. I see that many people
 expect that to work and hit errors when it doesn't. I see that nanomsg
 has threadsafe sockets and no multipart. I see that sharing sockets
 across threads would make some actor models simpler, which is nice.
 
 This is the classic problem with thread safe anything.  Threads are hard, and 
 there is a balance between the complexity of making a thread safe construct 
 and the skill required of a programmer to use unsafe construct in a safe 
 manner.  I still think if the concrete problem is very short lived threads 
 causing slow joiner problems, then the simple solution is pools (troupes of 
 actors?).
 
 -Michel
  
 
 
 
 
 
 
 On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
 pelletier.mic...@gmail.com mailto:pelletier.mic...@gmail.com wrote:
  I think Brian has some good points here, there are numerous unrelated issues
  being discussed in this thread.
 
  A few points that I have:
 
  Multi part messages have also bothered me.  However as a Python programmer i
  see Min's points about the expense of buffer creation.  To my knowledge
  zproto does not (yet) have Python generation support either, or maybe
  something like generated cffi or ctypes wrappers around the zproto generated
  C code.  That being said there are a variety of serialization libraries for
  Python.  With some ctypes and mmap magic they can also be done zero copy
  but it's not pretty:
 
  https://gist.github.com/michelp/7522179 
  https://gist.github.com/michelp/7522179
 
  Multi part envelops are also how multi-hop routing is done.  I don't see how
  the new ideas handle that.  I don't think we can just say multi hop routing
  is bad and get rid of it.
 
  Thread 

Re: [zeromq-dev] not binging to all LISTEN port

[Charles Remes]

Mohit,

Your application should have end-to-end latency and throughput measurements. I 
build this functionality into any project that uses zeromq so that I can track 
the latency between any two endpoints (and any intermediate points too). This 
data is printed to a log in a format that is easily parsed by an external tool. 
The tool reads the log and calculates all of the timing measurements along with 
statistics (mean, median, mode, standard deviation, etc). It outputs the 
results into a tabular format which can then be loaded into Excel for pretty 
graphs  charts.

During testing, you set a baseline for messages/sec, latency, or whatever else 
you think is important. The reporting tool can then highlight any parts that 
are outside of bounds from your baseline.

This information then let’s me decide if I have any bottlenecks.

Good luck. It takes some effort early on to build this mechanism but it is 
reusable on any future project.

cr

On Sep 23, 2014, at 12:35 PM, Mohit Anchlia mohitanch...@gmail.com wrote:

 I understand the basic part of performance testing and all other related 
 things that need to happen. I was really looking from troubleshooting 
 perspective. Say load increased more than anticipated or tested numbers, when 
 that happens are there any metrics available that can point to router/dealer 
 being a bottleneck. How can I really tell if I need to scale?
 On Tue, Sep 23, 2014 at 10:30 AM, Gregg Irwin gr...@pointillistic.com wrote:
 Hi Mohit,
 
 MA I was more of referring to debugging overload issues in prod.
 MA that we might not have been able to catch in performance
 MA environment. How can I tell that router/dealer is overloaded and
 MA need more router/dealers?
 
 While it's always good to test, if you provide a rough hardware
 outline, and the number and size of messages you need to support, you
 may get one of three answers borne of experience by users here.
 
 1) No problem. You are well within a single socket's capabilities.
 
 2) One socket will not support that load. You might need N sockets.
 
 3) It could be close. You should test in your environment.
 
 -- Gregg
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] not binging to all LISTEN port

[Charles Remes]

No, home grown to read our logs. The instrumentation code also relies on our 
application protocol to identify sources and sinks. Unfortunately there would 
be very little to gain by releasing the source.

It might be an interesting exercise for the list to develop a standard 
application protocol that has support for this kind of instrumenting built in.

cr

On Sep 23, 2014, at 4:50 PM, Matthew Hawn matth...@donaanacounty.org wrote:

 Charles,
 
 I am very curious what tool you use to analyze the traffic data.  Is it by 
 chance open-source?
 
 Matt
 From: zeromq-dev-boun...@lists.zeromq.org 
 [zeromq-dev-boun...@lists.zeromq.org] on behalf of Charles Remes 
 [li...@chuckremes.com]
 Sent: Tuesday, September 23, 2014 11:44 AM
 To: ZeroMQ development list
 Subject: Re: [zeromq-dev] not binging to all LISTEN port
 
 Mohit,
 
 Your application should have end-to-end latency and throughput measurements. 
 I build this functionality into any project that uses zeromq so that I can 
 track the latency between any two endpoints (and any intermediate points 
 too). This data is printed to a log in a format that is easily parsed by an 
 external tool. The tool reads the log and calculates all of the timing 
 measurements along with statistics (mean, median, mode, standard deviation, 
 etc). It outputs the results into a tabular format which can then be loaded 
 into Excel for pretty graphs  charts.
 
 During testing, you set a baseline for messages/sec, latency, or whatever 
 else you think is important. The reporting tool can then highlight any parts 
 that are outside of bounds from your baseline.
 
 This information then let’s me decide if I have any bottlenecks.
 
 Good luck. It takes some effort early on to build this mechanism but it is 
 reusable on any future project.
 
 cr
 
 On Sep 23, 2014, at 12:35 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
 
 I understand the basic part of performance testing and all other related 
 things that need to happen. I was really looking from troubleshooting 
 perspective. Say load increased more than anticipated or tested numbers, 
 when that happens are there any metrics available that can point to 
 router/dealer being a bottleneck. How can I really tell if I need to scale?
 On Tue, Sep 23, 2014 at 10:30 AM, Gregg Irwin gr...@pointillistic.com 
 wrote:
 Hi Mohit,
 
 MA I was more of referring to debugging overload issues in prod.
 MA that we might not have been able to catch in performance
 MA environment. How can I tell that router/dealer is overloaded and
 MA need more router/dealers?
 
 While it's always good to test, if you provide a rough hardware
 outline, and the number and size of messages you need to support, you
 may get one of three answers borne of experience by users here.
 
 1) No problem. You are well within a single socket's capabilities.
 
 2) One socket will not support that load. You might need N sockets.
 
 3) It could be close. You should test in your environment.
 
 -- Gregg
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Using zmq sockets from multiple threads

[Charles Remes]

Yes.

http://zeromq.org/area:faq

cr

On Aug 21, 2014, at 9:53 AM, Peter Durkan pdur...@lucerahq.com wrote:

 Hi,
 
 Is it safe to use sockets from multiple threads if you prevent concurrent 
 access using a mutex? 
 
 i.e.
 
 {
   std::lock_guardstd::mutex lck(send_mtx_);
   ... send a message on the  socket
 }
 
 I can also create a new socket each time a new thread calls the function and 
 use the thread_id as a key but was wondering if using a mutex like the above 
 would get around that. 
 
 Thanks,
 Peter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ROUTER and Credit-Based Flow Control

[Charles Remes]

If you are using credit-based flow control then you shouldn’t care about the 
HWM settings. Set them to infinite.

cr

On Aug 21, 2014, at 1:17 PM, Gabriel Sanchez gave2...@gmail.com wrote:

 Hi,
 
 I am trying to implement credit-based flow control from Chapter 7 of the 
 guide using ROUTER-DEALER. However, I am dropping messages on the ROUTER side 
 when reaching the SNDHWM. If I set the maximum amount of credits to a 
 fraction of the SNDHWM I don't drop any messages. I checked the code in 
 pipe.cpp and saw that the reader side of the pipe only notifies the writer 
 when it has read a multiple of the LWM.
 
 Does this mean that if the number of messages written by the ROUTER to a peer 
 pipe approaches the SNDHWM and in the meantime I get more credits before the 
 reader reaches the LWM I may drop messages? If so, are there any workarounds? 
 I was thinking of setting the maximum credit value to SNDHWM - LWM to ensure 
 that the number of messages written doesn't go above the SNDHWM.
 
 I am using libzmq 3.2.3 and I noticed that in newer versions there's a patch 
 for notifying the application when sending a message to a ROUTER fails.
 
 Thanks in advance
 
 Gabriel
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ROUTER and Credit-Based Flow Control

[Charles Remes]

Ha, I love those eureka moments! I wish I had more of them…

cr

On Aug 21, 2014, at 3:39 PM, Gabriel Sanchez gave2...@gmail.com wrote:

 Of course! I hadn't thought of that. Thanks!
 
 Gabriel
 
 
 On Thu, Aug 21, 2014 at 1:23 PM, Charles Remes li...@chuckremes.com wrote:
 If you are using credit-based flow control then you shouldn’t care about the 
 HWM settings. Set them to infinite.
 
 cr
 
 On Aug 21, 2014, at 1:17 PM, Gabriel Sanchez gave2...@gmail.com wrote:
 
  Hi,
 
  I am trying to implement credit-based flow control from Chapter 7 of the 
  guide using ROUTER-DEALER. However, I am dropping messages on the ROUTER 
  side when reaching the SNDHWM. If I set the maximum amount of credits to a 
  fraction of the SNDHWM I don't drop any messages. I checked the code in 
  pipe.cpp and saw that the reader side of the pipe only notifies the writer 
  when it has read a multiple of the LWM.
 
  Does this mean that if the number of messages written by the ROUTER to a 
  peer pipe approaches the SNDHWM and in the meantime I get more credits 
  before the reader reaches the LWM I may drop messages? If so, are there any 
  workarounds? I was thinking of setting the maximum credit value to SNDHWM - 
  LWM to ensure that the number of messages written doesn't go above the 
  SNDHWM.
 
  I am using libzmq 3.2.3 and I noticed that in newer versions there's a 
  patch for notifying the application when sending a message to a ROUTER 
  fails.
 
  Thanks in advance
 
  Gabriel
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Multiple Pub, single sub multipart messages

[Charles Remes]

Not possible. There must be a bug somewhere. Send a link to working code to 
this list so someone can try and duplicate it.

cr

On Aug 8, 2014, at 10:03 AM, christian.ko...@rohde-schwarz.com wrote:

 Hello, 
 
 I have an application with two publisher and one subscriber. Both publishers 
 send a multipart message [SubscriptionTopic,Data] at the same time: 
 
 Pub1 [A,B] 
 Pub2 [C,D] 
 
 And the Sub receives: 
 
 [A,C,B,D] 
 
 So the message parts are messing up. 
 
 My general question: 
 Is this possible at all? My feeling is that this is not supported because the 
 incoming messages can not be separated by publisher. On the other hand in the 
 documentation is stated that a multipart message is sent all or nothing 
 manner. 
 
 Could anybody please enlighten me? 
 
 Thanks, 
 Christian___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] resetting REQ/REP state machines

[Charles Remes]

Does anyone think it’s worthwhile to provide a zmq_setsockopt() to allow for 
resetting the statemachine for a REQ or a REP socket?

One of the biggest stumbling blocks for new users is the lock-step 
send/reply/send/reply sequence enforced by a pair of REQ/REP sockets. If a 
message gets lost, we now force the user to close the socket(s) and create a 
new one. This essentially resets the statemachine for the socket.

Wouldn’t it be better to allow the user to force reset the state?

e.g. (pseudo-code)

req = zmq_socket(ZMQ_REQ);
rc = zmq_send(req, buf);

// some timeout expires
rc = zmq_setsockopt(req, ZMQ_RESETSTATE);

rc = zmq_send(req, buf);

I haven’t looked at the source in a while but I can’t imagine this would be all 
that difficult. The difficulty may come in dealing with late message arrivals 
but that can probably just be pushed onto the user to detect and handle.

Thoughts?

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] resetting REQ/REP state machines

[Charles Remes]

I didn’t even know that option (and its associated ZMQ_REQ_CORRELATE) were in 
the library! The library is moving so fast these days that I can’t even keep up 
(which is a good thing).

Those options are a superior solution to my proposal. Glad they are in the 
library.

cr

On Jul 29, 2014, at 9:15 AM, Pieter Hintjens p...@imatix.com wrote:

 Check ZMQ_REQ_RELAXED, which may do what you want.
 
 On Tue, Jul 29, 2014 at 3:35 PM, Thomas Rodgers rodg...@twrodgers.com wrote:
 The difficulty may come in dealing with late message arrivals but that can
 probably just be pushed onto the user to detect and handle.
 
 
 Would that be the lesser of the 'evils' in question here?  To me, throwing
 away the socket on a hard failure (e.g. timeout of the remote endpoint)
 makes a clean break with the endpoint and keeps the protocol cruft to detect
 other failures down.
 
 
 On Tue, Jul 29, 2014 at 7:49 AM, Charles Remes li...@chuckremes.com wrote:
 
 Does anyone think it’s worthwhile to provide a zmq_setsockopt() to allow
 for resetting the statemachine for a REQ or a REP socket?
 
 One of the biggest stumbling blocks for new users is the lock-step
 send/reply/send/reply sequence enforced by a pair of REQ/REP sockets. If a
 message gets lost, we now force the user to close the socket(s) and create a
 new one. This essentially resets the statemachine for the socket.
 
 Wouldn’t it be better to allow the user to force reset the state?
 
 e.g. (pseudo-code)
 
 req = zmq_socket(ZMQ_REQ);
 rc = zmq_send(req, buf);
 
 // some timeout expires
 rc = zmq_setsockopt(req, ZMQ_RESETSTATE);
 
 rc = zmq_send(req, buf);
 
 I haven’t looked at the source in a while but I can’t imagine this would
 be all that difficult. The difficulty may come in dealing with late message
 arrivals but that can probably just be pushed onto the user to detect and
 handle.
 
 Thoughts?
 
 cr
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] queue overhead

[Charles Remes]

Justin,

The way you would structure this is by setting up a zmq_proxy() that would bind 
to the “well known” port to receive tasks. On the back end, the proxy would 
also bind to a local tcp port (or via ipc, perf should be the same). The 
workers on the box would all connect to this backend port. The proxy would just 
receive incoming messages and then dole them out to the workers. If you use 
push/pull sockets, it will round robin them. If you need something more 
sophisticated, you can build your own proxy component and make the routing and 
load balancing logic as smart (or as dumb) as you like.

The perf hit will be measurable but it’s likely on the order of a millisecond 
or two. It’s impossible to be more definitive than that since it is very 
dependent upon your hardware. If this is on a dedicated box with 10GB 
interfaces, the router will be very fast. If this is on a Micro AWS instance in 
the cloud, then the router will probably takes a few 10s of milliseconds to do 
its work. If you use TCP endpoints, you can start to spread your workers over 
multiple boxes and scale horizontally. It’s pretty neat.

None of this is difficult, so benchmark it on your own hardware to determine 
the exact overhead. I think the flexibility that a configuration like this 
offers is well worth a tiny overhead.

If you need redundancy then I suggest reading through the advanced patterns in 
the zguide. All of this is covered there in detail usually with working code.

cr

On Jul 27, 2014, at 1:13 PM, Justin Karneges jus...@affinix.com wrote:

 I have a stable (in the addressing sense) worker that I want to take 
 advantage of multiple cores. So, I run N instances of this worker, where 
 N is the number of cores on the host machine, and each worker binds to 
 its own socket. Components that wish to make use of this worker service 
 connect to all N worker instances.
 
 Unfortunately this is a little awkward. The connecting components must 
 be configured with the N socket specs. And it's hard to automate this, 
 since even if the connecting components could generate socket specs 
 programmatically, this still requires knowing the number of cores of the 
 remote machine.
 
 What I'd like to do is put an adapter component in front of the N worker 
 instances (on the same machine as the worker instances) that binds to a 
 single socket. It would route to the N workers, and this is easily done 
 since the adapter lives on the same machine and knows the number of 
 cores. Connecting components could then simply connect to this adapter, 
 and not need to care about the number of remote cores.
 
 The question I have is what kind of overhead this introduces. An MxN set 
 of connections between M remote components and the N workers seems like 
 it would be far more efficient than M-1-N, which looks like a 
 bottleneck. But maybe in practice, if the routing is very simple, then 
 it becomes negligible?
 
 Justin
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Router-Dealer Example not working in C on 4.0.4 ZMQ release

[Charles Remes]

This tutorial that focuses on dealer and router sockets might also help explain 
things a bit more too. When in doubt, the zguide text takes precedence but this 
tutorial should still be correct.

http://zeromq.org/tutorials:dealer-and-router

cr

On Jul 25, 2014, at 3:17 PM, Badhrinath Manoharan badh...@gmail.com wrote:

 Hi Peter,
 
 No issues at all. I totally get your thoughts here. Thanks a lot again for 
 the detailed response. 
 
 I just got misled that we could use Router - Dealer as similar to Rep - Req 
 socket. Moreover I had one client and one server so thought it might work.
 
 Anyways thanks a lot for correcting my understanding. Will surely give it a 
 try and update the thread.
 
 Thanks again
 Badhri
 
 Sent from my iPhone
 
 On Jul 25, 2014, at 1:33 AM, Pieter Hintjens p...@imatix.com wrote:
 
 Sorry if that email seemed unpolite; it was late and I'd been driving
 for 10 hours with three small kids who refused to sleep.
 
 So, ROUTER sockets always prefix the incoming message with an extra
 frame that holds a routing id (aka identity), that identifies which
 pipe the message came in from. You stick that same frame back before
 outgoing replies, and the ROUTER socket then knows which pipe to send
 it to.
 
 These IDs start with a binary zero. If you try to printf them, the
 result is an empty string. In my example code, if you tried it, the
 print shows this ID frame quite clearly.
 
 This is explained in detail in Chapter 3 of the Guide, and briefly in
 the zmq_socket man page for ZMQ_ROUTER.
 
 It is not trivial, and you should really read the Guide and work
 through the examples before trying to use ROUTER.
 
 -Pieter
 
 On Fri, Jul 25, 2014 at 12:53 AM, Pieter Hintjens p...@imatix.com wrote:
 You're not reading the man pages, nor documentation, nor example
 output properly.
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Speed of Subscriptions

[Charles Remes]

Should take less than a second assuming a LAN or other low-latency network.

cr

On Jul 1, 2014, at 3:32 PM, Johnny Lee j...@peaceatwork.org wrote:

  Hello, 
 
 I have a question about how fast an average workstation can subscribe to 
 its publishers using the Pub/Sub method.
 
 We have developed a client application (Publisher) that sends simple, 
 pre-formatted message to receivers (Subscribers).  
 
 When the receiver program launches and at specified intervals, it subscribes 
 to its list of publishers. 
 
 As our program may have a receiver  listening to 100's of senders, 
 the receiver needs to subscribe to those hundreds of publishers first.
 
 As those publisher stations may be coming on and off at through out the day, 
 we were going to run this process every 15 minutes or so. 
 
 But how long would that take?
 
 I realize that average workstation may have different meaning 
 but say a standard Windows workstation that can be found in a typical office 
 environment
 (Example: Intel Core i3-2130 CPU @ 3.40 GHz; 6 GB RAM; 64-bit Operating 
 System)
 
 trying to subscribe to 300 publishers. 
 
 5 seconds, 10 seconds, 40 seconds, 1 minute?
 
 
 Please let me know if you need any clarification. 
 
 thank you, 
 Johnny
 
 
  
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Questions about Shutdown

[Charles Remes]

Your original sequence is correct. No one has been recommending swapping the 
order of 4  5.

You could try and terminate a context in a thread, but it will *block* until 
all sockets close and clean up. Unless you want this behavior, you should clean 
up sockets first.

cr

On Jun 21, 2014, at 11:19 AM, Paulmichael Blasucci pblasu...@gmail.com wrote:

 So, after 3 years of putting ZMQ in production, I'm worried I may be 
 operating under mistaken assumptions. In the guide (and in my code), there is 
 a fairly consistent pattern:
 
 1. Create/configure context
 2. Create/configure sockets
 3. Actual code which uses sockets
 4. Clean up sockets 
 5. Clean up context
 
 This seems quite logical, and has served me quite well. However, after some 
 recent discussions, I'm concerned the last two steps above can, on occasion, 
 be reversed. In other words, an application may (intentionally) dispose of 
 the context while one, or more, sockets are still live. Is this true? If 
 so, can anyone elaborate on when this is necessary?
 
 Thanks!
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] PUB/SUB unreliabiliity

[Charles Remes]

Let’s back up for a second.

Take a look at the man page for zmq_setsockopt and read the section on 
ZMQ_SNDHWM. It clearly states that zero means “no limit.” Second, it also 
states that when the socket reaches its exceptional state then it will either 
block or drop messages depending on socket type.

Next, look at the man page for zmq_socket and check the ZMQ_PUB section. The 
socket will reach its mute state (its exceptional state) when it reaches it 
high water mark. When it’s mute, it will drop messages.

So, taking the two together then a socket with a ZMQ_SNDHWM of 0 should never 
drop messages because it will never reach its mute state.

The one exception to this is when there are no SUB sockets connected to the PUB 
socket. When there are no connections, all messages are dropped (because no one 
is listening and there are no queues created).

However, I highly recommend *against* setting HWM to 0 for a PUB socket. Here’s 
why:

1. It gives you a false sense of security that all messages will be delivered.
If the publishing process dies, any messages in queue go with it so they’ll 
never get delivered.

2. Your subscribers might be too slow.
If your subscribers can’t keep up with the message flow and the publisher 
starts queueing, it *will* run out of memory. You’ll either exhaust the amount 
of memory allowed by your process, or your OS will start paging  swapping and 
you’ll wish the process had just died.

cr


On Jun 13, 2014, at 5:34 PM, Gerry Steele gerry.ste...@gmail.com wrote:

 Hi Brian
 
 I noticed your comment on another thread about this and I think you got it a 
 bit wrong:
 
  The high water mark is a hard limit on the maximum number of outstanding 
  messages ØMQ shall queue in memory for any single peer that the 
  specifiedsocket is communicating with. A value of zero means no limit.
 
 and from your link:
 
  Since v3.x, ØMQ forces default limits on its internal buffers (the 
  so-called high-water mark or HWM), so publisher crashes are rarer unless 
  you deliberately set the HWM to infinite.
 
 Nothing I read indicates anything other than the fact that no messages post 
 connections being made should be dropped.
 
 Thanks
 G
 
 
 
 On 13 June 2014 23:17, Brian Knox bk...@digitalocean.com wrote:
 From what i've read, PUB SUB should be reliable when the _HWM are set to 
 zero (don't drop). By reliable I mean no messages should fail to be delivered 
 to an already connected consumer.
 
 
 Your understanding of pub-sub behavior and how  it interacts with the HWM is 
 incorrect.  Please see: http://zguide.zeromq.org/php:chapter5
 
 Brian
 
 
 
 
 On Fri, Jun 13, 2014 at 2:33 PM, Gerry Steele gerry.ste...@gmail.com wrote:
 I've read everything I can find including the Printed book, but I am at a 
 loss as to the definitive definition as to how PUB/SUB should behave in zmq.
 
 A production system I'm using is experiencing message loss between several 
 nodes using PUB/SUB.
 
 From what i've read, PUB SUB should be reliable when the _HWM are set to zero 
 (don't drop). By reliable I mean no messages should fail to be delivered to 
 an already connected consumer.
 
 I implemented some utilities to reproduce the message loss in my system :
 
 zmq_sub: https://gist.github.com/easytiger/992b3a29eb5c8545d289
 zmq_pub: https://gist.github.com/easytiger/e382502badab49856357
 
 
 zmq_pub takes a number of events to send and the logging frequency and 
 zmq_sub only takes the logging frequency. zmq prints out the number of msgs 
 received vs the packet contents containing the integer packet count from the 
 publisher.
 
 It can be seen when sending events in a tight loop that messages simply go 
 missing mid way through (loss is not at beginning or end ruling out slow 
 connectors etc)
 
 In a small loop it usually works ok:
 
 $ ./zmq_pub 2000 1000
 sent MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #1000 with rc=58
 sent MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #2000 with rc=58
 
 $ ./zmq_sub 1
 
 RECV:1|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #1
 RECV:2|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #2
 RECV:3|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #3
 RECV:4|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #4
 [...]
 RECV:2000|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #2000
 
 You can see every message was sent as the counts align.
 
 However increase the message counts and messages start going missing
 
 $ ./zmq_pub 20 10 
   
 sent MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #10 with rc=60
 sent MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #20 with rc=60
 
 ./zmq_sub 1
 RECV:1|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #11000
 RECV:2|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #21000
 RECV:3|MESSAGE PAYLOAD OF A NONTRIVAL SIZE KIND OF AND SUCH #31610
 RECV:4|MESSAGE PAYLOAD OF A NONTRIVAL SIZE 

Re: [zeromq-dev] Explicit control over socket buffers?

[Charles Remes]

Maybe it’s time to learn a lesson from nanomsg and eliminate SND/RCV queues 
altogether. AFAIK, nanomsg pushes all queueing down a layer into the OS and 
let’s the OS handle this task via whatever mechanisms exist (various buffers, 
network stack, etc).

cr

On Jun 10, 2014, at 1:09 PM, Pieter Hintjens p...@imatix.com wrote:

 It's not just imprecision. The measurement can be several orders of
 magnitude wrong / out of date, particularly when there's a high
 message rate. We know full and empty and vague mutterings in
 between. Simply measuring the pipe state would interfere with its
 operation. (I believe, happy to be proven wrong).
 
 I know only two ways to safely manage the number of messages
 explicitly. One is from the client side, using credit-based flow
 control. Second is to offload messages into a single-threaded memory
 queue (a list) that can be managed explicitly by its owning task.
 
 -Pieter
 
 On Tue, Jun 10, 2014 at 4:27 PM, Andrew Hume and...@research.att.com wrote:
 there is a minor way in which i disagree (and have always done so) with
 this:
 
 there are often potentially many useful ways “internals” can be exposed,
 even if approximately. for example, there is little reason why an
 approximate
 answer to “how many messages are buffered” can’t be generated. and for
 nearly
 all cases, knowing the answer +/- a handful of messages would be good
 enough.
 and most of the time, allowing a small degree of imprecision allows an
 inexpensive
 solution.
 
 of course, YMMV.
 
 andrew
 
 On Jun 9, 2014, at 11:48 PM, Pieter Hintjens p...@imatix.com wrote:
 
 It's the same reason TCP doesn't let you discard packets that are on
 the wire. If you expose the internals of the plumbing to the
 application then you cannot scale. It makes things considerably more
 complex, and potentially slower by orders of magnitude. Even trying to
 expose a concept like how many messages are queued? breaks things
 (libzmq's pipes are filled and emptied asynchronously by two threads
 and there is simply no way to measure the current state except by
 freezing everything).
 
 If you want explicit management of message queues, you build this on top.
 
 -Pieter
 
 On Mon, Jun 9, 2014 at 9:24 AM, Dmitry Antipov anti...@dev.rtsoft.ru
 wrote:
 
 Hello,
 
 why 0mq doesn't offer the facilities to control socket buffers (except
 setting send/receive watermarks and default socket policy to discard/block)?
 In particular, someone can ask about functions like 1) zmq_peek, to tell
 how many messages are buffered for input/output, and 2) zmq_discard, to
 discard all queued messages.
 
 It's quite possible that such a questions just illustrates some
 misunderstanding
 about what 0mq sockets are and how they should be used; but can someone
 explain
 the (obviously by-design) solution to _not_ provide these features?
 
 Thanks in advance,
 Dmitry
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 
 ---
 Andrew Hume
 949-707-1964 (VO and best)
 732-420-0907 (NJ)
 and...@research.att.com
 
 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zeromq performance

[Charles Remes]

Yes, the default is one I/O thread. This assumes you are using IPC and/or TCP 
for transport. If you use INPROC then I think it skips the creation of any I/O 
thread.

So, to duplicate your test using the ZMQ_AFFINITY socket option, you would want 
to specify that all endpoints are using the same CPU.

cr

On Jun 7, 2014, at 12:00 PM, Paul Phillips p...@marketgridsystems.com wrote:

 I can look at this.  However, at this point, I am calling zmq_ctx_new() and 
 not setting any options, so I assume that for each of my processes I only 
 have one zmq I/O thread for all its sockets and connections.  Is that correct?
 
 Regards, Paul Phillips
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 On 8 Jun 2014, at 2:55 am, Charles Remes li...@chuckremes.com wrote:
 
 This is a very interesting result. It’s probably worthwhile to add this to 
 the FAQ.
 
 It would be interesting to test this scenario with the ZMQ_AFFINITY option 
 for zmq_setsockopt. If you have a chance, could you let us know if setting 
 the I/O thread affinity results in a similar performance difference?
 
 cr
 
 On Jun 7, 2014, at 7:44 AM, Paul Phillips p...@marketgridsystems.com wrote:
 
 I have tracked down the source of this “problem”.  It turns out that the 
 running multiple local-remote sets is not the key, it is actually related 
 to what processor core things are running on.  If the local_lat and the 
 remote_lat run on the same core then the comms is very fast (whether ipc or 
 tcp).  If they are on separate cores, the comms is slower.  When I ran 
 multiple sets, it just turned out that for some reason, the second local 
 and remote set would be on the same core.  I can replicate the whole thing 
 using taskset to force local_lat and remote_lat either to different or the 
 same cores.
 
 I have pasted my results from two runs, the first using separate cores and 
 the second using the same core and the results are very different - 25 
 micros latency vs 7 micros latency (I have printed a couple of extra things 
 at the end and renamed the executables to xlocal_lat and xremote_lat but 
 they are the same code from the perf directory in the distribution of 
 4.0.4).
 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xlocal_lat 
 ipc://testport2 100 10 
 [1] 3566
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 1 ./xremote_lat 
 ipc://testport2 100 10 
 [2] 3578
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ message size: 100 [B]
 roundtrip count: 10
 average latency: 25.421 [us]
 elapssed: 5084170
 throughput: 19668
 
 [1]-  Donetaskset -c 0 ./xlocal_lat ipc://testport2 100 
 10
 [2]+  Donetaskset -c 1 ./xremote_lat ipc://testport2 
 100 10
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xlocal_lat 
 ipc://testport2 100 10 
 [1] 3581
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xremote_lat 
 ipc://testport2 100 10 
 [2] 3584
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ message size: 100 [B]
 roundtrip count: 10
 average latency: 7.016 [us]
 elapssed: 1403130
 throughput: 71269
 
 [1]-  Donetaskset -c 0 ./xlocal_lat ipc://testport2 100 
 10
 [2]+  Donetaskset -c 0 ./xremote_lat ipc://testport2 
 100 10
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 
 Regards, Paul Phillips
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 On 7 Jun 2014, at 6:17 pm, Pieter Hintjens p...@imatix.com wrote:
 
 How many messages are you sending? There will be a start-up cost that
 can be disproportionate if you send only a few messages.
 
 On Sat, Jun 7, 2014 at 6:54 AM, Paul Phillips
 p...@marketgridsystems.com wrote:
 Hi.  I have an interesting scenario when testing zeromq 4.0.4 on CentOS 
 6.5.
 When I run local_lat and remote_lat using ipc, I get a latency of around 
 30
 micros.  However, if I run them once in the background with a large round
 trips setting (so they keep running for a long time) and then run a second
 set, the second set always returns a latency of around 7 micros.  
 Basically,
 once I have one lot of stuff running in the background, subsequent stuff
 always seems to run much faster.  Is there any known explanation for this
 behaviour?
 
 Regards, Paul Phillips
 
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zeromq performance

[Charles Remes]

I don’t know the answer to your question. Hopefully someone on this list can 
answer it.


On Jun 9, 2014, at 8:24 AM, Paul Phillips p...@marketgridsystems.com wrote:

 I can set my processes and threads to use the same CPU (and I can then 
 duplicate my test).
 
 One thing that I am interested to know more about though is the ZMQ_AFFINITY 
 option.  It seems to me that once zmq creates its own thread (or threads), I 
 can’t control which CPU core the zmq thread(s) runs on.  I believe that 
 ZMQ_AFFINITY will let me lock a socket to a particular zmq thread but the zmq 
 thread itself could wander around over the CPU cores.  Even though I assign 
 the parent application process (or thread) to a particular core, I believe 
 that the zmq thread that is spawned would not be bound to that core.  Is that 
 correct?
 
 Regards, Paul Phillips
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 On 9 Jun 2014, at 11:18 pm, Charles Remes li...@chuckremes.com wrote:
 
 Yes, the default is one I/O thread. This assumes you are using IPC and/or 
 TCP for transport. If you use INPROC then I think it skips the creation of 
 any I/O thread.
 
 So, to duplicate your test using the ZMQ_AFFINITY socket option, you would 
 want to specify that all endpoints are using the same CPU.
 
 cr
 
 On Jun 7, 2014, at 12:00 PM, Paul Phillips p...@marketgridsystems.com 
 wrote:
 
 I can look at this.  However, at this point, I am calling zmq_ctx_new() and 
 not setting any options, so I assume that for each of my processes I only 
 have one zmq I/O thread for all its sockets and connections.  Is that 
 correct?
 
 Regards, Paul Phillips
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 On 8 Jun 2014, at 2:55 am, Charles Remes li...@chuckremes.com wrote:
 
 This is a very interesting result. It’s probably worthwhile to add this to 
 the FAQ.
 
 It would be interesting to test this scenario with the ZMQ_AFFINITY option 
 for zmq_setsockopt. If you have a chance, could you let us know if setting 
 the I/O thread affinity results in a similar performance difference?
 
 cr
 
 On Jun 7, 2014, at 7:44 AM, Paul Phillips p...@marketgridsystems.com 
 wrote:
 
 I have tracked down the source of this “problem”.  It turns out that the 
 running multiple local-remote sets is not the key, it is actually related 
 to what processor core things are running on.  If the local_lat and the 
 remote_lat run on the same core then the comms is very fast (whether ipc 
 or tcp).  If they are on separate cores, the comms is slower.  When I ran 
 multiple sets, it just turned out that for some reason, the second local 
 and remote set would be on the same core.  I can replicate the whole 
 thing using taskset to force local_lat and remote_lat either to different 
 or the same cores.
 
 I have pasted my results from two runs, the first using separate cores 
 and the second using the same core and the results are very different - 
 25 micros latency vs 7 micros latency (I have printed a couple of extra 
 things at the end and renamed the executables to xlocal_lat and 
 xremote_lat but they are the same code from the perf directory in the 
 distribution of 4.0.4).
 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xlocal_lat 
 ipc://testport2 100 10 
 [1] 3566
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 1 ./xremote_lat 
 ipc://testport2 100 10 
 [2] 3578
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ message size: 100 [B]
 roundtrip count: 10
 average latency: 25.421 [us]
 elapssed: 5084170
 throughput: 19668
 
 [1]-  Donetaskset -c 0 ./xlocal_lat ipc://testport2 
 100 10
 [2]+  Donetaskset -c 1 ./xremote_lat ipc://testport2 
 100 10
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xlocal_lat 
 ipc://testport2 100 10 
 [1] 3581
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xremote_lat 
 ipc://testport2 100 10 
 [2] 3584
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ message size: 100 [B]
 roundtrip count: 10
 average latency: 7.016 [us]
 elapssed: 1403130
 throughput: 71269
 
 [1]-  Donetaskset -c 0 ./xlocal_lat ipc://testport2 
 100 10
 [2]+  Donetaskset -c 0 ./xremote_lat ipc://testport2 
 100 10
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 
 Regards, Paul Phillips
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 On 7 Jun 2014, at 6:17 pm, Pieter Hintjens p...@imatix.com wrote:
 
 How many messages are you sending? There will be a start-up cost that
 can be disproportionate if you send only a few messages.
 
 On Sat, Jun 7, 2014 at 6:54 AM, Paul Phillips
 p...@marketgridsystems.com wrote:
 Hi.  I have an interesting scenario when testing zeromq 4.0.4 on CentOS 
 6.5.
 When I run local_lat and remote_lat using ipc, I get a latency

Re: [zeromq-dev] zeromq performance

[Charles Remes]

This is a very interesting result. It’s probably worthwhile to add this to the 
FAQ.

It would be interesting to test this scenario with the ZMQ_AFFINITY option for 
zmq_setsockopt. If you have a chance, could you let us know if setting the I/O 
thread affinity results in a similar performance difference?

cr

On Jun 7, 2014, at 7:44 AM, Paul Phillips p...@marketgridsystems.com wrote:

 I have tracked down the source of this “problem”.  It turns out that the 
 running multiple local-remote sets is not the key, it is actually related to 
 what processor core things are running on.  If the local_lat and the 
 remote_lat run on the same core then the comms is very fast (whether ipc or 
 tcp).  If they are on separate cores, the comms is slower.  When I ran 
 multiple sets, it just turned out that for some reason, the second local and 
 remote set would be on the same core.  I can replicate the whole thing using 
 taskset to force local_lat and remote_lat either to different or the same 
 cores.
 
 I have pasted my results from two runs, the first using separate cores and 
 the second using the same core and the results are very different - 25 micros 
 latency vs 7 micros latency (I have printed a couple of extra things at the 
 end and renamed the executables to xlocal_lat and xremote_lat but they are 
 the same code from the perf directory in the distribution of 4.0.4).
 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xlocal_lat 
 ipc://testport2 100 10 
 [1] 3566
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 1 ./xremote_lat 
 ipc://testport2 100 10 
 [2] 3578
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ message size: 100 [B]
 roundtrip count: 10
 average latency: 25.421 [us]
 elapssed: 5084170
 throughput: 19668
 
 [1]-  Donetaskset -c 0 ./xlocal_lat ipc://testport2 100 
 10
 [2]+  Donetaskset -c 1 ./xremote_lat ipc://testport2 100 
 10
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xlocal_lat 
 ipc://testport2 100 10 
 [1] 3581
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ taskset -c 0 ./xremote_lat 
 ipc://testport2 100 10 
 [2] 3584
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ message size: 100 [B]
 roundtrip count: 10
 average latency: 7.016 [us]
 elapssed: 1403130
 throughput: 71269
 
 [1]-  Donetaskset -c 0 ./xlocal_lat ipc://testport2 100 
 10
 [2]+  Donetaskset -c 0 ./xremote_lat ipc://testport2 100 
 10
 (MarketGrid)[Paul@CentOS65-Dev tmp]$ 
 
 Regards, Paul Phillips
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 On 7 Jun 2014, at 6:17 pm, Pieter Hintjens p...@imatix.com wrote:
 
 How many messages are you sending? There will be a start-up cost that
 can be disproportionate if you send only a few messages.
 
 On Sat, Jun 7, 2014 at 6:54 AM, Paul Phillips
 p...@marketgridsystems.com wrote:
 Hi.  I have an interesting scenario when testing zeromq 4.0.4 on CentOS 6.5.
 When I run local_lat and remote_lat using ipc, I get a latency of around 30
 micros.  However, if I run them once in the background with a large round
 trips setting (so they keep running for a long time) and then run a second
 set, the second set always returns a latency of around 7 micros.  Basically,
 once I have one lot of stuff running in the background, subsequent stuff
 always seems to run much faster.  Is there any known explanation for this
 behaviour?
 
 Regards, Paul Phillips
 
 Director, MarketGrid Systems Pty Ltd
 t: +61 419 048 874
 e: p...@marketgridsystems.com
 
 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Welcome to the zeromq-dev mailing list

[Charles Remes]

Take a look at the Clone pattern in the zguide.

http://zguide.zeromq.org/page:all#Reliable-Pub-Sub-Clone-Pattern

This might be what you need.

cr

On May 29, 2014, at 11:20 AM, Jeremy Richemont jrichem...@gmail.com wrote:

 
 Hi. I am struggling to work out how to use zmq to implement the architecture 
 I need. I have a classic publish/subscribe situation except that once client 
 x has subscribed to a topic I need the topic data to be sent to it to be 
 cached if the client dies and resent on reconnect. The data order is 
 important and I can't miss messages should the client be offline for a while.
 
 The PUB/SUB pattern doesn't seem to know about individual clients and will 
 just stop sending to client x if it dies. Plus I can't find out this has 
 happened and cache the messages, or know when it reconnects.
 
 To try to get around this I used the REQ/REP pattern so the clients can 
 announce themselves and have some persistence but this is not ideal for a 
 couple of reasons:
 
 1) The clients must constantly ask got any data for me? which offends my 
 sensibilities
 
 2) What happens if there's no data to send to client x but there is to client 
 y? Without zmq I'd have had a thread per client and simply block the one with 
 no data but I can't block client x without also blocking client y in a single 
 thread. 
 
 Am I trying to shove a round peg in a square hole, here? Is there some way I 
 can get feedback from PUB saying 'failed to send to client x'? so I can cache 
 the messages instead? Or is there some other pattern I should be using?
 
 Otherwise it's back to low level tcp for me...
 
 Many thanks;
 
 Jeremy
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] RFC: finer control of socket type / behaviour

[Charles Remes]

I don’t think it’s a bad idea, however I would suggest using the existing 
zmq_setsockopt() and zmq_getsockopt() calls to get/set socket options.

cr

On May 23, 2014, at 9:40 AM, Goswin von Brederlow goswin-...@web.de wrote:

 Hi,
 
 in ZeroMQ there are different kinds of sckets with different characteristics:
 
 From the zmq_socket manpage: e.g. REQ socket:
 
Summary of ZMQ_REQ characteristics
Compatible peer socketsZMQ_REP, ZMQ_ROUTER
Direction  Bidirectional
Send/receive pattern   Send, Receive, Send, Receive, #
Outgoing routing strategy  Round-robin
Incoming routing strategy  Last peer
Action in mute state   Block
 
 While ZeroMQ has many different socket types it does not cover every
 sensible (or not so sensible) combination of characteristics. Most
 anyoing the Action in mutet state is dictated by the socket type
 while one might want different behaviour.
 
 I wonder how hard it would be to make the Action in mute state a
 socket option that could be modified seperately from the socket type.
 The socket type would still set different defaults for backward
 compatibility. But one could create a ZMQ_ROUTER socket and then
 change the Action in mute state to Block instead of Drop.
 
 
 Probably more dificult would be to make all (the 3 defining ones)
 characteristic of a socket selectable from the start. Namely:
 
 * Incoming routing strategy
  - Fair-qeueued
  - Last peer (restricts send/receive pattern)
  - None (unidirectional)
  - Meta (unidirectional except for meta messages [subscription for XPUB])
  - Single (only one connection allowed)
 * Outgoing routing strategy
  - Round-robin
  - Fan out
  - Routed (peer address is first frame)
  - Last peer (restricts send/receive pattern)
  - None (unidirectional)
  - Meta (unidirectional except for meta messages [subscription for XSUB])
  - Single (only one connection allowed)
 * Action in mute state
  - EAGAIN
  - Block
  - Drop
 
 A fourth characteristic is required to reflect the full range of socket types:
 
 * Identity handling
  - Router (append on receive, strip on send)
  - Dealer (no change to messages in or out)
  - Last peer (strip all on receive, append all on send)
 
 The other charactestics can be derived from the incoming and outgoing
 strategies and not all combinations make sense.
 
 Note: strategy Single could be dropped. What is the point of ZMQ_PAIR
 anyway?
 
 Note 2: strategy Routed implies Router handling, maybe that could be
 merged with Last peer strategy. In both cases the peer is picked by
 its identity.
 
 
 
 So what do you think? Does it make sense to add a new function:
 
   void *zmq_custom_socket(void *context, in_strategy in, out_strategy out, 
 id_handling handling);
 
 and new socket option ZMQ_MUTE_STATE?
 
 MfG
   Goswin
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Question about zeromq throughput benchmarks.

[Charles Remes]

Share your newest code with the list. Maybe someone will run it and identify 
further improvements.


On Mar 30, 2014, at 9:28 AM, Alexander V Vershilov 
alexander.vershi...@gmail.com wrote:

 I have tried 2M messages the result is roughly the same (as in second 
 letter), i.e. TCP outperform ZMQ Push-Pull in maximum 2 times, and on 
 extreme size results are the same. Its possible that I've made some mistake 
 in TCP so the programs are not equivalent.
 
 However I didn't try to optimize any zmq or kernel options.
 
 So now difference looks OK to use ZeroMQ in my program. But if there is a 
 room for additional optimizations or better patterns it will be very good.
 
 --
 Alexander
 
 On Mar 29, 2014 2:17 AM, Pieter Hintjens p...@imatix.com wrote:
 10,000 messages is often too little to get significant results. Try
 sending 2M small messages...
 
 zmq_send copies the data, which for large messages costs more.
 
 On Fri, Mar 28, 2014 at 10:20 AM, Alexander V Vershilov
 alexander.vershi...@gmail.com wrote:
 
 
  On 27 March 2014 16:48, Charles Remes li...@chuckremes.com wrote:
 
  On Mar 27, 2014, at 4:41 AM, Alexander V Vershilov
  alexander.vershi...@gmail.com wrote:
 
  Hi,
 
  I'm trying to write a small benchmark program using zeromq-4.0.4 that will
  be used as a
  prototype for higher level library. Test program creates a pair of
  asynchronous sockets and
  send a bunch of messages with no acknowledgement and a the end reads a
  reply.
 
 
  Surprisingly, this test program does not compare favourably with an
  equivalent direct
  implementation over TCP. I have the following timings for sending 10,000
  messages of the
  given size on the localhost:
 
 
  This is surprising.
 
 
  Is there something I am misunderstanding here? I have gone through several
  `iterations of
  my benchmarks, but perhaps you can point out any problem with it?
 
 
 
  Have you tried comparing your results to the built-in local_lat/remote_lat
  and local_thr/remote_thr benchmark programs? You could easily modify the
  throughput benchmark to use PUSH/PULL sockets and see if the results differ
  wildly from the pub/sub results.
 
 
 
  I've tried the tests from local_thr/remote_thr they are also use PUSH/PULL
  socket pair. The only difference is that
  that benchmarks are preparing message 'zmq_msg_init_size' and then
  'zmq_sendmsg', while in my
  benchmark I'm just using zemq_send, this better reflect TCP case. New
  variant  gives me extra ~2-10kMb/s depending
  on the message size. However TCP is still faster, except very big messages
  where it's on par with ZeroMQ.
 
  Thanks.
 
  --
  Alexander Vershilov
mail-to: alexander.vershi...@tweag.io
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] 2ms roundtrip with python bindings

[Charles Remes]

On Mar 27, 2014, at 12:36 PM, Mark markree...@gmail.com wrote:

 Mark markreed99 at gmail.com writes:
 
 
 I modified the hello world client to time a single send and recv across 
 my network and get 1.6ms:
 
 Note that I removed the sleep from hwserver.py. 
 
 import time
 import zmq
 
 context = zmq.Context()
 socket = context.socket(zmq.REP)
 socket.bind(tcp://*:port)
 
 while True:
message = socket.recv()
socket.send(b”World)

No, the problem is that you are comparing apples and oranges. 

The perf latency test does 10,000 roundtrips and averages the number. Your 
first few messages are probably going to be slower due to TCP congestion algos 
and whatnot.

Your second test sends just ONE message. You never give the TCP stream any time 
to warm up.

You could rewrite that second test in pure C and get the same result.

Nothing wrong with the python bindings.

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] zeromq, rewritten in Rust

[Charles Remes]

There was a little twitter chat over the weekend regarding an attempt at 
writing a ground-up zeromq library using the new systems language Rust. If you 
haven’t heard of Rust, it is a new language under development by the good folks 
at Mozilla. It’s original designer has said that he has learned quite a few 
things from implementing dozens of languages over the years that he felt he 
could solve some new problems and create a cleaner language. Rust is his 
attempt at such a feat.

It supposedly solves the problem by borrowing the best from many popular 
languages.

* OOP of C++ without the large, unwieldy syntax

* performance of C while providing good namespacing, OOP, safe memory 
(i.e. no dangling pointers)

* the functional expressiveness of Haskell but not at the expense of 
imperative forms

* the massive concurrency of Erlang but with a better syntax and a more 
flexible memory model (borrowed pointers, immutable defaults, etc)

I recently did a small test project to learn the syntax. The language is still 
evolving, so it’s a bit of a moving target. It’s at release 0.9 with a 1.0 
slated for later this year, but they’ve already slipped on delivering a 1.0 for 
at least a year so I assume it will slip again.

Anyway, I’d like to volunteer to try and spike a simple example to get things 
started. However, I’d like to start a thread here to discuss “lessons learned” 
from the existing codebase. We already have a great write-up from Martin 
Sustrik (primary author of earlier versions of zeromq) here:  

http://250bpm.com/blog:4

http://250bpm.com/blog:8

I’m hoping that others who have read through the source have additional 
insights that they’d like to share. For instance, I have seen comments that 
zeromq might have more consistent performance it it was wrapped around a 
Disruptor (google for that pattern if it’s new to you). People also seem to 
really dislike the concept of the context (nanomsg has already eliminated this… 
it still exists but is hidden by the library).

Any other insights?

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] default MAX_SOCKETS on Windows set to 63?

[Charles Remes]

Aha! That solved it. It finally built a libzmq.dll with the correct FD_SETSIZE. 

Now my question is why the top-level Makefile doesn’t work. At minimum it 
should know that it’s on a mingw32 platform and then invoke that child Makefile 
to build things correctly.

Makefiles are an arcane art beyond my ken… 

On Feb 17, 2014, at 9:12 PM, KIU Shueng Chuan nixch...@gmail.com wrote:

 Oops. Should have been mingw32-make -f Makefile.mingw32
 
 On 18 Feb 2014 11:04, Charles Remes li...@chuckremes.com wrote:
 I am using an msys shell to do the compilation directly on Windows (no 
 cross-compiling). I just tried to run “mingw32-make Makefile.mingw32” from 
 the proper directory but it just said “Nothing to be done for 
 Makefile.mingw32” and exited.
 
 I wish I could figure out the secret incantation to get my FD_SETSIZE 
 override to work. Better yet, it would be useful if the generated Makefile 
 automatically set it to that value when run in a mingw32 build host.
 
 
 On Feb 17, 2014, at 5:42 PM, KIU Shueng Chuan nixch...@gmail.com wrote:
 
 Looking at the ctx.cpp source, setting of MAX_SOCKETS goes through 
 clipped_maxsocket() which clips it to max_fds()-1, where max_fds() is 
 equal to FD_SETSIZE as returned by zmq::select_t::max_fds(). So your issue 
 does appear to be related to FD_SETSIZE.
 
 
 
 I tried it out again on zeromq4-x 571c668fa2e0b101f607b4c4a34c30da2f174499.
 
 Cross-compile on Linux using MinGW32:
  CPPFLAGS=-DFD_SETSIZE=1024 ./configure --host=i686-pc-mingw32
 
  grep CPPFLAGS Makefile
 CPPFLAGS = -pedantic -Werror -Wall -Wno-long-long -D_REENTRANT 
 -D_THREAD_SAFE -DFD_SETSIZE=1024 -DZMQ_FORCE_SELECT
 
 Adding the following snippet in one of the source files shows that it does 
 get picked up.
 #if FD_SETSIZE!=1024
 #error FD_SETSIZE wrong
 #endif
 
 
 
 If you are compiling directly on Windows and do not want to install MSYS 
 (using MSYS may yield other quirks!), an alternative to using configure is 
 to go to the subdir builds/mingw32.
 There is a simple Makefile.mingw32 there. 
 Just run mingw32-make Makefile.mingw32
 Haven't tried this for some time and it is supposed to be automatically 
 generated nowadays.
 
 
 
 On Tue, Feb 18, 2014 at 12:27 AM, Charles Remes li...@chuckremes.com wrote:
 I’ve tried a few different ways of providing FD_SETSIZE but nothing has 
 worked for me so far.
 
 I have tried:
 
 * CFLAGS=-DFD_SETSIZE=1024 ./configure
 
 * CPPFLAGS=-DFD_SETSIZE=1024 ./configure
 
 Neither one is picked up properly by the system. What else should I try?
 
 On Feb 14, 2014, at 5:56 PM, KIU Shueng Chuan nixch...@gmail.com wrote:
 
 https://github.com/zeromq/czmq/issues/104
 
 If compiling using the configure script you need to provide FD_SETSIZE.
 
 On 15 Feb 2014 01:28, Charles Remes li...@chuckremes.com wrote:
 I received a bug report from a user who compiled zeromq 4.0.3 on Windows 
 using 32-bit mingw. The code that blew up was:
 
 context = LibZMQ.zmq_ctx_new
 rc = LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS, 1023)
 
 At this point, “rc” was returning -1 and the error message is “Invalid 
 Argument”. After playing around with this a bit I discovered that the 
 default MAX_SOCKETS was 63 on this platform.
 
 context = LibZMQ.zmq_ctx_new
 print LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS) # = prints “63”
 
 So, trying to set a value above 63 always fails.
 
 For further confirmation, I ran “make check” and noted that 
 “test_ctx_options” also fails (though it doesn’t tell me why).
 
 Does anyone know why MAX_SOCKETS is set to 63 when compiling on Windows 
 using mingw? When I look at the source, MAX_SOCKETS_DFLT is set to 1023!
 
 cr
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] default MAX_SOCKETS on Windows set to 63?

[Charles Remes]

I’ve tried a few different ways of providing FD_SETSIZE but nothing has worked 
for me so far.

I have tried:

* CFLAGS=-DFD_SETSIZE=1024 ./configure

* CPPFLAGS=-DFD_SETSIZE=1024 ./configure

Neither one is picked up properly by the system. What else should I try?

On Feb 14, 2014, at 5:56 PM, KIU Shueng Chuan nixch...@gmail.com wrote:

 https://github.com/zeromq/czmq/issues/104
 
 If compiling using the configure script you need to provide FD_SETSIZE.
 
 On 15 Feb 2014 01:28, Charles Remes li...@chuckremes.com wrote:
 I received a bug report from a user who compiled zeromq 4.0.3 on Windows 
 using 32-bit mingw. The code that blew up was:
 
 context = LibZMQ.zmq_ctx_new
 rc = LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS, 1023)
 
 At this point, “rc” was returning -1 and the error message is “Invalid 
 Argument”. After playing around with this a bit I discovered that the default 
 MAX_SOCKETS was 63 on this platform.
 
 context = LibZMQ.zmq_ctx_new
 print LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS) # = prints “63”
 
 So, trying to set a value above 63 always fails.
 
 For further confirmation, I ran “make check” and noted that 
 “test_ctx_options” also fails (though it doesn’t tell me why).
 
 Does anyone know why MAX_SOCKETS is set to 63 when compiling on Windows using 
 mingw? When I look at the source, MAX_SOCKETS_DFLT is set to 1023!
 
 cr
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] default MAX_SOCKETS on Windows set to 63?

[Charles Remes]

I am using an msys shell to do the compilation directly on Windows (no 
cross-compiling). I just tried to run “mingw32-make Makefile.mingw32” from the 
proper directory but it just said “Nothing to be done for Makefile.mingw32” and 
exited.

I wish I could figure out the secret incantation to get my FD_SETSIZE override 
to work. Better yet, it would be useful if the generated Makefile automatically 
set it to that value when run in a mingw32 build host.


On Feb 17, 2014, at 5:42 PM, KIU Shueng Chuan nixch...@gmail.com wrote:

 Looking at the ctx.cpp source, setting of MAX_SOCKETS goes through 
 clipped_maxsocket() which clips it to max_fds()-1, where max_fds() is equal 
 to FD_SETSIZE as returned by zmq::select_t::max_fds(). So your issue does 
 appear to be related to FD_SETSIZE.
 
 
 
 I tried it out again on zeromq4-x 571c668fa2e0b101f607b4c4a34c30da2f174499.
 
 Cross-compile on Linux using MinGW32:
  CPPFLAGS=-DFD_SETSIZE=1024 ./configure --host=i686-pc-mingw32
 
  grep CPPFLAGS Makefile
 CPPFLAGS = -pedantic -Werror -Wall -Wno-long-long -D_REENTRANT -D_THREAD_SAFE 
 -DFD_SETSIZE=1024 -DZMQ_FORCE_SELECT
 
 Adding the following snippet in one of the source files shows that it does 
 get picked up.
 #if FD_SETSIZE!=1024
 #error FD_SETSIZE wrong
 #endif
 
 
 
 If you are compiling directly on Windows and do not want to install MSYS 
 (using MSYS may yield other quirks!), an alternative to using configure is to 
 go to the subdir builds/mingw32.
 There is a simple Makefile.mingw32 there. 
 Just run mingw32-make Makefile.mingw32
 Haven't tried this for some time and it is supposed to be automatically 
 generated nowadays.
 
 
 
 On Tue, Feb 18, 2014 at 12:27 AM, Charles Remes li...@chuckremes.com wrote:
 I’ve tried a few different ways of providing FD_SETSIZE but nothing has 
 worked for me so far.
 
 I have tried:
 
 * CFLAGS=-DFD_SETSIZE=1024 ./configure
 
 * CPPFLAGS=-DFD_SETSIZE=1024 ./configure
 
 Neither one is picked up properly by the system. What else should I try?
 
 On Feb 14, 2014, at 5:56 PM, KIU Shueng Chuan nixch...@gmail.com wrote:
 
 https://github.com/zeromq/czmq/issues/104
 
 If compiling using the configure script you need to provide FD_SETSIZE.
 
 On 15 Feb 2014 01:28, Charles Remes li...@chuckremes.com wrote:
 I received a bug report from a user who compiled zeromq 4.0.3 on Windows 
 using 32-bit mingw. The code that blew up was:
 
 context = LibZMQ.zmq_ctx_new
 rc = LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS, 1023)
 
 At this point, “rc” was returning -1 and the error message is “Invalid 
 Argument”. After playing around with this a bit I discovered that the 
 default MAX_SOCKETS was 63 on this platform.
 
 context = LibZMQ.zmq_ctx_new
 print LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS) # = prints “63”
 
 So, trying to set a value above 63 always fails.
 
 For further confirmation, I ran “make check” and noted that 
 “test_ctx_options” also fails (though it doesn’t tell me why).
 
 Does anyone know why MAX_SOCKETS is set to 63 when compiling on Windows 
 using mingw? When I look at the source, MAX_SOCKETS_DFLT is set to 1023!
 
 cr
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] default MAX_SOCKETS on Windows set to 63?

[Charles Remes]

I received a bug report from a user who compiled zeromq 4.0.3 on Windows using 
32-bit mingw. The code that blew up was:

context = LibZMQ.zmq_ctx_new
rc = LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS, 1023)

At this point, “rc” was returning -1 and the error message is “Invalid 
Argument”. After playing around with this a bit I discovered that the default 
MAX_SOCKETS was 63 on this platform.

context = LibZMQ.zmq_ctx_new
print LibZMQ.zmq_ctx_set(context, ZMQ::MAX_SOCKETS) # = prints “63”

So, trying to set a value above 63 always fails. 

For further confirmation, I ran “make check” and noted that “test_ctx_options” 
also fails (though it doesn’t tell me why).

Does anyone know why MAX_SOCKETS is set to 63 when compiling on Windows using 
mingw? When I look at the source, MAX_SOCKETS_DFLT is set to 1023!

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] LGPLv3 patent clause

[Charles Remes]

It took me a long time, but I have come around to share Pieter’s view.

cr

On Jan 31, 2014, at 5:19 AM, Pieter Hintjens p...@imatix.com wrote:

 This is off-topic for ZeroMQ, but if you want my detailed
 deconstruction of the patent system, and why I've spent years of my
 life fighting it, see here: http://hintjens.com/blog:31.
 
 My views are on the public record. For two years I was president of
 the FFII, the leading European NGO defining software patent policy in
 Europe. We contributed to the relevant clauses in the GPLv3. I chose
 that for ZeroMQ with the very specific intention of excluding any
 contributor that used patents in their software business strategy.
 
 In short, the patent system does not help small inventors except in
 freak cases. The _entire_ patent system is, in my view, a legal
 parasite designed by professional sociopaths to feed off real economic
 activity, leaving nothing but ruin behind. Every claim for patents is
 based on the lure of easy money, wrong assumptions, bogus economic
 theory, mistaken theories of innovation, naivety about legal process,
 or outright lies.
 
 On Fri, Jan 31, 2014 at 9:29 AM, Laurent Alebarde l.aleba...@free.fr wrote:
 IMHO, everything can be hell or haven, including LGPL. It depends on one's
 motivations and intentions. For patents, lawyers are the tools of the
 owners, so what happens depends on the owners (except the patent officers,
 but they infringe only the owners). I agree also many patents are abusive
 and shaming.
 
 That's why I am also against patents on software. But it is not that simple.
 Software is just a tool. Many technologies need software to come to life but
 are not intrinsically software. Then patents may be IMHO legitimate. When
 you spend thousands of hours to develop an innovative technology, you can
 expect earning some money with it, at least to feed your children, isn't it
 ?
 
 Open source and associated consulting and expertise is a great economical
 model, but not the only one. Moreover, it fits some people, and not others.
 Other people are creative and technology centric. For them, the patent may
 be the less bad idea, even if many times it is finaly a real bad idea
 because filing a patent requires that you have funds to defend it. But it
 may also be a good idea, especially if your invention needs to be opened, or
 once again, the less bad idea. Many corporates are unfair, isn't it ? The
 guy who invented car wipers had to fight all his life to obtain the first
 dollar from his invention.
 
 When it comes to technical innovation, when you want to keep your freedom in
 a small business, when you want to spend your time in creation more than in
 service, what do you propose ?
 
 Of course, you cannot answer: consulting is great, make your technology
 free, raise a community, do expertise and be happy. That would be just what
 fits to some people, but not others. Besides, if the technology is simple to
 use, nobody will need your expertise.
 
 One may also hire a few people to do the consulting, and stay on the
 creation, but then you have to spend most of your time in management, what
 is creativity counter  productive. You can also hire a CE, but then you may
 have other kind of troubles.
 
 You may also just stay an employee in a corporate with a cool RD.
 
 There is no legitimate reason to prevent anyone to get freedom and to be,
 behave and work as he is, as far as the freedom and dignity of everyone else
 is respected and preserved.
 
 My 2 cts.
 
 Laurent Alebarde
 
 
 Le 30/01/2014 14:00, Pieter Hintjens a écrit :
 
 On Thu, Jan 30, 2014 at 4:29 AM, Sojan James sojan.ja...@gmail.com wrote:
 
 I think zeromq is a great library to use for my project.  At my company, all
 open source usage needs to be approved for use by a legal team.  Usage of
 GLPv3 and LGPLv3 is flatly refused. One of the reasons is the clause about
 the patents.  I don't fully understand the implications of the patent clause
 of LGPLv3 w.r.t zeromq.  Does the static linking exception affect my
 obligations in any way?
 
 I'm basically looking for any information that I can provide to the legal
 team to discuss further if an exception can be granted for this library.
 
 The patent clause in the LGPLv3 applies to anyone using ZeroMQ, no
 matter how they link it. This is deliberate. The goal is to reduce
 patent lawsuits around ZeroMQ. You can read the full implications
 elsewhere; it's standard part of GPLv3. Any firm that flatly refuses
 (L)GPLv3 for these reasons is welcome to go buy commercial software,
 or use other products.
 
 Speaking personally, (software) patents are an infectious abomination
 that allow pseudo-lawyers to parasite the work of productive people.
 Firms do make the choice: either they embrace that abomination or they
 reject it. If they embrace it, they are dangerous partners and
 unreliable contributors. The LGPLv3 patent clause makes a good filter
 to ensure such firms do not join our community. /End of personal
 

Re: [zeromq-dev] Newbee Question

[Charles Remes]

As a matter of fact, we have. On your machine you should run the 
local_lat/remote_lat and local_thr/remote_thr benchmark programs. You can 
specify the transport on the command line and measure how fast it is on your 
box.


On Jan 18, 2014, at 3:48 AM, Goswin von Brederlow goswin-...@web.de wrote:

 On Fri, Jan 17, 2014 at 03:18:00PM -0600, Charles Remes wrote:
 Right now the IPC transport is build on top of UNIX sockets and does
 not use shared memory. If someone were willing to write and contribute
 the code to use a shared-memory approach, we would love to have it.
 The ?roadmap? for zeromq is whatever the community wants it to be. If
 someone wants to contribute a shared memory implementation for IPC
 then we?ll merge it. No one is paid to work on zeromq full time.
 
 So, aside from a few OS-level optimizations that might be in place,
 IPC is about the same performance as TCP transport.
 
 It would be difficult to say how zeromq performs in comparison to a
 shared memory implementation. Someone would need to write a benchmark
 to compare the two. I am not aware of anyone who has done that.
 
 On Jan 17, 2014, at 2:25 PM, Srinivas Kotamarti 
 srinivas.kotama...@genband.com wrote:
 
 Hi,
 I had read the guide, from what I understand the inproc:// transport is for 
 inter thread communication. What I am looking for is for communicating 
 between processes on the same machine. Since shared memory is faster for 
 communication between the processes on the same machine, I was wondering if 
 ZeroMQ has a transport for shared memory. 
 
 My search on the dev-list couple of  discussions regarding the same and a 
 possibility of 'memory mapped' file transport was  mentioned in the emails. 
 So does ZeroMQ already have a transport for shared memory or in the plans 
 for future?
 
 How does ZeroMQ ipc transport perform compare to a shared memory based 
 communication implementation?
 
 Thanks
 
 --Srinivas
 
 -Original Message-
 From: zeromq-dev-boun...@lists.zeromq.org 
 [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Goswin von 
 Brederlow
 Sent: Friday, January 17, 2014 5:37 AM
 To: zeromq-dev@lists.zeromq.org
 Subject: Re: [zeromq-dev] Newbee Question
 
 On Fri, Jan 17, 2014 at 02:44:20AM +, Srinivas Kotamarti wrote:
 Hi,
 I am new to ZeroMQ. I have a bunch of questions !!
 
 I was wondering if the ZeroMQ has an ipc for shared memory, if not does 
 anyone know what is the best IPC to use for communication on localhost?
 
 Is the ipc transport a unix domain socket?
 
 How does ZeroMQ ipc transport perform compare to a shared memory based 
 communication implementation?
 
 Answers are greatly appreciated !
 
 Thanks.
 --Srinivas
 
 Read the guide and look for inproc://.
 
 MfG
 Goswin
 
 Implementation wise wouldn't that be just like inproc://, a simple
 ypipe, except that the ypipe needs to be allocated in the shared
 memory and messages and payload must be allocated there too or copied?
 
 So how much faster is inproc over tcp? Has anyone benchmarked that?
 
 MfG
   Goswin
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Newbee Question

[Charles Remes]

Right now the IPC transport is build on top of UNIX sockets and does not use 
shared memory. If someone were willing to write and contribute the code to use 
a shared-memory approach, we would love to have it. The “roadmap” for zeromq is 
whatever the community wants it to be. If someone wants to contribute a shared 
memory implementation for IPC then we’ll merge it. No one is paid to work on 
zeromq full time.

So, aside from a few OS-level optimizations that might be in place, IPC is 
about the same performance as TCP transport.

It would be difficult to say how zeromq performs in comparison to a shared 
memory implementation. Someone would need to write a benchmark to compare the 
two. I am not aware of anyone who has done that.

On Jan 17, 2014, at 2:25 PM, Srinivas Kotamarti 
srinivas.kotama...@genband.com wrote:

 Hi,
 I had read the guide, from what I understand the inproc:// transport is for 
 inter thread communication. What I am looking for is for communicating 
 between processes on the same machine. Since shared memory is faster for 
 communication between the processes on the same machine, I was wondering if 
 ZeroMQ has a transport for shared memory. 
 
 My search on the dev-list couple of  discussions regarding the same and a 
 possibility of 'memory mapped' file transport was  mentioned in the emails. 
 So does ZeroMQ already have a transport for shared memory or in the plans for 
 future?
 
 How does ZeroMQ ipc transport perform compare to a shared memory based 
 communication implementation?
 
 Thanks
 
 --Srinivas
 
 -Original Message-
 From: zeromq-dev-boun...@lists.zeromq.org 
 [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Goswin von Brederlow
 Sent: Friday, January 17, 2014 5:37 AM
 To: zeromq-dev@lists.zeromq.org
 Subject: Re: [zeromq-dev] Newbee Question
 
 On Fri, Jan 17, 2014 at 02:44:20AM +, Srinivas Kotamarti wrote:
 Hi,
 I am new to ZeroMQ. I have a bunch of questions !!
 
 I was wondering if the ZeroMQ has an ipc for shared memory, if not does 
 anyone know what is the best IPC to use for communication on localhost?
 
 Is the ipc transport a unix domain socket?
 
 How does ZeroMQ ipc transport perform compare to a shared memory based 
 communication implementation?
 
 Answers are greatly appreciated !
 
 Thanks.
 --Srinivas
 
 Read the guide and look for inproc://.
 
 MfG
   Goswin
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Timing issues

[Charles Remes]

You should resend your answer. No one on the list has seen it.

On Jan 17, 2014, at 3:34 PM, Apostolis Xekoukoulotakis xekou...@gmail.com 
wrote:

 Gmane reports the correct list. I googled mailman and delete and it is a 
 common technique to delete the content but keep the rest of the mail. 
 Maybe though it is a bug of mailman, who knows. 
 I just want to express my disagreement in case it happened.
 
 17 Ιαν 2014, 23:12, ο/η Pieter Hintjens p...@imatix.com έγραψε:
 
 On Fri, Jan 17, 2014 at 4:55 PM, Apostolis Xekoukoulotakis
 xekou...@gmail.com wrote:
 I noticed that my last email got deleted from the archives. Please give 
 credit to the ones that deserve it and have peace.
 
 We don't ever delete emails from the archives... you sure you sent it?
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Timing issues

[Charles Remes]

Nagle’s algo is already disabled in the codebase (you can confirm that with a 
quick grep). I think what Bruno is referring to is that zeromq batches small 
messages into larger ones before sending. This improves throughput at the cost 
of latency as expected.

Check out the “performance” section of the FAQ for an explanation:  
http://zeromq.org/area:faq


On Jan 16, 2014, at 7:04 AM, Lindley French lindl...@gmail.com wrote:

 Ah, that would explain it, yes. It would be great to have a way of disabling 
 Nagle's algorithm (TCP_NODELAY sockopt).
 
 
 On Thu, Jan 16, 2014 at 4:24 AM, Bruno D. Rodrigues 
 bruno.rodrig...@litux.org wrote:
 Without looking at the code I assume ØMQ is not trying to send each 
 individual message as a TCP PDU but instead, as the name implies, queues 
 messages so it can batch them together and get the performance.
 
 This then means the wire will be filled up when some internal buffer fills, 
 or after a timeout, which looks like 100ms.
 
 On the other hand I can’t see any setsockopt to configure this possible 
 timeout value.
 
 Any feedback from someone else before I have time to  look at the code?
 
 On Jan 15, 2014, at 16:20, Lindley French lindl...@gmail.com wrote:
 
  I have a test case in which I'm communicating between two threads using zmq 
  sockets. The fact that the sockets are in the same process is an artifact 
  of the test, not the real use-case, so I have a TCP connection between them.
 
  What I'm observing is that a lot of the time, it takes ~100 milliseconds 
  between delivery of a message to the sending socket and arrival of that 
  message on the receiving socket. Other times (less frequently) it is a 
  matter of microseconds. I imagine this must be due to some kernel or thread 
  scheduling weirdness, but I can't rule out that it might be due to 
  something in 0MQ.
 
  If I follow the TCP socket write with one or more UDP writes using 
  Boost.Asio, the 100 millisecond delay invariably occurs for the ZMQ TCP 
  message but the UDP messages arrive almost instantly (before the TCP 
  message).
 
  My design requires that the TCP message arrive before *most* of the UDP 
  messages. It's fine if some come through firstUDP is faster after all, 
  that's why I'm using itbut this big of a delay is more than I counted 
  on, and it's concerning. I don't know if it would apply across a real 
  network or if it's an artifact of testing in a single process.
 
  Any insights?
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Forwarding ROUTER to PUB

[Charles Remes]

Create a socket for each worker thread and have your main thread resend the 
message down the appropriate socket. Sometimes it isn’t a good idea to try and 
shoe-horn every zeromq socket pattern into your app. :)

On Jan 16, 2014, at 9:54 AM, Lindley French lindl...@gmail.com wrote:

 A problem I was wrestling with was, how do I deal with a TCP connection where 
 messages of different types may arrive, and may need to be dealt with in 
 different threads? The TCP socket can't be touched directly by multiple 
 threads, of course. The obvious solution was to immediately forward messages 
 arriving on the TCP socket to an inproc socket.
 
 I then took it one step further: why not make that inproc socket a PUB socket 
 and make the first part of each message be a topic identifier, so that 
 whichever thread knows how to deal with a particular message can just 
 subscribe to it and ignore the rest?
 
 That's a great design, right up until I try to do it with the TCP socket 
 being a ROUTER. Now, no matter what the first part of the sent message is, 
 the identity will end up being the first part on the receiving end. The 
 PUB/SUB won't work without some tweaking.
 
 I don't want to just drop the identity; that's useful information. I could 
 swap the first two parts; that will work, but it's unintuitive and could 
 cause confusion down the road.
 
 Any other ideas?
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Forwarding ROUTER to PUB

[Charles Remes]

I don’t know. Perhaps someone else has an idea.

On Jan 16, 2014, at 1:45 PM, Lindley French lindl...@gmail.com wrote:

  This is a common issue. If you can’t recover from dropped messages, PUB/SUB 
  is not the correct pattern.
 
 In many cases, this is correct. I do not believe inproc is one of those 
 cases, however. With inproc, you should have full control of who is 
 subscribing and when they come up relative to the publisher. If your 
 subscribers aren't running when you expect them to be running, that's a bug. 
 If they aren't running fast enough, dropping messages *might* be a solution, 
 or it might not. I don't feel that's a decision that can be made in the 
 general case.
 
 Let me put it this way: If I need one-to-many semantics with backpressure and 
 filtering, what should I use? PUB is the only one-to-many socket type. I can 
 write my own filtering code, keep a vector of push sockets, etc. but that 
 seems to defeat the point of ZMQ patterns. PUB is exactly what I want in 
 every way *except* the HWM behavior.
 
 
 On Thu, Jan 16, 2014 at 2:11 PM, Charles Remes li...@chuckremes.com wrote:
 This is a common issue. If you can’t recover from dropped messages, PUB/SUB 
 is not the correct pattern.
 
 This of PUB as a radio antenna. It broadcasts regardless of whether or not 
 anyone is listening. If there are no listeners, every packet gets dropped. If 
 a listener is slow, then packets will get dropped. If you need further 
 guarantees about delivery, then you need to build some kind of protocol (ack, 
 nak, ack window, etc) on top of DEALER/ROUTER.
 
 Also, as of libzmq 3.3, I believe the default HWM is 1000 (to prevent memory 
 exhaustion in the default configuration). If you want “infinite” then 
 setsockopt to -1.
 
 As for the dropped messages on inproc, you need to be careful to confirm that 
 a listener (SUB) is actually up, running and *connected* before you start 
 PUB’ing otherwise the PUB socket will drop messages. Synchronization for this 
 is discussed in the guide. Alternately, just have your PUB “sleep” for a 
 second after the SUB bind/connects and you should be okay.
 
 
 On Jan 16, 2014, at 1:03 PM, Lindley French lindl...@gmail.com wrote:
 
 Well, aside from the router issue I do like the arrangement for easily 
 handling different messages in different places. However, there may be a 
 fatal flaw at the moment: PUB's desire to drop messages at the HWM. While 
 making drop a default behavior for PUB is fine, I really don't like that 
 it's the *only* behavior possible.
 
 Then again, that may or may not be the issue here. I haven't touched the 
 HWM, so it should still be 0 which is theoretically infinite. Nonetheless, a 
 bunch of my messages in a row vanished into the ether somewhere between PUB 
 and SUB inproc sockets.
 
 
 On Thu, Jan 16, 2014 at 1:02 PM, Lindley French lindl...@gmail.com wrote:
 I tend to stuff in as many different features as I can when I'm first 
 learning something new, it helps me get a feel for it.
 
 You should have seen my first major python program.
 
 
 On Thu, Jan 16, 2014 at 12:46 PM, Charles Remes li...@chuckremes.com wrote:
 Create a socket for each worker thread and have your main thread resend the 
 message down the appropriate socket. Sometimes it isn’t a good idea to try 
 and shoe-horn every zeromq socket pattern into your app. :)
 
 On Jan 16, 2014, at 9:54 AM, Lindley French lindl...@gmail.com wrote:
 
  A problem I was wrestling with was, how do I deal with a TCP connection 
  where messages of different types may arrive, and may need to be dealt 
  with in different threads? The TCP socket can't be touched directly by 
  multiple threads, of course. The obvious solution was to immediately 
  forward messages arriving on the TCP socket to an inproc socket.
 
  I then took it one step further: why not make that inproc socket a PUB 
  socket and make the first part of each message be a topic identifier, so 
  that whichever thread knows how to deal with a particular message can just 
  subscribe to it and ignore the rest?
 
  That's a great design, right up until I try to do it with the TCP socket 
  being a ROUTER. Now, no matter what the first part of the sent message is, 
  the identity will end up being the first part on the receiving end. The 
  PUB/SUB won't work without some tweaking.
 
  I don't want to just drop the identity; that's useful information. I could 
  swap the first two parts; that will work, but it's unintuitive and could 
  cause confusion down the road.
 
  Any other ideas?
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http

Re: [zeromq-dev] Timing issues

[Charles Remes]

To save time in additional stack traversals. That’s where things get slow. 
Plus, the batching algorithm could (potentially) be tuned for different 
workloads and exposed as a setsockopt for additional flexibility (though no one 
has done this yet).


On Jan 16, 2014, at 8:53 AM, Lindley French lindl...@gmail.com wrote:

 Maybe I'm missing something, but what purpose is there in disabling Nagle's 
 algorithm, only to then re-implement the same concept one layer higher?
 
 
 On Thu, Jan 16, 2014 at 9:15 AM, Charles Remes li...@chuckremes.com wrote:
 Nagle’s algo is already disabled in the codebase (you can confirm that with a 
 quick grep). I think what Bruno is referring to is that zeromq batches small 
 messages into larger ones before sending. This improves throughput at the 
 cost of latency as expected.
 
 Check out the “performance” section of the FAQ for an explanation:  
 http://zeromq.org/area:faq
 
 
 On Jan 16, 2014, at 7:04 AM, Lindley French lindl...@gmail.com wrote:
 
 Ah, that would explain it, yes. It would be great to have a way of disabling 
 Nagle's algorithm (TCP_NODELAY sockopt).
 
 
 On Thu, Jan 16, 2014 at 4:24 AM, Bruno D. Rodrigues 
 bruno.rodrig...@litux.org wrote:
 Without looking at the code I assume ØMQ is not trying to send each 
 individual message as a TCP PDU but instead, as the name implies, queues 
 messages so it can batch them together and get the performance.
 
 This then means the wire will be filled up when some internal buffer fills, 
 or after a timeout, which looks like 100ms.
 
 On the other hand I can’t see any setsockopt to configure this possible 
 timeout value.
 
 Any feedback from someone else before I have time to  look at the code?
 
 On Jan 15, 2014, at 16:20, Lindley French lindl...@gmail.com wrote:
 
  I have a test case in which I'm communicating between two threads using 
  zmq sockets. The fact that the sockets are in the same process is an 
  artifact of the test, not the real use-case, so I have a TCP connection 
  between them.
 
  What I'm observing is that a lot of the time, it takes ~100 milliseconds 
  between delivery of a message to the sending socket and arrival of that 
  message on the receiving socket. Other times (less frequently) it is a 
  matter of microseconds. I imagine this must be due to some kernel or 
  thread scheduling weirdness, but I can't rule out that it might be due to 
  something in 0MQ.
 
  If I follow the TCP socket write with one or more UDP writes using 
  Boost.Asio, the 100 millisecond delay invariably occurs for the ZMQ TCP 
  message but the UDP messages arrive almost instantly (before the TCP 
  message).
 
  My design requires that the TCP message arrive before *most* of the UDP 
  messages. It's fine if some come through firstUDP is faster after all, 
  that's why I'm using itbut this big of a delay is more than I counted 
  on, and it's concerning. I don't know if it would apply across a real 
  network or if it's an artifact of testing in a single process.
 
  Any insights?
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Forwarding ROUTER to PUB

[Charles Remes]

I don’t think there is any resistance to having pluggable HWM policies. No one 
has wanted this badly enough to write it.

On Jan 16, 2014, at 2:38 PM, Lindley French lindl...@gmail.com wrote:

 That will work, of courseI'm just curious what the resistance is to 
 letting the HWM policy be settable.
 
 
 On Thu, Jan 16, 2014 at 3:27 PM, Goswin von Brederlow goswin-...@web.de 
 wrote:
 On Thu, Jan 16, 2014 at 02:45:31PM -0500, Lindley French wrote:
   This is a common issue. If you can?t recover from dropped messages,
  PUB/SUB is not the correct pattern.
 
  In many cases, this is correct. I do not believe inproc is one of those
  cases, however. With inproc, you should have full control of who is
  subscribing and when they come up relative to the publisher. If your
  subscribers aren't running when you expect them to be running, that's a
  bug. If they aren't running fast enough, dropping messages *might* be a
  solution, or it might not. I don't feel that's a decision that can be made
  in the general case.
 
  Let me put it this way: If I need one-to-many semantics with backpressure
  and filtering, what should I use? PUB is the only one-to-many socket type.
  I can write my own filtering code, keep a vector of push sockets, etc. but
  that seems to defeat the point of ZMQ patterns. PUB is exactly what I want
  in every way *except* the HWM behavior.
 
 
  On Thu, Jan 16, 2014 at 2:11 PM, Charles Remes li...@chuckremes.com wrote:
 
   This is a common issue. If you can?t recover from dropped messages,
   PUB/SUB is not the correct pattern.
  
   This of PUB as a radio antenna. It broadcasts regardless of whether or not
   anyone is listening. If there are no listeners, every packet gets dropped.
   If a listener is slow, then packets will get dropped. If you need further
   guarantees about delivery, then you need to build some kind of protocol
   (ack, nak, ack window, etc) on top of DEALER/ROUTER.
  
   Also, as of libzmq 3.3, I believe the default HWM is 1000 (to prevent
   memory exhaustion in the default configuration). If you want ?infinite?
   then setsockopt to -1.
  
   As for the dropped messages on inproc, you need to be careful to confirm
   that a listener (SUB) is actually up, running and *connected* before you
   start PUB?ing otherwise the PUB socket will drop messages. Synchronization
   for this is discussed in the guide. Alternately, just have your PUB 
   ?sleep?
   for a second after the SUB bind/connects and you should be okay.
  
  
   On Jan 16, 2014, at 1:03 PM, Lindley French lindl...@gmail.com wrote:
  
   Well, aside from the router issue I do like the arrangement for easily
   handling different messages in different places. However, there may be a
   fatal flaw at the moment: PUB's desire to drop messages at the HWM. While
   making drop a default behavior for PUB is fine, I really don't like that
   it's the *only* behavior possible.
  
   Then again, that may or may not be the issue here. I haven't touched the
   HWM, so it should still be 0 which is theoretically infinite. Nonetheless,
   a bunch of my messages in a row vanished into the ether somewhere between
   PUB and SUB inproc sockets.
  
  
   On Thu, Jan 16, 2014 at 1:02 PM, Lindley French lindl...@gmail.comwrote:
  
   I tend to stuff in as many different features as I can when I'm first
   learning something new, it helps me get a feel for it.
  
   You should have seen my first major python program.
  
  
   On Thu, Jan 16, 2014 at 12:46 PM, Charles Remes 
   li...@chuckremes.comwrote:
  
   Create a socket for each worker thread and have your main thread resend
   the message down the appropriate socket. Sometimes it isn?t a good idea 
   to
   try and shoe-horn every zeromq socket pattern into your app. :)
  
   On Jan 16, 2014, at 9:54 AM, Lindley French lindl...@gmail.com wrote:
  
A problem I was wrestling with was, how do I deal with a TCP
   connection where messages of different types may arrive, and may need 
   to be
   dealt with in different threads? The TCP socket can't be touched 
   directly
   by multiple threads, of course. The obvious solution was to immediately
   forward messages arriving on the TCP socket to an inproc socket.
   
I then took it one step further: why not make that inproc socket a PUB
   socket and make the first part of each message be a topic identifier, so
   that whichever thread knows how to deal with a particular message can 
   just
   subscribe to it and ignore the rest?
   
That's a great design, right up until I try to do it with the TCP
   socket being a ROUTER. Now, no matter what the first part of the sent
   message is, the identity will end up being the first part on the 
   receiving
   end. The PUB/SUB won't work without some tweaking.
   
I don't want to just drop the identity; that's useful information. I
   could swap the first two parts; that will work, but it's unintuitive and
   could cause confusion down the road.
   
Any

Re: [zeromq-dev] Helgrind

[Charles Remes]

Search your local repository for ZMQ_MAKE_VALGRIND_HAPPY. I assume helgrind is 
a fork of valgrind so it likely has similar issues.

On Jan 15, 2014, at 8:04 AM, Lindley French lindl...@gmail.com wrote:

 I just ran my zmq program through helgrind to track down an unrelated error, 
 but I noticed that helgrind *really* has some complaints about libzmq. Now, I 
 know helgrind can give plenty of false positives, so maybe there's nothing to 
 worry about. I was just wondering, has anyone looked at the issues reported 
 by helgrind? Has anyone created a zmq suppression file?
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Helgrind

[Charles Remes]

Then you’ll need to dig into it a bit more. I’m at the limit of my knowledge on 
this topic.

On Jan 15, 2014, at 8:34 AM, Lindley French lindl...@gmail.com wrote:

 That macro only appears one place, and it's just zeroing some memory. That 
 might make memcheck happy, but probably wouldn't affect helgrind.
 
 
 On Wed, Jan 15, 2014 at 9:32 AM, Lindley French lindl...@gmail.com wrote:
 Not a fork exactly, it's a tool within valgrind to detect race conditions. 
 The default tool if you don't specify one is memcheck.
 
 
 On Wed, Jan 15, 2014 at 9:24 AM, Charles Remes li...@chuckremes.com wrote:
 Search your local repository for ZMQ_MAKE_VALGRIND_HAPPY. I assume helgrind 
 is a fork of valgrind so it likely has similar issues.
 
 On Jan 15, 2014, at 8:04 AM, Lindley French lindl...@gmail.com wrote:
 
  I just ran my zmq program through helgrind to track down an unrelated 
  error, but I noticed that helgrind *really* has some complaints about 
  libzmq. Now, I know helgrind can give plenty of false positives, so maybe 
  there's nothing to worry about. I was just wondering, has anyone looked at 
  the issues reported by helgrind? Has anyone created a zmq suppression file?
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Inter thread communication for scalability

[Charles Remes]

You need to provide some kind of synchronization mechanism between the producer 
and the consumer. An explicit ACK for each message or an ACK for a bucket of 
messages is one way. Another is to enforce a particular shutdown sequence so 
that the producer always shuts down first. With an infinite LINGER (-1), the 
socket will not close until the consumer has read all messages on queue.


On Jan 15, 2014, at 9:49 AM, Kenneth Adam Miller kennethadammil...@gmail.com 
wrote:

 
 On Jan 15, 2014 9:43 AM, Lindley French lindl...@gmail.com wrote:
 
  The thing is, it should be possible to make inproc sockets give you normal 
  C++ object semantics in certain conditions using placement new and 
  placement delete. However, any such effort is hindered by the fact that 
  libzmq's interface is in C. Even if though you're working in C++, and 
  libzmq is implemented in C++, that interface makes it challenging to figure 
  out how such an upgrade could be implemented.
 
 
  On Wed, Jan 15, 2014 at 10:33 AM, KIU Shueng Chuan nixch...@gmail.com 
  wrote:
 
  During socket shutdown, with linger set to 0, messages in-flight would be 
  dropped?
 
  I use synchronized queues to hold the buffer pointers like the OP but use 
  zeromq to send a signal to the consumer thread to pop a buffer from the 
  queue.
 
  On 15 Jan 2014 22:06, Goswin von Brederlow goswin-...@web.de wrote:
 
  On Tue, Jan 14, 2014 at 03:40:17PM -0500, Lindley French wrote:
   I'm going to caution you about passing pointers through inproc. It may 
   be
   possible to do safely, but I haven't yet figured out how to manage
   ownership semantics in an environment where messages (pointers) can be
   silently dropped.
 
  Does a PUSH/PULL inproc socket ever drop messages?
 
 
 Well I absolutely do not want any messages between threads getting dropped. 
 How can I configure zmq to not drop any in this context?
 

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Inter thread communication for scalability

[Charles Remes]

Let me restate that final point. The producer socket will not close until all 
messages have been SENT from that socket which means they are enqueued 
elsewhere. For inproc, there is no OS buffer or other structure that may 
disappear and cause a message drop from the source socket closing, so your 
consumer will receive those messages.

Perhaps someone more familiar with the internals of inproc transport can 
explain the strength of that guarantee.

On Jan 15, 2014, at 10:01 AM, Charles Remes li...@chuckremes.com wrote:

 You need to provide some kind of synchronization mechanism between the 
 producer and the consumer. An explicit ACK for each message or an ACK for a 
 bucket of messages is one way. Another is to enforce a particular shutdown 
 sequence so that the producer always shuts down first. With an infinite 
 LINGER (-1), the socket will not close until the consumer has read all 
 messages on queue.
 
 
 On Jan 15, 2014, at 9:49 AM, Kenneth Adam Miller 
 kennethadammil...@gmail.com wrote:
 
 
 On Jan 15, 2014 9:43 AM, Lindley French lindl...@gmail.com wrote:
 
  The thing is, it should be possible to make inproc sockets give you normal 
  C++ object semantics in certain conditions using placement new and 
  placement delete. However, any such effort is hindered by the fact that 
  libzmq's interface is in C. Even if though you're working in C++, and 
  libzmq is implemented in C++, that interface makes it challenging to 
  figure out how such an upgrade could be implemented.
 
 
  On Wed, Jan 15, 2014 at 10:33 AM, KIU Shueng Chuan nixch...@gmail.com 
  wrote:
 
  During socket shutdown, with linger set to 0, messages in-flight would be 
  dropped?
 
  I use synchronized queues to hold the buffer pointers like the OP but use 
  zeromq to send a signal to the consumer thread to pop a buffer from the 
  queue.
 
  On 15 Jan 2014 22:06, Goswin von Brederlow goswin-...@web.de wrote:
 
  On Tue, Jan 14, 2014 at 03:40:17PM -0500, Lindley French wrote:
   I'm going to caution you about passing pointers through inproc. It may 
   be
   possible to do safely, but I haven't yet figured out how to manage
   ownership semantics in an environment where messages (pointers) can be
   silently dropped.
 
  Does a PUSH/PULL inproc socket ever drop messages?
 
 
 Well I absolutely do not want any messages between threads getting dropped. 
 How can I configure zmq to not drop any in this context?
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] [zmqpp] Proposed interface addition....possibly breaking

[Charles Remes]

Code up a patch against libzmq and submit it. The C4 process is very liberal 
about what is accepted.

On Jan 15, 2014, at 11:13 AM, Ben Gray b...@benjamg.com wrote:

 Personally I'd prefer if the template parameters where lower case so 
 sendblocking but otherwise I like the idea.
 
 I'm not sure if it would be better to enable a default of blocking, like it 
 is now, so that anyone using send( hello world ) will basically have 
 nothing to change but then require people who used the boolean before to 
 switch to this system or to just require everyone does. But either way this 
 would have to bump the major version.
 
 
 
 On 15 January 2014 13:23, Lindley French lindl...@gmail.com wrote:
 Allow me to clarify: the variadic template syntax with the blocking option 
 would look like:
 
 // Sender
 socket.sendBLOCKING(Hello,5,8.4,Have a nice day);
 
 
 On Wed, Jan 15, 2014 at 8:19 AM, Lindley French lindl...@gmail.com wrote:
 I have an idea for updating the zmqpp::socket interface. Unfortunately, I 
 don't think it's compatible with the current interface, so I want to run it 
 by the list before going down this route.
 
 The zmqpp::socket interface already includes send() and receive() overloads 
 which take a std::string instead of a zmqpp::message. My idea is to extend 
 this to arbitrary lists of types supported by zmqpp::message, using variadic 
 templates. This would allow you to do things like:
 
 // Sender
 socket.send(Hello,5,8.4,Have a nice day);
 
 // Receiver
 std::string str1, str2;
 int val;
 double dval;
 socket.receive(str1,val,dval,str2);
 
 So long as the types match up, everything would just work. And frankly, 
 matching the types up is already a requirement so that's not adding anything 
 new.
 
 I've already figured out how to do this in code. The problem is that the 
 existing send(string) and receive(string) overloads take a second parameter 
 indicating whether or not to block. This second parameter can't be used with 
 the variadic template overload, and if the existing overload is retained, it 
 would be ambiguous. My proposal is to use a function template parameter, 
 BLOCKING or NONBLOCKING, to communicate this information.
 
 The send(zmqpp::message, bool dont_block) overload would be retained, and in 
 fact the variadic overloads would eventually end up calling this one.
 
 There is an alternative approach which doesn't break the existing interface. 
 I don't think it's quite as cool, but it ought to work: use a builder 
 pattern instead of variadic templates to construct multipart messages. I 
 haven't figured out the code yet but the syntax might look something like:
 
 // Sender
 socket.build(Hello)(5,8)(4)(Have a nice day).send();
 
 // Receiver
 std::string str1, str2;
 int val;
 double dval;
 socket.receive()(str1)(val)(dval)(str2);
 
 Not quite as elegant or intuitive, but now compatible with the existing 
 interface.
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Inter thread communication for scalability

[Charles Remes]

If you are doing this from C and can access the raw memory, an inproc socket 
can pass pointers around. If you are using a managed language or one where 
accessing raw memory is difficult, you’ll want to figure out how to “fake” 
passing a pointer (or an object reference). In your case it seems like 
serializing/deserializing would be a big performance hit. That said, if that is 
the direction you must go then pick something fast like msgpack as your 
serializer.


On Jan 14, 2014, at 1:29 PM, Kenneth Adam Miller kennethadammil...@gmail.com 
wrote:

 @AJ No, but I understand exactly why you suggested that. It's because I 
 haven't explained that thread 1 is doing critical work and it needs to 
 offload tasks to other threads as quickly as possible.
 
 @Lindley, Thanks so much for helping me see the truth! I was getting awful 
 confused considering all the different bolony that could go on if I was stuck 
 with semaphores, and I couldn't really re-envision it. Is there any kind of 
 convenience function or core utility for de-serializing the data you receive 
 over inproc messages? 
 
 
 On Tue, Jan 14, 2014 at 12:49 PM, AJ Lewis aj.le...@quantum.com wrote:
 In the zeromq example, couldn't you just skip thread 1 entirely?  Then the
 PULL socket from thread 2 takes uncompressed input from the source,
 compresses it, and shoves it out the PUSH socket to thread 3 for output.
 
 In this case, the PULL socket is the uncompressed pool and the PUSH socket
 is the compressed pool.  Just make sure your uncompressed pool doesn't fill
 up faster than thread 2 can compress it, or you'll need to implement some
 logic to prevent it from using up all the memory.
 
 AJ
 
 On Tue, Jan 14, 2014 at 01:16:32PM -0500, Lindley French wrote:
  In this case your buffers are really just messages, aren't they? A thread
  grabs one (receives a message), processes it, and writes the result into
  another buffer (sends a message).
 
  The hard part is that ZeroMQ sockets don't like to be touched by multiple
  threads, which complicates the many-to-many pattern you have going here.
  I'm no expert, but I would suggest
 
  Each pool, A and B, becomes a single thread with two ZMQ inproc sockets,
  one push and one pull. These are both bound to well-known endpoints. All
  the thread does is continually shove messages from the pull socket to the
  push socket.
 
  Each thread in Thread set 1 has a push inproc socket connected to pool
  A's pull socket.
 
  Each thread in Thread set 2 has a pull inproc socket connected to pool
  A's push socket and a push inproc socket connected to pool B's pull socket.
  For each message it receives, it just processes it and spits it out the
  other socket.
 
  The thread in Thread set 3 has a pull inproc socket connected to pool B's
  push socket. It just continually receives messages and outputs them.
 
  This may seem complicated because concepts that were distinct before
  (buffer pools and worker threads) are now the same thing: they're both just
  threads with sockets. The critical difference is that the buffer pools
  bind to well-known endpoints, so you can only have a few of them, while the
  worker threads connect to those well-known endpoints, so you can have as
  many as you like.
 
  Will this perform as well as your current code? I don't know. Profile it
  and find out.
 
 
  On Tue, Jan 14, 2014 at 12:23 PM, Kenneth Adam Miller 
  kennethadammil...@gmail.com wrote:
 
   So, I have two pools of shared buffers; pool A, which is a set of buffers
   of uncompressed data, and pool B, for compressed data. I three sets of
   threads.
  
   Thread set 1 pulls from pool A, and fills buffers it receives from pool A
   up with uncompressed data.
  
   Thread set 2 is given a pool from A that has recently been filled. It
   pulls a buffer from pool B, compresses from A into B, and then returns the
   buffer it was given, cleared, back to pool A.
  
   Thread set 3 is a single thread, that is continually handed compressed
   data from thread set 2, which it outputs. When data is finished output, it
   returns the buffer to pool B, cleared.
  
   Can anybody describe a scheme to me that will allow thread sets 1  2 to
   scale?
  
   Also, suppose for pools A and B, I'm using shared queues that are just C++
   stl lists. When I pop from the front, I use a lock for removal to make 
   sure
   that removal is deterministic. When I enqueue, I use a separate lock to
   ensure that the internals of the STL list is respected (don't want two
   threads receiving iterators to the same beginning node, that would 
   probably
   corrupt the container or cause data loss, or both). Is this the 
   appropriate
   way to go about it? Thread sets 1  2 will likely have more than one
   thread, but there's no guarantee that thread sets 1  2 will have equal
   threads.
  
   I was reading the ZeroMQ manual, and I read the part about multi-threading
   and message passing, and I was wondering what approaches should be 

Re: [zeromq-dev] Choose zeromq communication pattern for data acquisition from sensors

[Charles Remes]

If you need ACK, then I recommend reading chapter 4 of the guide (advanced 
patterns). Take a look at the lazy pirate and clone patterns.


On Dec 18, 2013, at 1:02 PM, Roman consulttele...@gmail.com wrote:

 I'm trying to figure out the appropriate zeromq communication pattern to use 
 in my case, and I'd like some assistance getting started down the right path.
 My task is the following - to get data from remote sensors at centralized 
 server for subsequent processing. Sensors are installed in different 
 locations and connected via Internet. Connection may be NAT-ed, can be 
 unreliable. Sensors send quite frequently (once in 1 to 20 seconds) small 
 amounts of near real-time data (up to 1-2 kbytes). A number of sensors can be 
 quite large, up to thousand per server.
 There is no need to ACK received data but heartbeat would be appropriate in 
 order to store data locally on sensor.
 
 Thank you in advance for your considerations!
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ZeroMQ IPC Windows

[Charles Remes]

Yes, IPC on Windows uses TCP sockets so there is no performance benefit to 
using IPC transport versus TCP transport. They are implemented the same way.

We would love to have a named pipes implementation for Windows.


On Dec 15, 2013, at 9:10 PM, Francisco Freire francisco.m.fre...@gmail.com 
wrote:

 Charles, can you explain me better? Are you talking about using tcp?
 
 
 2013/12/16 Francisco Freire francisco.m.fre...@gmail.com
 Yes
 http://api.zeromq.org/4-1:zmq-ipc (ZeroMQ API reference)
 
 
 
 2013/12/16 crocket crockabisc...@gmail.com
 Does ZeroMQ lack support for IPC sockets on windows?
 
 
 On Fri, Dec 13, 2013 at 9:34 PM, Francisco Freire 
 francisco.m.fre...@gmail.com wrote:
 Hi,
 
 I'm doing some research at Instituto Superior Técnico concerning distributed 
 systems and inter-process communication. Last days I was studying a little 
 bit of ZeroMQ and now want to use it in my project since my goal is to 
 improve performance between the communication of two different apps running 
 on the same machine.
 
 I want to know if there is already someone implementing the ØMQ local 
 inter-process communication transport using Windows Sockets and I would like 
 to know more about this. As I said my focus is on performance and if needed I 
 could start implementing this functionality. 
 
 Meanwhile I will use tcp for this communication.
 
 Best Regards,
 
 Francisco Freire
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ZeroMQ IPC Windows

[Charles Remes]

It supports IPC via sockets instead of using named pipes or other more 
efficient structures on Windows. There has been periodic talk over the years 
about using a real named pipe implementation but it has never culminated into 
real code.


On Dec 15, 2013, at 7:42 PM, crocket crockabisc...@gmail.com wrote:

 Does ZeroMQ lack support for IPC sockets on windows?
 
 
 On Fri, Dec 13, 2013 at 9:34 PM, Francisco Freire 
 francisco.m.fre...@gmail.com wrote:
 Hi,
 
 I'm doing some research at Instituto Superior Técnico concerning distributed 
 systems and inter-process communication. Last days I was studying a little 
 bit of ZeroMQ and now want to use it in my project since my goal is to 
 improve performance between the communication of two different apps running 
 on the same machine.
 
 I want to know if there is already someone implementing the ØMQ local 
 inter-process communication transport using Windows Sockets and I would like 
 to know more about this. As I said my focus is on performance and if needed I 
 could start implementing this functionality. 
 
 Meanwhile I will use tcp for this communication.
 
 Best Regards,
 
 Francisco Freire
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] When unreliability is desired

[Charles Remes]

Feel free to rubber duck with the list as much as you want. :)

cr

On Dec 5, 2013, at 3:08 PM, Lindley French lindl...@gmail.com wrote:

 Disregard the question about timers, it appears this was a change made to 
 Crossroads IO after the fork. I found the relevant commit:
 https://github.com/crossroads-io/libxs/commit/2df873a435ff139cf9d1b10b666d75e6dc6da442#diff-45c9686e09ed422318a2da658745dedd
 I will revert the code to the 0MQ way of doing things for now.
 
 
 On Thu, Dec 5, 2013 at 3:48 PM, Lindley French lindl...@gmail.com wrote:
 I'm trying to make sure I understand how the timer_ids work, since these were 
 not in the Crossroads IO code. It appears several classes define timer_id 
 constants as anonymous enums. Furthermore, the values of these constants 
 don't appear to follow any particular pattern, *except* that constants with 
 the same name (same purpose?) are assigned the same value. Therefore, I 
 expect I ought to adopt tx_timer_id and rx_timer_id from pgm_sender without 
 changing their values for udp_sender etc. Am I understanding this correctly?
 
 
 On Thu, Dec 5, 2013 at 3:06 PM, Lindley French lindl...@gmail.com wrote:
 It appears that at some point, activate_in was renamed to restart_input, and 
 likewise with output. I'm assuming there are no functional changes other than 
 the names in the UDP case. It also appears that zap_msg_available() can be 
 ignored for now. Let me know if I'm wrong about these.
 
 
 On Thu, Dec 5, 2013 at 2:01 PM, Lindley French lindl...@gmail.com wrote:
 Okay, I'm working on porting the UDP stuff to the latest ZMQ now. Most of the 
 changes necessary are purely a matter of namespace, but there are a few 
 others that seem to reflect design changes in ZMQ since XS was forked. These 
 are the ones I'm not immediately sure how to solve:
 
 1) The XS UDP code references encoder_t and decoder_t, but ZMQ has 
 v1_encoder_t and v2_encoder_t etc instead.
 2) i_engine appears to have three pure virtual methods that are not defined 
 in the XS code: restart_input, restart_output, and zap_msg_available().
 
 Any suggestions how to resolve these?
 
 
 On Thu, Dec 5, 2013 at 10:52 AM, Lindley French lindl...@gmail.com wrote:
 Thanks, I found it. I don't see any reason given as to why UDP support was 
 reverted, though. Are there issues with the code, philosophical objections, 
 etc?
 
 
 On Wed, Dec 4, 2013 at 5:16 PM, Ivan Pechorin ivan.pecho...@gmail.com wrote:
 UDP support was reverted in libxs just before 1.2.0 release: check the 
 commits history on github around June 13th, 2012.
 
 05.12.2013 6:48 пользователь Lindley French lindl...@gmail.com написал:
 
 Can you clarify where in the Crossroads IO library the UDP transport code 
 lives? I've downloaded the 1.2.0 tarball here:
 http://download.crossroads.io/libxs-1.2.0.tar.gz
 but so far, I don't see a UDP transport in that code. I also checked the 
 github version:
 https://github.com/crossroads-io/libxs
 but I don't see it there either.
 
 On Wed, Nov 27, 2013 at 11:26 AM, Charles Remes li...@chuckremes.com wrote:
 That defunct library (crossroads io) has the code that you want. That lib was 
 a fork of zeromq, so moving the UDP transport from that library to zeromq 
 should be easy (for varying degrees of easy). Once it makes it into zeromq, 
 it will be supported.
 
 
 On Nov 27, 2013, at 9:50 AM, Lindley French lindl...@gmail.com wrote:
 
 I've never used ZeroMQ before so writing up a new transport would be just a 
 bit ambitious right now. (I did write something in Java last year that, in 
 retrospect, was solving basically the same problem as ZeroMQ so I have some 
 familiarity with the problem space.)
 
 I'm also leery of adopting a defunct library for a new project.
 
 I'll keep the udp transport option in mind.
 
 
 On Tue, Nov 26, 2013 at 1:18 PM, Pieter Hintjens p...@imatix.com wrote:
 Hi Lindley,
 
 The right solution would be to make a UDP transport for ZeroMQ. It's
 not a trivial project but could start with, for instance, just pub/sub
 (like PGM).
 
 It might be worth looking at Crossroads.io for that, which is
 abandoned but had afair a UDP transport, and shared the same original
 codebase with ZeroMQ.
 
 -Pieter
 
 On Tue, Nov 26, 2013 at 6:15 PM, Lindley French lindl...@gmail.com wrote:
  I have a networking application that I'd like to use ZeroMQ in. However, my
  use-case demands minimum latency even at the expense of lost messages. I'm
  weary of using ZeroMQ's TCP transport because if packets are dropped, TCP
  will block further messages until it has retransmitted the last one, and I
  don't want that behavior.
 
  I don't mind FEC codes or other strategies to improve reliability by 
  sending
  more data up-front, but I do not need complete reliability and I want to
  avoid retransmission of messages, or at the least avoid blocking later
  messages if earlier ones need to be retransmitted.
 
  Is there an existing ZeroMQ transport that will provide the behavior I 
  want?
  I was thinking

Re: [zeromq-dev] When unreliability is desired

[Charles Remes]

That defunct library (crossroads io) has the code that you want. That lib was a 
fork of zeromq, so moving the UDP transport from that library to zeromq should 
be easy (for varying degrees of easy). Once it makes it into zeromq, it will be 
supported.


On Nov 27, 2013, at 9:50 AM, Lindley French lindl...@gmail.com wrote:

 I've never used ZeroMQ before so writing up a new transport would be just a 
 bit ambitious right now. (I did write something in Java last year that, in 
 retrospect, was solving basically the same problem as ZeroMQ so I have some 
 familiarity with the problem space.)
 
 I'm also leery of adopting a defunct library for a new project.
 
 I'll keep the udp transport option in mind.
 
 
 On Tue, Nov 26, 2013 at 1:18 PM, Pieter Hintjens p...@imatix.com wrote:
 Hi Lindley,
 
 The right solution would be to make a UDP transport for ZeroMQ. It's
 not a trivial project but could start with, for instance, just pub/sub
 (like PGM).
 
 It might be worth looking at Crossroads.io for that, which is
 abandoned but had afair a UDP transport, and shared the same original
 codebase with ZeroMQ.
 
 -Pieter
 
 On Tue, Nov 26, 2013 at 6:15 PM, Lindley French lindl...@gmail.com wrote:
  I have a networking application that I'd like to use ZeroMQ in. However, my
  use-case demands minimum latency even at the expense of lost messages. I'm
  weary of using ZeroMQ's TCP transport because if packets are dropped, TCP
  will block further messages until it has retransmitted the last one, and I
  don't want that behavior.
 
  I don't mind FEC codes or other strategies to improve reliability by sending
  more data up-front, but I do not need complete reliability and I want to
  avoid retransmission of messages, or at the least avoid blocking later
  messages if earlier ones need to be retransmitted.
 
  Is there an existing ZeroMQ transport that will provide the behavior I want?
  I was thinking maybe epgm would do the trick, even though I don't really
  need multicast. Ideally, I'd want a transport that uses pure UDP for
  messages, perhaps with some TCP behind the scenes for out-of-band
  handshaking.
 
  I may end up just using UDP myself for the time-critical messages, and
  ZeroMQ for less critical data, but I'd prefer to avoid multiple-API creep.
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] How can a subscriber drop its own message..

[Charles Remes]

I would suggest something like this:

frame0 - topic
frame1 - UUID
frame2 - message body

Every subscriber would need to check frame1 for a UUID that they originated. If 
they find a match, ignore the rest of the message.

In other words, this has to be done in your application logic. There is no 
mechanism for telling zeromq subscribers to ignore messages that they 
originate particularly since SUB sockets can *only receive.*

cr

On Nov 20, 2013, at 1:35 AM, George Kumar grgkum...@gmail.com wrote:

 I am using a push/sub pattern in a ZMQ client. When the server publishes the 
 messages, I want all subscribers to see the message except the one that sent 
 it originally. In other words on subscribe, I want to see all messages, 
 except the ones sent by me. What is the best way to achieve it.
 
 Thanks.
 George
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Request-reply broker with one worker and many async tasks

[Charles Remes]

I'll paste my response from IRC here since it is relevant and I'll add a few 
more thoughts.

cremes:cosmoharrigan_: how is the response sent back via the callback? If it's 
via that single DEALER socket and the callback runs in another thread, you need 
to protect the socket with a mutex.
[3:46pm]cremes:Best to just give each thread its own socket. It's a cleaner 
design plus it let's you push thread  lock management into zeromq.

If the reply from the callback needs to be routed back to the client, you 
should continue to use DEALER(client) - ROUTER (broker) - DEALER (worker). 
In this case there is a single worker but the message passing works exactly the 
same.

cr

On Nov 6, 2013, at 3:43 PM, Cosmo Harrigan cosmo.harri...@singularityu.org 
wrote:

 Hi,
 
 I want to implement a design similar to the Request-Reply Broker from the 
 zguide (https://raw.github.com/imatix/zguide/master/images/fig17.png) but 
 instead of having multiple workers running ahead of time, I want to have one 
 worker that receives all the requests, and spawns asynchronous operations 
 (tasks) to handle them and send the responses.
 
 What would be the proper combination of socket types to achieve this?
 
 Thanks,
 Cosmo Harrigan
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] handling DNS updates

[Charles Remes]

There is no flag or configuration option to work around that.

I would recommend handling name resolution directly in your code and then 
building the transport connection string using the resulting IP address.

Make sense?

cr

On Oct 21, 2013, at 11:03 AM, pablo fernandez fernandezpabl...@gmail.com 
wrote:

 We're serving a zmq_proxy through DNS and would like clients to re-resolve 
 the DNS address after some period of time. AFAIK sockets now cache the dns 
 value forever, this is inconvenient.
 
 Is there a config flag or workaround that doesn't involve 
 disconnecting/connecting all clients after a period of time?
 
 Thank you.
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Proposal for ZeroMQ certificate format

[Charles Remes]

I was kind of hoping the bike shedding would go on for a few more days. :-P

cr

On Oct 16, 2013, at 9:12 AM, shancat shannenlap...@gmail.com wrote:

  I suggest we use SHA512 truncated to 6 bytes, and prefixed by the
  first 6 bytes of the sender's public key. To create a fraudulent
  certificate an attacker would have to find a double collision.
 
 Very good idea, I agree with this entirely.
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] 4.x api changes

[Charles Remes]

I have a vague memory that zmq_init and zmq_term were supposed to be deprecated 
and replaced by zmq_ctx_new and zmq_ctx_term. If that's the case, are these 
older functions going to be removed from the 4.x api?

I have a feeling the answer is no since the 4.01 stable has already been 
released. Breaking API changes aren't allowed anymore under semantic versioning 
rules.

Can anyone clarify?

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] 0MQ 4 - how to detect closed stream socket ?

[Charles Remes]

Via a timeout.

On Oct 11, 2013, at 12:15 PM, itli...@schrievkrom.de wrote:

 How can I detect, that the server has closed a stream client socket ?
 
 Marten
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] how to implement the subscriber in the black box pattern

[Charles Remes]

Will the upcoming version 4 release enforce proper socket pairings so that 
doing something odd like connecting PUSH and SUB will raise an error?

cr

On Sep 19, 2013, at 2:46 AM, Pieter Hintjens p...@imatix.com wrote:

 You can't connect a PUSH to a SUB; use a PUB socket to send the data.

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] how to implement the subscriber in the black box pattern

[Charles Remes]

Good to know!

If someone does make this mistake, further calls against that socket should 
probably return an error that makes the problem obvious. We have EFSM and other 
library-created errors, so perhaps EINCOMPATIBLESOCKET (or similar) should be 
returned.

On Sep 19, 2013, at 5:01 AM, Pieter Hintjens p...@imatix.com wrote:

 OK, I've checked and libzmq master _does_ check socket types, thanks
 to our cherished Martin Hurton.
 
 Commit 2a4aef is the one. The code is in mechanism.cpp:check_socket_type.
 
 My comments about raising the error to the user still apply; we now
 get a failed connection but no signalling to the application.
 
 -Pieter
 
 On Thu, Sep 19, 2013 at 11:57 AM, Pieter Hintjens p...@imatix.com wrote:
 This was certainly our intention. I'm not sure the code does that type
 checking yet, but the protocol supports it.
 
 However, it's not as simple as it sounds, since connections are
 background actions and invisible to applications. So connection
 failures are too. We have the same issue with security, and the
 current answer is to simply deny the connection, as if there was no
 network connection. But this won't help users debug problems. I'm not
 a fan of having to do extra work (context monitoring) to catch such
 errors.
 
 Perhaps the better approach is to raise a fatal error on the socket so
 all further operations on it fail, after a socket type error, or a
 security failure. The error would happen asynchronously, but would at
 least show on the radar without extra work.
 
 -Pieter
 
 
 
 
 
 On Thu, Sep 19, 2013 at 11:34 AM, Charles Remes li...@chuckremes.com wrote:
 Will the upcoming version 4 release enforce proper socket pairings so that 
 doing something odd like connecting PUSH and SUB will raise an error?
 
 cr
 
 On Sep 19, 2013, at 2:46 AM, Pieter Hintjens p...@imatix.com wrote:
 
 You can't connect a PUSH to a SUB; use a PUB socket to send the data.
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] notify pattern

[Charles Remes]

I usually recommend the majordomo pattern for work of this nature. It isn't a 
perfect fit, but you could modify the broker piece pretty easily to keep 
track of tasks and reissue them after a timeout. It's a good foundation to 
start from.

cr

On Sep 9, 2013, at 8:16 AM, R ryan.dee...@gmail.com wrote:

 Hello folks,
 
 I am new to zmq and was implementing the basic reliable queuing pattern to 
 distribute work across multiple workers. The workers are intended to do a 
 task that could take several minutes once kicked off. 
 
 I wanted to maintain a list of tasks and workers in the queue so that if a 
 worker has crashed/is unresponsive for an hour(or so) I can send the task to 
 another worker. I will have the worker notify the queue once a task is 
 complete so that the task is taken out of the queue. 
 
 Is there a better pattern to do this. 
 
 Thanks a lot.
 
 R
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Mountain Lion build issues: strndup error, and 'make check' gave 1 of 20 tests failed on test_shutdown_stress

[Charles Remes]

This is a long standing problem with all versions of OSX. By default they have 
insufficient file descriptors for the stress test to function properly. We do 
not want the tests to rely on sudo or some other privilege elevator to 
increase the number of FDs. Perhaps there should be a warning printed on OSX 
when this specific test fails along with a short instruction on how to manually 
fix it.

cr

On Sep 3, 2013, at 7:03 PM, Peter Jurgensen peterjurgen...@gmail.com wrote:

 The standup problem was easily fixed.  It was the 'make check' fail that I 
 was concerned about.
 
 test_shutdown_stress should work, but didn't.
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Storm rejecting ZeroMQ

[Charles Remes]

I remember when Nathan Marz was filing bug reports and asking questions on this 
list to help get things stabilized. Oh wait, no I don't because as far as I 
know he never did any of those things.

I think 3.3+ works pretty well for a lot of people.

This is common with open source software so I wouldn't read too much into it. 
Nathan preferred to roll his own with Netty. I hope he succeeds.

cr


On Aug 26, 2013, at 10:22 AM, Andy Pook andy.p...@gmail.com wrote:

 comment from Nathan Marz in a presentation called Lessons Learned Building 
 Storm
 
 http://www.infoq.com/presentations/storm-lessons
 42:50
 
 Q: “What do you think about the ZeroMQ library?”
 
 A: pause… It saved a lot of time initially. Then we ran into a lot of 
 problems 
 with it. A lot of limitations. So we’re actually getting rid of ZeroMQ in 
 Storm … We’re replacing ZeroMQ with Netty. A lot of problems I had with 
 ZeroMQ 
 were, like, a lot of weird stuff with the community … Lots of regression in 
 the code. Nothing above 2.1.7 works … It just a weird, very unstable piece of 
 software. It could have been a really awesome library. But it just has all 
 these weird problems with it.
 
 
 Thoughts?
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Storm rejecting ZeroMQ

[Charles Remes]

The 2.x to 3.x transition was very painful. The original 3.0 and 3.1 pretty 
much didn't work with anything else (and let's not forget about the original 
4.x branch either). That all got addressed by 3.2 and its successors. It was a 
bad time for the community because Martin Sustrik and Mikko were unhappy with 
the commit policies that Pieter devised. They went their own way (Crossroads 
I/O and then Nanomsg) while many of us worked hard to get 3.2 stable. Things 
are looking pretty good now.

cr

On Aug 26, 2013, at 2:38 PM, Brian Knox bri...@talksum.com wrote:

 I've been a user of zeromq for a few years now.  There were in my opinion 
 some rough spots around the 2.x to 3.x transition.  However, I stuck with it, 
 and from my perspective both the library and the community around the library 
 has been maturing nicely.
 
 As Charles said - search for Nathan looking for help on the list and being 
 ignored, or for him filing bug reports that were ignored, if his statements 
 give you pause.  I'm an empiricist :)
 
 
 
 
 On Mon, Aug 26, 2013 at 3:24 PM, Charles Remes li...@chuckremes.com wrote:
 I remember when Nathan Marz was filing bug reports and asking questions on 
 this list to help get things stabilized. Oh wait, no I don't because as far 
 as I know he never did any of those things.
 
 I think 3.3+ works pretty well for a lot of people.
 
 This is common with open source software so I wouldn't read too much into it. 
 Nathan preferred to roll his own with Netty. I hope he succeeds.
 
 cr
 
 
 On Aug 26, 2013, at 10:22 AM, Andy Pook andy.p...@gmail.com wrote:
 
  comment from Nathan Marz in a presentation called Lessons Learned Building
  Storm
 
  http://www.infoq.com/presentations/storm-lessons
  42:50
 
  Q: “What do you think about the ZeroMQ library?”
 
  A: pause… It saved a lot of time initially. Then we ran into a lot of 
  problems
  with it. A lot of limitations. So we’re actually getting rid of ZeroMQ in
  Storm … We’re replacing ZeroMQ with Netty. A lot of problems I had with 
  ZeroMQ
  were, like, a lot of weird stuff with the community … Lots of regression in
  the code. Nothing above 2.1.7 works … It just a weird, very unstable piece 
  of
  software. It could have been a really awesome library. But it just has all
  these weird problems with it.
 
 
  Thoughts?
 
  ___
  zeromq-dev mailing list
  zeromq-dev@lists.zeromq.org
  http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Storm rejecting ZeroMQ

[Charles Remes]

Zed's issues were with the widespread use of assert in the code base. Also, 
at that time, the library did very little validation of incoming data so it was 
easy to crash the library using a fuzzing attack. Again, most (all?) of those 
concerns have been resolved with 3.2 and later.

It's hard to say if the Nitro folks think 3.2+ resolves their earlier concerns 
or if their decision to roll their own still holds up. I imagine it was 
probably a good idea for their use-case considering they have 10s of thousands 
of mostly quiescent connections in a pub/sub format, so they need a more 
efficient subscription filtering mechanism. I think Nanomsg has solved some of 
this too by using a different kind of trie to track subscriptions.

cr

On Aug 26, 2013, at 3:15 PM, Steven McCoy steven.mc...@miru.hk wrote:

 On 26 August 2013 11:22, Andy Pook andy.p...@gmail.com wrote:
 Nothing above 2.1.7 works … It just a weird, very unstable piece of
 software. It could have been a really awesome library. But it just has all
 these weird problems with it.
 
 Didn't Zed Shaw also have similar comments, presumably the people behind 
 Nitro too?
 
 I think a lot of these were resolved and Pieter introduced the new workflow 
 to ensure things don't break similarly in future.
 
 -- 
 Steve-o 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] 0MQ 3.2.4 - any planned release date?

[Charles Remes]

On Aug 6, 2013, at 11:41 AM, Pieter Hintjens p...@imatix.com wrote:

 Hi Lukasz,
 
 I'll release 3.2.4 from the 3-x stabilization repo after the summer.
 It has a few cleanups and backports; nothing major but nice to have.
 
 In theory the current libzmq master would be 3.3.0. However we've made
 so many changes such as security that I'm wondering if we should not
 instead aim for a 4.0 release.
 
 Thoughts?
 
 -Pieter

If the project is still following semantic versioning, then I believe the new 
APIs probably merit a 4.0 designation. All of the work on secure transport is a 
major addition.

My 2 cents…

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Build ZMQ as static library under MinGW

[Charles Remes]

Sounds like a limitation of MinGW32. Have you tried asking in the #mingw 
channel on irc or checking their FAQs?

cr

On Jun 25, 2013, at 7:43 AM, Beyer, Marcel marcel.be...@oce.com wrote:

 Hello,
  
 I am using MinGW as compiler and tried to build ZMQ as a static library when 
 I received this error while running configure:
  
 “configure: error: Building static libraries is not supported under MinGW32”
  
 Why is this? Is there an easy way to include the support for MinGW?
  
  
 
 This message and attachment(s) are intended solely for use by the addressee 
 and may contain information that is privileged, confidential or otherwise 
 exempt from disclosure under applicable law. If you are not the intended 
 recipient or agent thereof responsible for delivering this message to the 
 intended recipient, you are hereby notified that any dissemination, 
 distribution or copying of this communication is strictly prohibited. If you 
 have received this communication in error, please notify the sender 
 immediately by telephone and with a 'reply' message. Thank you for your 
 co-operation. ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Unable to run Weather update server (wuserver.c) example

[Charles Remes]

The second bind in the original example (rc = zmq_bind (publisher, 
ipc://weather.ipc);) is completely unnecessary because the client never 
connects to an ipc endpoint.

Delete that line of code and try the example again.

cr

On Jun 17, 2013, at 10:41 AM, Shaukat Mahmood Ahmad wr...@sma.im wrote:

 Dear Pieter,
 
 Regarding your suggestion I have a question regarding Weather Update
 Server example 
 (https://github.com/imatix/zguide/blob/master/examples/C/wuserver.c),
 said example uses dual binding tcp and ipc is it neceesary for PubSub
 style communication to have two bindings? And as per your suggestion
 will I have to replace the second binding with tcp binding as per
 following code snippet.
 
 
 --- Original Code Snippet from Example
 
 void *context = zmq_ctx_new ();
 void *publisher = zmq_socket (context, ZMQ_PUB);
 int rc = zmq_bind (publisher, tcp://*:5556);
 assert (rc == 0);
 rc = zmq_bind (publisher, inproc://workers);
 assert (rc == 0);
 
 
 --- Revised code as per your suggestion (if i got it right)
 void *context = zmq_ctx_new ();
 void *publisher = zmq_socket (context, ZMQ_PUB);
 int rc = zmq_bind (publisher, tcp://*:5556);
 assert (rc == 0);
 rc = zmq_bind (publisher, tcp://*.5557);
 assert (rc == 0);
 
 
 
 I have tried this on my side Server and Client run successfully,
 however no updates are collected by client, can you help me to resolve
 this issue so that I can run PubDub example on my side.
 
 
 On Mon, Jun 17, 2013 at 6:51 PM, Pieter Hintjens p...@imatix.com wrote:
 Shaukat,
 
 Sorry about this. I'd recommend changing the ipc:// endpoint to a
 tcp:// one (with valid endpoints). It will work just the same. The
 later examples all do this.
 
 -Pieter
 
 On Mon, Jun 17, 2013 at 3:10 PM, shancat shannenlap...@gmail.com wrote:
 Not that I know of but I haven't gone that deep into zmq on Windows. You
 might be sacrificing performance but I don't think there's up to date
 performance information on Windows. Maybe you can do some benchmarks? If go
 to http://api.zeromq.org/3-2:zmq-bind and go to the part about transports
 you can go to each inbuilt transport doc page and read up.
 
 On Jun 17, 2013 10:44 PM, Shaukat Mahmood Ahmad wr...@sma.im wrote:
 
 Thanks for being supportive, one last thing if I go with C/C++ on
 Windows what else will I have  to sacrifice (other than ipc://)? Can
 you guide me to a reference / contents / list of zeroMQ protocols /
 transports with supported platforms.
 
 Regards,
 SMA
 
 On Mon, Jun 17, 2013 at 5:34 AM, shancat shannenlap...@gmail.com wrote:
 It depends what you're trying to do with it but yes I would use it on
 Windows myself. You might want to look at the C# binding
 (https://github.com/zeromq/clrzmq) or the native C# port
 (https://github.com/zeromq/netmq).
 
 On Jun 17, 2013 4:37 AM, Shaukat Mahmood Ahmad wr...@sma.im wrote:
 
 Thank shancat, yes finally I found it in API reference guide and in an
 old thread (under zmq mailing list) where some one had requested and
 tried to add ipc:// support for windows in 2011 but no success till
 2013. As a user can you recommend zMQ on windows as I was evaluating
 it for a commercial software product currently running on Windows?
 
 
 On Sun, Jun 16, 2013 at 8:22 PM, shancat shannenlap...@gmail.com wrote:
 From the documentation: The inter-process transport is currently only
 implemented on operating systems that provide UNIX domain sockets. So
 that's why bit doesn't work on Windows, failing with a helpful
 Protocol
 not
 supported message.
 
 On Jun 16, 2013 10:30 PM, Shaukat Mahmood Ahmad wr...@sma.im wrote:
 
 I am new to zeroMQ so sorry for incomplete information in my last
 message in this thread, after debugging I have found that following
 code is failing on Windows platform with error # 135 and message
 against this error is Protocol not supported, there is some
 advancement but I am still confused  to get exact reason of failure
 (as I am unable to locate some data in help / manual).
 
 Environment:
 Windows 7 Professional 64bit
 Visual Studio 2012
 zeroMQ   version 3.2.3
 
 code snippet:
 
rc = zmq_bind (publisher, ipc://weather.ipc); // returns rc = -1
err = zmq_errno();   // return err = 135
errmsg = (char*) zmq_strerror(err); = returns error message string
 as Protocol not supported
 
 Can anybody tell me about ipc protocol support on Windows platform.
 
 -
 End Note:
 Now I am really  confused about statements made about zeroMQ such as
 it will provide ease will reduce time to solve bigger problems, but
 the way it is presented and documented is really useless, to use this
 library either you  have extra time to debug the available source and
 re-write to fit your need or simply find some other suitable library.
 
 0/100  for zeroMQ because I have wasted my whole weekend on trying to
 understand this useless library.
 
 
 
 On Sun, Jun 16, 2013 at 2:12 PM, Shaukat 

Re: [zeromq-dev] epgm memory leak - need hepl

[Charles Remes]

Also, make sure that the message you send is closed after you send it. I 
don't know if this binding automatically handles that for you when you send a 
message but it is worth investigating. See the zmq_msg_close() man page for 
more information.

cr

On Jun 17, 2013, at 1:48 PM, Parag Patel parag.pa...@fusionts.com wrote:

 Any ideas about this?  We’re seeing something similar with a Java 
 application.  We noticed that the JVM memory size is not growing, however, 
 the process’s memory space is growing.
 
  
 
 From: zeromq-dev-boun...@lists.zeromq.org 
 [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Alexander Zhitlenok
 Sent: Friday, June 14, 2013 6:07 PM
 To: zeromq-dev@lists.zeromq.org
 Subject: [zeromq-dev] epgm memory leak - need hepl
  
 
 Hello,
 
 My name is Alex Zhitlenok.
 
 We are using ZeroMq in a custom C# Application on Windows.
 
 We use the multicast/epgm protocol.
 
 When we run our application the memory consistently grows and sometimes the 
 app crashes due to an Out of Memory Exception.
 
 To try and detect what causes the problem, we’ve created a very simple test 
 (just a few lines to avoid GC influence, etc.)
 
 Even when we run this simple test program, the memory continues to grow (at a 
 slow and linear rate, but continues to grow regardless)
 
 To avoid “no listeners” problem in the test we run mdump.exe as a fake-client 
 listening to the multicast port.
 
 What are we doing wrong?
 
 We use
 ZeroMQ 3.2.3
 OpenPGM 5.2.122 (http://code.google.com/p/openpgm/)
 Clrzmq build for 3.2.3 (https://github.com/zeromq/clrzmq)
 Mdump (https://community.informatica.com/solutions/informatica_mtools)
  
 Software\Hardware:
 Win7 Professional\64 bit
 10g network
  
 Here is the test code (the real addresses are substituted with XYZ):
  
 static void Main(string[] args)
 {
 ZmqContext context = ZmqContext.Create();
 ZmqSocket soc = context.CreateSocket(ZeroMQ.SocketType.PUB);
  
 soc.MulticastHops = 16;
 soc.MulticastRate = 10;
 //soc.SendBufferSize = 100;
 soc.SendHighWatermark = 100;
  
 soc.Bind(epgm://192.168.XYZ.XYZ;239.10.10.10:PORT);
  
 byte[] test = new byte[1000];
 test[1] = (byte)('X');
  
 for (int i = 0; ; ++i)
 {
 test[0] = (byte) ('0' + (i%10));
 SendStatus ss = soc.Send(test);
 if( ss != SendStatus.Sent )
 
 System.Diagnostics.Debug.WriteLine(String.Format(Status:{0}, ss));
 Thread.Sleep(100);
 }
 }
  
 Thank you,
 Alex
  
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] About zeroMQ 3.2.3 2.2.2

[Charles Remes]

You are confusing the library with the available bindings. There are libzmq 
bindings available in C, C++, Ruby, Python, Scheme, Perl and probably 20 more 
languages. Use the one that is the best fit for you and your problem space. 
There is no performance difference between the C and C++ bindings.

The library is written in C++. That has not changed from 2.x to 3.x. Martin 
Sustrik is working on a new library called nanomsg that he is writing in C. 
That is a different library.

cr

On Jun 17, 2013, at 2:13 PM, Shaukat Mahmood Ahmad wr...@sma.im wrote:

 zeroMQ version 3.2.3 is shipped with zmq.h (pure c implementation),
 however the older version (2.2.2) does contain zmq.hpp (C++ Version).
 So I am curious to know about recommendation is zeroMQ 3.2.3 is
 recomended with C only. I was reading
 artcilehttp://www.250bpm.com/blog:4 (Why should I have written ZeroMQ
 in C, not C++ Martin Sústrik, May 10th, 2012) I guess at that time the
 zeroMQ version was 2.2.2 and the next version was written in pure C?
 Most of examples under current documentation are also written in pure
 C, however at github zmq.hpp is also available so what is best option
 to write applications using zeroMQ C or C++?  is it just matter of
 choice and expertise or will there be some performance hit with C++?
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Any concerns about having SUB bind to socket and many PUBS publish data to it?

[Charles Remes]

Pay attention to the SNDHWM on the PUB side and the RCVHWM on the SUB side. If 
you absolutely, positively cannot drop any messages then you should take a look 
at the clone pattern in the guide or consider using DEALER/ROUTER sockets 
along with an explicit ack/nak protocol (think zmodem or tcp). 

Outside of that, I can't think of anything else noteworthy.

cr

On Jun 17, 2013, at 2:27 PM, Christopher Crotty crotty.christop...@gmail.com 
wrote:

 We have a need for many asynchronous publishers and a single subscriber for 
 data.
 I have setup a SUB that binds to a socket and several PUBS which connect it 
 and send data.
 
 The SUB appears to get the data from all PUBS ok, but I'm curious if there 
 any internal gotchas that I need to be careful of.
 
 Thanks,
 Chris
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] efficiency of multipart messages

[Charles Remes]

Do whichever method best expresses your solution. There is no speed or latency 
difference.

As a reminder, a multipart message is sent as an atomic unit. Until you send 
the final frame, the receiving side cannot see any parts.

cr

On May 23, 2013, at 9:05 AM, Davide Faconti faco...@icarustechnology.com 
wrote:

 Hi,
 
 I have a question that, probably is a bit naive.
 I have a published that send complex data (an easy to serialize struct, that 
 is about 200 bytes, but I guess this is not relevant).
 
 I publish this data in a non periodic way, in other words sometimes I have to 
 deliver 10-20 samples of this data to the subscriber. Let's use N to refer to 
 the number of data samples I want to send.
 
 I have two way to do that:
 
 1) Either I publish N messages.
 2) I use multipart messages provided by ZMQ; one message the message as N 
 parts.
 
 Is one of these two ways more efficient in terms of, let's say latency or 
 speed?
 
 Thanks in advance.
 
 -- 
 Davide Faconti
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ruby binding for zmq

[Charles Remes]

rbzmq is a C extension. It can be used with MRI or Rubinius (but JRuby no 
longer supports C extensions).

ffi-rzmq uses FFI to interface to libzmq so it is supported by *all* of the 
Ruby runtimes. I'm the author and may be a bit biased, but it is updated, 
tested and used quite a bit more than rbzmq.

The APIs exposed by each of them are incompatible meaning that you can't write 
code for rbzmq and then switch to ffi-rzmq without changing some of your code. 
I tried to sync this up some time ago but we couldn't agree on how to do it.

I suggest that you use ffi-rzmq.

On Apr 18, 2013, at 12:27 AM, Pritesh Acharya priteshacha...@gmail.com wrote:

 I tried  rbzmq from https://github.com/zeromq but it doesn't install. I think 
 this binding is no longer support for newer zmq. When I looked around i found 
 ffi-rzmq is also ruby binding for zmq. I wanna know the difference between 
 gem zqm and ffi-zmq. Which one should I use ?
 Thanks,
 Pritesh
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Problem with DEALER - ROUTER (assertions in mailbox.cpp:79)

[Charles Remes]

Niels,

Before we go any further, can you tell us if you have read the guide yet? 
Chapter 4 would very likely answer your questions so I'm curious to know if you 
went through that chapter (and its examples) but you still have a problem or 
question to resolve. Please advise.

cr

On Apr 7, 2013, at 1:09 PM, Niels Berglund niels.it.bergl...@gmail.com wrote:

 Hi all!
 So I am playing around with ZeroMq i order to get to grips how to create 
 applications using it. I am running on Windows using the ClrZmq binding, and 
 I am getting assert exceptions as per the title of this mail. When reading 
 about this assert, it seems that it is related to threading issues, but I 
 can't for the life of me see how I can handle the code in any other way than 
 what I am doing.
 
 Anyway, the test consists of two applications a client (DEALER socket) and a 
 service (ROUTER socket). The client sends in a very tight loop a number of 
 messages - this is configured in a config file. The service (the ROUTER) 
 receives the messages and sends a reply back. On the client I receive the 
 message and increments a counter of how many messages I have received.
 
 The problem is that up to a certain number of messages (let's say the 
 configured value is 50,000) everything works fine, then when I set the number 
 to be higher (60,000 for example) I get this assert error. The thing is 
 however that I do not receive it after the 50,000 - but I can receive it 
 anywhere in the process (i.e. 5,000 , 10,000 etc). That baffles me a bit. The 
 HighWatermark for both send and receive is set to a very large number, much 
 higher than the configured value.
 
 My assumption is that this assert has something to do with threading, but I 
 cannot see how to architect the app and running everything from one thread. 
 At the moment I have a TestSocket class, where I create the socket, have the 
 receive and send logic etc. A receive will fire an event exposing the message 
 received.  A TestClass creates the TestSocket class, subscribes to the 
 receive event and also initiates the send of messages. 
 
 I have read how the created socket always have to be accessed by the thread 
 that created it, but I can't see how I can do that and have a poller which 
 polls for messages. So in my TestSocket class I create the poller and then 
 starts to poll on a separate thread.
 
 Does anyone have any insights in this?!!
 
 Thanks!
 
 Niels
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Perennial thread safety question

[Charles Remes]

On Apr 8, 2013, at 12:40 PM, Matthew Woehlke matthew.woeh...@kitware.com 
wrote:

 On 2013-04-08 08:37, Pieter Hintjens wrote:
 So sharing sockets between threads is really a non-question, it's
 only something people will try when they have not yet understood how
 to use 0MQ properly. The one exception to this general rule is in
 esoteric cases where you may create a socket in one thread, for use by
 an other thread.
 
 I'm actually doing this. Essentially, I have a 'thread class' that has a 
 run() method that is a thread entry point, but the ctor/dtor execute in 
 the 'parent' thread.
 
 Is it safe to create and set up, and tear down, my 0MQ socket in the 
 ctor/dtor (i.e. in thread A, and poll/recv/etc. in thread B)? Is there 
 something I should be doing to make it safe?
 
 The class does guarantee that ctor/run()/dtor will never execute in 
 parallel.

This is answered in the FAQ (http://www.zeromq.org/area:faq). Short answer is: 
Yes, go ahead and do this. If it crashes, protect the socket with some kind of 
memory barrier (e.g. mutex).

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Perennial thread safety question

[Charles Remes]

+1. The FAQ and guide are scary for a reason. :)

cr

On Apr 8, 2013, at 5:10 PM, Pieter Hintjens p...@imatix.com wrote:

 The scary text emerged as the best way to keep people from skimming
 the docs, sharing sockets between threads, and getting crashes. Anyone
 who knows what a full memory barrier is can migrate sockets. The rest
 of us don't do that, and stick to simpler, safer patterns (each thread
 creates the sockets it needs).
 
 -Pieter
 
 
 On Tue, Apr 9, 2013 at 12:04 AM, Matthew Woehlke
 matthew.woeh...@kitware.com wrote:
 On 2013-04-08 16:12, Charles Remes wrote:
 On Apr 8, 2013, at 12:40 PM, Matthew Woehlke wrote:
 Is it safe to create and set up, and tear down, my 0MQ socket in the
 ctor/dtor (i.e. in thread A, and poll/recv/etc. in thread B)? Is there
 something I should be doing to make it safe?
 
 This is answered in the FAQ (http://www.zeromq.org/area:faq). Short answer 
 is: Yes, go ahead and do this. If it crashes, protect the socket with some 
 kind of memory barrier (e.g. mutex).
 
 Ah... I think I may have seen that at some point. However I was
 specifically remembering reading this in the guide:
 
 Don't share ØMQ sockets between threads. ØMQ sockets are not
 threadsafe. Technically it's possible to migrate a socket from one
 thread to another but it demands skill. The only place where it's
 remotely sane to share sockets between threads are in language bindings
 that need to do magic like garbage collection on sockets.
 
 ...which seems to contradict the FAQ. Maybe some less scary text should
 be used? (As I read the FAQ, it's perfectly safe to migrate a socket
 just as long as the migration is 'clean', e.g. memory gets synchronized
 between the last access by thread A and before the first access by
 thread B.)
 
 --
 Matthew
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] The meaning of ZMQ_POLLOUT for inproc

[Charles Remes]

It should always be set but if I recall correctly there is an open bug on this 
topic. For inproc transports, the ZMQ_POLLOUT doesn't work. 

Search the open issues on the bug tracker.

cr

On Mar 27, 2013, at 7:44 AM, Kah-Chan Low kahchan...@yahoo.com wrote:

 If I have a router-dealer connection using inproc, and I set ZMQ_SNDHWM to 0 
 (i.e. no limit). If I call getsockopt(ZMQ_EVENTS, ...) on the scoket, won't 
 the ZMQ_POLLOUT be always be set?  I mean, is there any time when the socket 
 is not ready for write?
 
 Thanks!
 
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Felix Geisendörfer's hack, C4 process

[Charles Remes]

I think this is a fantastic idea. One of the other open source projects where I 
contribute (github.com/rubinius) has had a similar policy in place for the last 
several years. The project literally has over 100 committers. Their idea was 
that if you create even one patch (or pull request) that is accepted, then you 
get a commit bit to the repository.

The fear has always been that this would lead to some maliciously committing 
bad patches or otherwise causing trouble, but those incidents have been very 
rare in practice.

So, put me down as another supporter for this idea.

On Mar 20, 2013, at 9:34 AM, Pieter Hintjens p...@imatix.com wrote:

 Hi,
 
 You might have seen Felix's pull request hack:
 http://felixge.de/2013/03/11/the-pull-request-hack.html
 
 We've been using something similar (C4) in libzmq and other projects
 for a while.
 
 Do people have opinion on whether we're enjoying C4, and whether it
 would be good to add contributors to the maintainers team for any
 given project on a more aggressive basis?
 
 E.g. once you make a patch that people like, you get commit rights
 (meaning you can merge other peoples' patches) automatically.
 
 -Pieter
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq::mtrie_t::rm_helper(..)

[Charles Remes]

Doesn't ring a bell for me.

A few questions…

Are all endpoints using 3.2.2 or is it a mix of 3.2.2 and 2.2?

What transports are you using (inproc, tcp, ipc)?

Are you using XPUB and XSUB directly or are the endpoints actually PUB/SUB but 
you are seeing a crash in the parent?

How many subscriptions do you have on each endpoint?

Are there any intermediate devices between the edge endpoints (e.g. a FORWARDER 
device)?

Any chance you can use a Windows OS that is not EOL? :)

cr

On Mar 5, 2013, at 8:55 AM, Pau p...@teleopsia.com wrote:

 
 Hi, they have found today 2 servers publishing using xsub crashed in 
 mtrie.cpp[line 293] in function zmq::mtrie_t::rm_helper(..) line 
 apparently tryng to acces if (pipes-empty ()). I guess pipes was rubish 
 but I could not debug and I have been unable to reproduce at the moment. 
 The function was called from xpub_t when publishing a message. It 
 happened at the same time in 2 apps in the same machine but unrelated.
 
 We work with 3.2.2 in Windows XP in an environment where there are like 
 5 o 6 different sockets per application and arounf 5 or 6 normally on. I 
 am trying to provoque it back to make a test case but at the moment I 
 couldn't.
 
 Rings a bell to anybody?,
 
 thanks,
 
 Pau Ceano
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] problem installing zmq on mac

[Charles Remes]

It's a known issue. By default OSX only allows 256 file descriptors per 
process, so the stress test fails. Take a look at the tuning guide on 
zeromq.org and it will explain how to resolve this.

Not a bug.

On Feb 21, 2013, at 5:08 AM, Pritesh Acharya priteshacha...@gmail.com wrote:

 
 
zeromq 3.2.3: tests/test-suite.log
 
 
 # TOTAL: 20
 # PASS:  19
 # SKIP:  0
 # XFAIL: 0
 # FAIL:  1
 # XPASS: 0
 # ERROR: 0
 
 .. contents:: :depth: 2
 
 FAIL: test_shutdown_stress
 ==
 
 test_shutdown_stress running...
 Invalid argument (stream_engine.cpp:96)
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Using zmq_ipc between threads?

[Charles Remes]

Using IPC transport to talk between two threads works. I have used that 
technique myself. 

If you have a short code snippet that shows how it fails, please post it.

cr

On Feb 21, 2013, at 10:16 AM, Tobias Scharnberg 
t.scharnb...@wenzel-elektronik.de wrote:

 
 Hi,
 maybe this is a really dumb question... but anyway. Can I use zmq_ipc 
 transport for exchanging messages between two threads? I've build 
 classes with router and dealer sockets which are connecting fine with 
 each other when used in different processes.
 Now I've tried to use them between two threads, because I didn't want to 
 exchange contexts for using inproc. But nothing seems to get through 
 between these two sockets from thread to thread?
 
 Greetings,
 Tobias
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Non-Blocking REQ/REP Server

[Charles Remes]

Take a look at the man page for zmq_poll. You can do a non-blocking poll for 
incoming messages on your socket. If it returns immediately with 0, then no 
sockets in your pollset have pending messages to read.

Be aware that when zmq_poll does indicate that you have messages, you must read 
*all* of them from the socket before zmq_poll will work again. I'm pretty sure 
the man page explains this.

Good luck.

On Feb 19, 2013, at 8:44 AM, Lee Sylvester lee.sylves...@gmail.com wrote:

 Hey guys,
 
 So, I've integrated ØMQ into my server.  Now, I want to use ØMQ as a means to 
 supply information to a HTTP server from a separate management app.  So, in 
 theory, it will look something like this
 
 int read_zmq_connections() {
int ret = 0;
while (zmq_has_messages(zmq_responder)) {
char *str = s_recv(zmq_responder);
parse_new_data(str);
free(str);
++ret;
}
return ret;
 }
 
 This way, if there are no messages on zmq_responder, then the function will 
 simply return.  What I don't know how to do (and can't quite find) is how to 
 check if messages exist on the connection.  Can anyone please point me in the 
 right direction?
 
 The reason why I need this non-blocking is that I will only be calling 
 'read_zmq_connections' approximately once every five minutes and I don't want 
 my app to hang while waiting for messages.
 
 Thanks loads in advance,
 Lee
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Non-Blocking REQ/REP Server

[Charles Remes]

Hmmm, I'm not sure that's exactly right.

The basic idea is that you want to check the return code from zmq_poll. If it 
is greater than 0, then the socket can be read from. You should then read from 
the socket until no more messages are available. I don't know how it works with 
the #s_recv() function (presumably that is part of the czmq binding) but you 
want to read until the socket is empty or you get EAGAIN. Perhaps that function 
does that for you under the covers. 

So, the loop should be around reading from the socket and *not* around 
zmq_poll. Does that make sense?



On Feb 19, 2013, at 9:16 AM, Lee Sylvester lee.sylves...@gmail.com wrote:

 Thank you, that's great.  So, based on what I've read, does this look correct 
 for what I'm trying to accomplish?
 
 int read_zmq_connections() {
   zmq_pollitem_t items [] = {
   { zmq_responder, 0, ZMQ_POLLIN, 0 }
   };
   while (1) {
   zmq_msg_t message;
   zmq_poll(items, 1, 0);
   if (items[0].revents  ZMQ_POLLIN) {
   char *str = s_recv(zmq_responder);
   parse_new_data(str);
   free(str);
   } else {
   break;
   }
   }
   return 0;
 }
 
 Thanks,
 Lee
 
 
 
 
 On 19 Feb 2013, at 14:52, Charles Remes li...@chuckremes.com wrote:
 
 Take a look at the man page for zmq_poll. You can do a non-blocking poll for 
 incoming messages on your socket. If it returns immediately with 0, then no 
 sockets in your pollset have pending messages to read.
 
 Be aware that when zmq_poll does indicate that you have messages, you must 
 read *all* of them from the socket before zmq_poll will work again. I'm 
 pretty sure the man page explains this.
 
 Good luck.
 
 On Feb 19, 2013, at 8:44 AM, Lee Sylvester lee.sylves...@gmail.com wrote:
 
 Hey guys,
 
 So, I've integrated ØMQ into my server.  Now, I want to use ØMQ as a means 
 to supply information to a HTTP server from a separate management app.  So, 
 in theory, it will look something like this
 
 int read_zmq_connections() {
  int ret = 0;
  while (zmq_has_messages(zmq_responder)) {
  char *str = s_recv(zmq_responder);
  parse_new_data(str);
  free(str);
  ++ret;
  }
  return ret;
 }
 
 This way, if there are no messages on zmq_responder, then the function will 
 simply return.  What I don't know how to do (and can't quite find) is how 
 to check if messages exist on the connection.  Can anyone please point me 
 in the right direction?
 
 The reason why I need this non-blocking is that I will only be calling 
 'read_zmq_connections' approximately once every five minutes and I don't 
 want my app to hang while waiting for messages.
 
 Thanks loads in advance,
 Lee
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Non-Blocking REQ/REP Server

[Charles Remes]

Yes, you are missing out on being able to differentiate between reads  writes 
since you aren't checking the revents. However, in your case you only have a 
single socket and you only register for POLLIN, so you can just use the return 
code and skip the hard stuff. Any time it returns 1 then you know that your 
socket is readable.


On Feb 19, 2013, at 10:18 AM, Lee Sylvester lee.sylves...@gmail.com wrote:

 Okay, thanks.  I'm looking at the reference and I can see that zmq_poll 
 returns the number of items, but it feels like I'm missing something when I 
 rely on that :-S
 
 Lee
 
 
 On 19 Feb 2013, at 15:52, Charles Remes li...@chuckremes.com wrote:
 
 Hmmm, I'm not sure that's exactly right.
 
 The basic idea is that you want to check the return code from zmq_poll. If 
 it is greater than 0, then the socket can be read from. You should then read 
 from the socket until no more messages are available. I don't know how it 
 works with the #s_recv() function (presumably that is part of the czmq 
 binding) but you want to read until the socket is empty or you get EAGAIN. 
 Perhaps that function does that for you under the covers. 
 
 So, the loop should be around reading from the socket and *not* around 
 zmq_poll. Does that make sense?
 
 
 
 On Feb 19, 2013, at 9:16 AM, Lee Sylvester lee.sylves...@gmail.com wrote:
 
 Thank you, that's great.  So, based on what I've read, does this look 
 correct for what I'm trying to accomplish?
 
 int read_zmq_connections() {
 zmq_pollitem_t items [] = {
 { zmq_responder, 0, ZMQ_POLLIN, 0 }
 };
 while (1) {
 zmq_msg_t message;
 zmq_poll(items, 1, 0);
 if (items[0].revents  ZMQ_POLLIN) {
 char *str = s_recv(zmq_responder);
 parse_new_data(str);
 free(str);
 } else {
 break;
 }
 }
 return 0;
 }
 
 Thanks,
 Lee
 
 
 
 
 On 19 Feb 2013, at 14:52, Charles Remes li...@chuckremes.com wrote:
 
 Take a look at the man page for zmq_poll. You can do a non-blocking poll 
 for incoming messages on your socket. If it returns immediately with 0, 
 then no sockets in your pollset have pending messages to read.
 
 Be aware that when zmq_poll does indicate that you have messages, you must 
 read *all* of them from the socket before zmq_poll will work again. I'm 
 pretty sure the man page explains this.
 
 Good luck.
 
 On Feb 19, 2013, at 8:44 AM, Lee Sylvester lee.sylves...@gmail.com wrote:
 
 Hey guys,
 
 So, I've integrated ØMQ into my server.  Now, I want to use ØMQ as a 
 means to supply information to a HTTP server from a separate management 
 app.  So, in theory, it will look something like this
 
 int read_zmq_connections() {
 int ret = 0;
 while (zmq_has_messages(zmq_responder)) {
char *str = s_recv(zmq_responder);
parse_new_data(str);
free(str);
++ret;
 }
 return ret;
 }
 
 This way, if there are no messages on zmq_responder, then the function 
 will simply return.  What I don't know how to do (and can't quite find) 
 is how to check if messages exist on the connection.  Can anyone please 
 point me in the right direction?
 
 The reason why I need this non-blocking is that I will only be calling 
 'read_zmq_connections' approximately once every five minutes and I don't 
 want my app to hang while waiting for messages.
 
 Thanks loads in advance,
 Lee
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Non-Blocking REQ/REP Server

[Charles Remes]

It looks better except for the use of size in the loop control. The return 
code from zmq_poll is the *number of sockets* that have pending events. The way 
you are using it appears as though you believe the return code indicates the 
*number of messages* which is incorrect. The way the loop is written now, it 
will loop once and exit at most.


On Feb 19, 2013, at 10:29 AM, Lee Sylvester lee.sylves...@gmail.com wrote:

 Okay, thank you.  So I now have the following:
 
 int read_zmq_connections() {
   zmq_pollitem_t items [] = {
   { zmq_responder, 0, ZMQ_POLLIN, 0 }
   };
   errno = 0;
   int ret = 0, size = zmq_poll(items, 1, 0);
   zmq_msg_t message;
   while ( size  0  errno == 0 ) {
   char *str = s_recv(zmq_responder);
   parse_new_data(str);
   free(str);
   ++ret;
   --size;
   }
   return ret;
 }
 
 I'm going to test it a little later once I've written my client code.  :-)
 
 Thanks again.
 
 Cheers,
 Lee
 
 
 
 On 19 Feb 2013, at 16:21, Charles Remes li...@chuckremes.com wrote:
 
 Yes, you are missing out on being able to differentiate between reads  
 writes since you aren't checking the revents. However, in your case you only 
 have a single socket and you only register for POLLIN, so you can just use 
 the return code and skip the hard stuff. Any time it returns 1 then you know 
 that your socket is readable.
 
 
 On Feb 19, 2013, at 10:18 AM, Lee Sylvester lee.sylves...@gmail.com wrote:
 
 Okay, thanks.  I'm looking at the reference and I can see that zmq_poll 
 returns the number of items, but it feels like I'm missing something when I 
 rely on that :-S
 
 Lee
 
 
 On 19 Feb 2013, at 15:52, Charles Remes li...@chuckremes.com wrote:
 
 Hmmm, I'm not sure that's exactly right.
 
 The basic idea is that you want to check the return code from zmq_poll. If 
 it is greater than 0, then the socket can be read from. You should then 
 read from the socket until no more messages are available. I don't know 
 how it works with the #s_recv() function (presumably that is part of the 
 czmq binding) but you want to read until the socket is empty or you get 
 EAGAIN. Perhaps that function does that for you under the covers. 
 
 So, the loop should be around reading from the socket and *not* around 
 zmq_poll. Does that make sense?
 
 
 
 On Feb 19, 2013, at 9:16 AM, Lee Sylvester lee.sylves...@gmail.com wrote:
 
 Thank you, that's great.  So, based on what I've read, does this look 
 correct for what I'm trying to accomplish?
 
 int read_zmq_connections() {
   zmq_pollitem_t items [] = {
   { zmq_responder, 0, ZMQ_POLLIN, 0 }
   };
   while (1) {
   zmq_msg_t message;
   zmq_poll(items, 1, 0);
   if (items[0].revents  ZMQ_POLLIN) {
   char *str = s_recv(zmq_responder);
   parse_new_data(str);
   free(str);
   } else {
   break;
   }
   }
   return 0;
 }
 
 Thanks,
 Lee
 
 
 
 
 On 19 Feb 2013, at 14:52, Charles Remes li...@chuckremes.com wrote:
 
 Take a look at the man page for zmq_poll. You can do a non-blocking poll 
 for incoming messages on your socket. If it returns immediately with 0, 
 then no sockets in your pollset have pending messages to read.
 
 Be aware that when zmq_poll does indicate that you have messages, you 
 must read *all* of them from the socket before zmq_poll will work again. 
 I'm pretty sure the man page explains this.
 
 Good luck.
 
 On Feb 19, 2013, at 8:44 AM, Lee Sylvester lee.sylves...@gmail.com 
 wrote:
 
 Hey guys,
 
 So, I've integrated ØMQ into my server.  Now, I want to use ØMQ as a 
 means to supply information to a HTTP server from a separate management 
 app.  So, in theory, it will look something like this
 
 int read_zmq_connections() {
 int ret = 0;
 while (zmq_has_messages(zmq_responder)) {
  char *str = s_recv(zmq_responder);
  parse_new_data(str);
  free(str);
  ++ret;
 }
 return ret;
 }
 
 This way, if there are no messages on zmq_responder, then the function 
 will simply return.  What I don't know how to do (and can't quite find) 
 is how to check if messages exist on the connection.  Can anyone please 
 point me in the right direction?
 
 The reason why I need this non-blocking is that I will only be calling 
 'read_zmq_connections' approximately once every five minutes and I 
 don't want my app to hang while waiting for messages.
 
 Thanks loads in advance,
 Lee
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq_unbind fails on inproc?

[Charles Remes]

Honestly I think that ENOENT is probably more correct. I can put together a 
pull request to fix the man pages for libzmq and zeromq3-x with the change.

cr

On Feb 19, 2013, at 12:34 PM, Pieter Hintjens p...@imatix.com wrote:

 Hi Chuck,
 
 Thanks for making the test cases. What would be better, ENOENT or EINVAL?
 
 -Pieter
 
 On Tue, Feb 19, 2013 at 3:30 PM, Charles Remes li...@chuckremes.com wrote:
 Pieter, thank you for confirming the problem.
 
 I have opened an issue for this (LIBZMQ-507) and added code to reproduce the 
 problem to the issues repository under the same issue number.
 
 While messing with this I also discovered two other minor problems. The 
 zmq_unbind and zmq_disconnect functions do not set zmq_errno to EINVAL when 
 given an unknown endpoint like the documentation suggests. Instead, they set 
 zmq_errno to ENOENT. I have opened issues for each of these and provided 
 repro code too. I am not sure these are library bugs or documentation bugs. 
 :)
 
 cr
 
 On Feb 19, 2013, at 12:39 AM, Pieter Hintjens p...@imatix.com wrote:
 
 I can confirm this fails on Linux in C. The sleep is irrelevant since
 connect/bind on inproc happen immediately.
 
 -Pieter
 
 On Mon, Feb 18, 2013 at 11:57 PM, Charles Remes li...@chuckremes.com 
 wrote:
 libzmq: 3.2.2
 OS: OSX 10.8.2
 
 I'd appreciate it if someone else could confirm that calling zmq_unbind on 
 an inproc transport always fails (it works for tpc and ipc without fail). 
 It doesn't matter what socket type is under test, nor does it matter if 
 you sleep to give the io thread time to complete the zmq_bind call.
 
 rc = LibZMQ.zmq_bind(socket, inproc://some_address);
 assert(rc == 0);
 
 rc = LibZMQ.zmq_unbind(@socket, inproc://some_address);
 assert(rc == 0); // zmq_errno returns ENOENT on OSX
 
 If someone else can confirm this, I'll open a ticket. If I am doing some 
 stupidly wrong, let me know that too. :)
 
 Thanks!
 
 cr
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq_unbind fails on inproc?

[Charles Remes]

On Feb 18, 2013, at 4:57 PM, Charles Remes li...@chuckremes.com wrote:

 libzmq: 3.2.2
 OS: OSX 10.8.2
 
 I'd appreciate it if someone else could confirm that calling zmq_unbind on an 
 inproc transport always fails (it works for tpc and ipc without fail). It 
 doesn't matter what socket type is under test, nor does it matter if you 
 sleep to give the io thread time to complete the zmq_bind call.

Oops, let me fix some obvious problems in the pseudo-code.

rc = zmq_bind(socket, inproc://some_address);
assert(rc == 0);

rc = zmq_unbind(socket, inproc://some_address);
assert(rc == 0); // zmq_errno returns ENOENT on OSX

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Repeated creation of ZMQ sockets lead to memory bloating?

[Charles Remes]

This has to do with memory fragmentation.

See here:  
http://stackoverflow.com/questions/447899/why-does-my-c-program-not-free-memory-as-it-should

Also, search the mailing list archives for a discussion on this topic a year or 
two ago.

cr

On Feb 15, 2013, at 4:55 PM, Kah-Chan Low kahchan...@yahoo.com wrote:

 Hi!
 In my experimental run I repeatedly create and destroy ZMQ sockets.  I simply 
 let the zmq::socket_t destructor take care of the deallocation.  I assume 
 that ZMQ knows how to clean up the connections the destructed sockets have 
 previously established.
 As time goes on I notice that the virtual memory of my process keeps going 
 up.  I am wondering if ZMQ has anything to do with the memory bloating.
 Thanks!!
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] ZMQ, PGM and Message Loss

[Charles Remes]

On Feb 8, 2013, at 8:22 AM, Dave dla...@gmail.com wrote:

 Hi,
 
 Sorry if this is a stupid question, but I can't seem to find information 
 online about how ZeroMQ deals with message loss when using multicast. Can 
 someone point me to some info or a guide on this? How do subscribers request 
 retransmissions? Is this entirely up to the application, or is it supported 
 by the message library?

You can configure the retry mechanisms in epgm itself but those can get overrun 
too. If you need more guarantees on delivery, take a look a the guid at 
zguide.zeromq.org and read up on the clone pattern.

cr


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Should 0MQ be used and what socket types to use?

[Charles Remes]

On Jan 25, 2013, at 11:58 PM, Niels Berglund niels.it.bergl...@gmail.com 
wrote:

 Hi all!
 I am working for a company whose business is similar to online financial 
 trading, where the similarities are the requirement for low latency and high 
 throughput. 
 
 The systems are situated in data centers across the world, and clients 
 connect over the internet to an individual system. A system consists of a 
 software router which routes requests from clients to back-end services, and 
 responses to the clients from the services. The services may also send 
 unsolicited messages to the clients.
 
 There are a limited number of back-end services, but there could 
 theoretically be an infinite number of concurrent users.  Obviously there are 
 limitations how many clients can connect to one specific software router, but 
 there would then be multiple machines with software routers for the clients 
 to connect to. 
 
 We are now looking at the future architecture, and part of that is 
 communication mechanisms. The underlying OS for this is Windows and at the 
 moment we are using Microsoft's socket library. Our socket code are 
 functional, but not pretty and I have a feeling that if we were to change to 
 something like 0MQ, etc., it would be beneficial for us.
 
 So my questions are:
 1. Would an architecture like the above be feasible using 0MQ?
 2. What 0MQ socket types would be best for an architecture like this? My 
 thoughts are that clients would connect to the software router using a DEALER 
 socket (seeing that we want full asynch, and that the client could send 
 multiple requests without requiring a reply in between). The client-facing 
 socket in the software router would be a ROUTER socket, but I am not sure if 
 the client service facing socket in the software router should be a ROUTER or 
 DEALER? As for the services, they would connect to the software router using 
 a DEALER socket.


1. Yes, this architecture is very feasible. I have personally created a very 
similar architecture for my own trading applications.

2. I recommend using DEALER/ROUTER exclusively for any bidirectional 
communications. Those socket types can be viewed as the foundation for any 
protocol (that is, you can create any of the other patterns such as push/pull 
or pub/sub using just dealer/router). 

You will want heartbeating to detect dead (or slow) clients and services, so 
again dealer/router is the correct choice. I have found the other socket types 
to primarily be useful for intra-thread communication since they are (from the 
programmer perspective) uni-directional.

My system runs in production on the Windows platform. I have also deployed some 
services to linux but Windows is the primary target platform.

If you have other specific questions, feel free to ask.

cr


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.

[Charles Remes]

You will save a ton of time if you read the guide (zguide.zeromq.org) and the 
FAQ. The guide may appear quite long, but for what you intend to accomplish you 
could probably stop after chapter 5.

I'll give you a head start on a few things.

1. inproc
0mq supports 4 transport types. 

inproc is for communicating between sockets *within the same* process. It 
essentially does some pointer trickery so it's very fast. It does *not* work 
between processes.

ipc is for inter-process communication. Most OSes support IPC via local unix 
domain sockets. 0mq is no different and just relies on the OS mechanism.

tcp is for communication between processes on the same or different boxes. All 
messages go up and down the entire TCP stack unless the OS has optimizations 
for localhost communication.

epgm is for multicast. It cannot be used for talking to processes on the same 
box. 

Luckily 0mq lets a single socket bind (and/or connect) to multiple transports 
at the same time so each limitation isn't a big deal.

2. Subscription forwarding
As you have discovered, the 3.2+ releases of libzmq make the PUB socket 
intelligent about where to send data. It can save quite a bit of bandwidth if 
you are on a constrained network.

3. Understanding 0mq
We encourage and invite you to read the source, but the project (and community) 
prides itself on having some of the best documentation around (on par with 
commercial products). You should not need to read the source to understand 
the library. Please join us on irc.freenode.net in #zeromq or post your 
questions here. Be warned that one of the first questions asked is did you 
read the guide? You'll get less help if you answer no because most of us old 
timers are tired of answering novice questions that are clearly answered in the 
excellent docs (guide, man pages, faq).

Welcome!

cr

On Jan 26, 2013, at 1:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote:

 Hello,
   My name is David W, I am a professional code monkey that was first
 introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 )
 but only now have gotten the chance to sit down and start learning the
 library.  I am on a sabbatical until ideal after PyCon since I got
 laid off and taking the time out to hit a bucket list ( including
 learning 0MQ ).
 
  On that note, a few years back as a re-invent the wheel project to
 learn Twisted and COMET I started writing a web MUD engine and
 centered the architecture around two message pipelines:  User action's
 were locked stepped ( User A moved left, tell server, wait for it to
 say yes/no), broadcast to other User's that User A moved left,
 broadcast to all of User A's group they moved left.  NPC's were just
 headless User's driven by a behavior time/tick subprocess that hooked
 upto the same pipelines.I set that project aside because I
 realized I needed a message queue of some sort and really didn't want
 to setup Rabbit or anything super industrial.
 
  Now along arrives 0MQ and since this is a personal project the
 priority is more about understanding how 0MQ works then accomplishing
 the actual project.   In the above example I can imagine using 0mq's
 inproc socket's where client's are SUB types ( subscribe to
 map/domain, subscribe to group chat ) and their is a master process
 that has a router socket for incoming work and a pub socket for
 products [ User A in map 1 moved left] ).
 
 So here's my questions:
   For PUB/SUB the impression is that the actual queue sit's on the
 client socket.  PUB pushes a message to all client's [ regardless of
 setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket
 filter's/clears the queue down to what the client is subscribed to.
 Is this correct or is the subscription more intelligent ( PUB keep's a
 subscription roster, see's no one is subscribed, drop's the message OR
 client receives a message, isn't subscribed so it drops the message ).
 
  Has anyone had any experience running multiple SUB based client's
 inside of one process and are their any severe consequences.  I
 imagine a SUB socket is going to instantiate the needed structures to
 hold a queue, the actual socket, and other house keeping structures
 but so far small tests (1-10 sockets) hasn't show much memory use.
 
   Additionally, if I do get past digging through 0MQ's mechanics, I
 was thinking it would be best to spin off the PUB side to it's only
 process.  Which leads me to wonder if 0MQ inproc PUB/SUB actually
 relies on some clever memory mapping.  eg Push a message on an inproc
 PUB socket which goes to a shared/mutex locked list and client's just
 read from this one list.
 
   Apologies if some of these questions seem naive, I haven't gotten
 the chance to read 0mq's C source code yet.
 
 Thanks,
   Dave
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev 

Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.

[Charles Remes]

On Jan 26, 2013, at 8:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote:

 Regarding 0mq why can't developers just use standard messaging solutions 
 like rabbit or apache mq
 
 what is so cool about 0mq?
 
 and yes i am being fully serious….?

You joined a mailing list to ask this question? No, you aren't serious.

cr


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] zmq fundamental clarification

[Charles Remes]

1. No.

2. I don't know.

3. Take a look at zmq_socket_monitor. That function may provide what you need.

cr

On Jan 25, 2013, at 3:10 AM, Joachim Worringen joachim.worrin...@iathh.de 
wrote:

 Greetings,
 
 we are considering to use zmq for our very latency-sensitive distributed 
 application. zmq offers very compelling features. I searched the web and 
 skimmed through the excellent zmq book, but still have some fundamental 
 issues:
 
 1. Is there a way to actually send messages in the context of the 
 application thread? Something like direct send (or call it synchronously).
 
 Background: The additonal hop between the application thread and zmq i/o 
 thread add latency and, most of all, jitter esp. on loaded machines. 
 Sending the data directly will improve this, esp. with solutions as 
 Solarflare/OpenOnload or similar.
 
 The benchmark data in the Wiki with pingpong/2 of around 30us supports 
 this, as we certainly target latencies 10us for this kind of test (and 
 we do achieve them easily with direct socket access).
 
 2. Does zmq support source-specific multicast?
 
 Background: we do not want to set up and maintain global VLANs between 
 our data centers. We are forced into doing this for certain reasons now, 
 but it's a PITA and we want to get rid of this.
 
 3. The transparent connection management, and buffering of messages that 
 can currently not be sent, is something we need to avoid in certain 
 situations. Instead, we'd like to get immediate notification on a failed 
 connectione (remote peer went disconnected). Is this possible?
 
 thanks, Joachim
 
 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] What protocol does 0mq heartbeat use?

[Charles Remes]

On Jan 24, 2013, at 11:45 AM, James Marcus marcus.ja...@gmail.com wrote:

 Hi,
 I'm working through some issues for our engineers and its not clear to me if 
 the heartbeat is using multicast or ICMP (ping) for the heartbeat?
 
 Our two servers can't communicate but I see ports open. I'm also able to use 
 omping for multicast testing, as well as ICMP ping.
 
 Any additional thoughts on testing connectivity between two machines would be 
 great.

To test connectivity between machines with 0mq, I recommend using the local_lat 
and remote_lat programs that come with the library.

As for what protocol 0mq is using for heartbeat, I'm not sure I understand the 
question. 0mq doesn't have any explicit heart beat unless you create one at the 
application level (for example, with the majordomo pattern). If you are talking 
about heartbeating at the TCP level, then it's not a 0mq issue but a TCP stack 
issue.

cr


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Zeromq messages getting dropped

[Charles Remes]

On Jan 16, 2013, at 3:50 PM, Ritesh Adval riteshad...@gaikai.com wrote:
 I found that people have reported this error and one suggestion is to put 
 sleep after sending message. If I put sleep after sending message using a 
 socket it does work but not always, specially under large message size and 
 heavy load.
 
 
 
 Anyone has idea on what could be the issue?

Your original message said that LINGER is set to a very large value. I am 
assuming that when you send your messages from each of the 100 threads, when 
they are done then you close the socket. The LINGER value controls how long a 
packet may sit in the buffer before the socket is forced closed and the packet 
gets dropped.

If a sleep fixes the problem, then clearly some socket is being closed 
prematurely and its LINGER times out thereby dropping packets. I would double 
check that all DEALER sockets and your ROUTER socket have a proper LINGER 
setting. 

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Zeromq messages getting dropped

[Charles Remes]

On Jan 16, 2013, at 4:08 PM, Ritesh Adval riteshad...@gaikai.com wrote:

 Hi Charles,
 
 Yes I close the socket in my thread after sending 100 messages, and I expect 
 that LINGER will make sure messages are sent to the other end, I expected 
 that context termination will block and make sure any pending messages are 
 sent, but thats not happening. context termination returns quickly.
 
 Just now tried again in my unit test by setting LINGER to Integer.MAX_VALUE 
 explicitly in all my sockets and ran the test again and it did fail with 
 messages getting dropped.
 
 The interesting thing is only the 100th message  (The last one) from some of 
 my concurrent threads are getting dropped.

Time to show someone the code. That's the easiest way to figure it out. If you 
can reproduce this in C, that will get a lot more attention.

Here's how to open an issue:

http://www.zeromq.org/docs:issue-tracking

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] xpub and xsub example

[Charles Remes]

On Jan 14, 2013, at 4:10 PM, Mohit Jaggi mohitja...@gmail.com wrote:

 Hi All,
 I am looking for examples of xpub and xsub. My use case is that multiple 
 threads in a process produce data that needs to be published over a TCP 
 socket. I was thinking of letting them call send but it seems that that is 
 not thread-safe. From the guide, it appears I can use inproc+xpub socket from 
 producers to a proxy and let subscribers connect over TCP/xsub. However, 
 there is no example code. 

I answered your question on irc. When asking a question, you should stick 
around for more than 5m to get an answer.

Create a forwarder device that all of your PUB sockets connect to. All SUB 
sockets should then subscribe to that forwarder. You do *not* need to mess with 
XPUB and XSUB sockets.

See here:

http://api.zeromq.org/3-2:zmq-proxy

and

http://zguide.zeromq.org/page:all#Intermediaries-and-Proxies

cr

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev