I see, thanks James. I had no idea config.hpp existed. Why would the queue grow if I have conflate set? Doesn’t that mean it should only queue one value?
I’m doing this because there is some data that I would like client programs to always have access to, but only the most recent one. So my solution is to use CONFLATE and always send it out. If you have a better solution to that, I’d be glad to hear it. But until then, maybe I should change that value to 1. What other penalties/trade offs are there for changing it to 1? > On Jun 21, 2018, at 4:39 AM, James Harvey <jhar...@factset.com> wrote: > > Hi Kevin, > > I ran your test code and saw the same issue. I also created the tests with > straight libzmq (not czmq) and the issue still occurs. > > The memory is not lost (valgrind confirms) but it does grow. > > Eventually I noticed its proportional to the number of times you call > zmq_recv(). > > Adding code to force heap cleanup and print the stats shows that every 100 > times zmq_recv is called, the queue is purged. > > malloc_trim(0); > malloc_stats(); > > Finally I found inbound_poll_rate which is used by socket_base to determine > how often to check for signals and commands. > https://github.com/zeromq/libzmq/blob/b77d7610cd91dbe25096b804c77b27065b7dd1fd/src/config.hpp#L53 > > Reducing this 1 (as a test) fixed the issue and memory never rose above 4Mb. > > Is this a real issue? I am not sure. > I guess for some edge cases where you receive a truck load of data and don’t > process it. But on the other hand why would you do that. > > Possible fixes I can think of would be a socket_option to modify > inbound_poll_rate > OR > Force process_commands() on a timer if no activity is occurring. > > Thanks > > James > > > > > > -----Original Message----- > From: zeromq-dev <zeromq-dev-boun...@lists.zeromq.org> On Behalf Of Kevin Wang > Sent: 20 June 2018 22:12 > To: zeromq-dev@lists.zeromq.org > Subject: Re: [zeromq-dev] Memory leak on SUB set with CONFLATE? > > The memory of the SUB process goes up. > > If I remove the getline() call and the SUB process keeps up, the RAM stays > still for both. Those programs compile with gcc pretty easily, if you feel > like trying it out. Maybe my Ubuntu is just messed up. > > I assume since its CONFLATE is on, maybe the PUB doesn't queue it, but SUB > still does? In that case, I also would assume that replacing the old message > with the new one would free the memory held by the old message. > > >> On 06/20/2018 01:38 PM, James Harvey wrote: >> When you say “the system memory skyrockets” are you talking about the memory >> used by the SUB process or PUB process? >> >> I would expect the memory of the PUB process to increase as it will be >> queueing messages. >> >>> On 20 Jun 2018, at 21:02, Kevin Wang <ke...@spikegadgets.com> wrote: >>> >>> the system memory skyrockets, >> _______________________________________________ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
