This is the statement that concerns me in your requirement: "Expecting ZeroMQ to implement some sort of reliable message shipping. ie. I want messages to be maintaned on a queue if the connection is no available or the remote machine crashed. Logstash is in a remote machine."
Two zeromq sockets connected to each other in and of themselves are not what I would consider a reliable queuing system. Zeromq sockets do queue messages in memory, and how many messages they will queue in memory is somewhat configurable - but even with high water mark settings, a PUSH / PULL connection in and of itself is not going to give you reliable message delivery (and PUB/SUB by definition will not). The "queue" in zeromq sockets should be thought of more as a buffer than a "message queue". If what you really need is reliable message shipping, such a thing certainly can be implemented with zeromq - but it's more complex than simply connecting two sockets together and sending messages over them. The ZeroMQ guide itself is a great place to get ideas for such an architecture: http://zguide.zeromq.org/page:all To the zeromq core developers on the list - if I'm misrepresenting anything here please feel free to chime in - but this is how I view zeromq from my own experience with it. Brian On Thu, Jan 10, 2013 at 9:44 AM, Simone <[email protected]> wrote: > Brian Knox <briank <at> talksum.com> writes: > > > > > "Moreover, I tried changing the zmq_connect to connect to a random IP > > address that does not exist and I still get a successfull connection but > > obviously nothing was connected..." > > > > This is normal / expected behavior with ZeroMQ. > > > > As for your logstash issue, I haven't worked with logstash before so I > > can't hazard a guess. However, using a ZMQ_PAIR socket to connect to > > another process is a bit strange and I'm uncertain why you would do > > that. It doesn't make sense to me that logstash would listen on a > > ZMQ_PAIR as ZMQ_PAIR is mainly for use with inter-thread message passing > > using the inproc transport ( see the documentation at > > http://api.zeromq.org/3-2:zmq-socket ). > > > > On 1/10/2013 5:18 AM, Simone wrote: > > > Hi Everyone > > > > > > I am using logstash to receive logs from my application. The > communication > is > > > done using ZeroMQ. The application sends to the logstash server using > ZMQ_PAIR. > > > > > > The connection and context creation on the application side return no > error, > > > however by invoking either zmq_send or zmq_recv I get -1 and errno set > to 0 > and > > > zmq_errno() set to 128. > > > > > > Unknown errors..... > > > > > > Moreover, I tried changing the zmq_connect to connect to a random IP > address > > > that > > > does not exist and I still get a successfull connection but obviously > nothing > > > was > > > connected... > > > > > > I am working on Windows x64 using ZeroMQ 2.2 on the client side (since > this > is > > > the version supported by Logstash). Logstash does not seem to be the > problem > > > since I am quite sure the connection is not even established. > > > > > > I installed ZeroMQ from > > > http://www.zeromq.org/distro:microsoft-windowsusing > 2.2 > > > for Windows 7 and using the library libzmq-v100-mt.dll for running, > precompiled, > > > contained in the bin directory of the installation dir. > > > > > > Any idea? > > > > > > thanks a lot > > > > > > Simone Zandara > > > Network Pogrammer - Massive Ubisoft Entertainment > > > > > > _______________________________________________ > > > zeromq-dev mailing list > > > zeromq-dev <at> lists.zeromq.org > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > I see, well I dont know why but Logstash only listen on TCP connections > over > ZeroMQ. > > Moreover I tried following a simple example like this > > http://learning-0mq-with- > pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pair.html > > Not implementing the server myself but using logstash as server. > It creates a very dummy connection over TCP. > > Hwever the link you gave me, explicitly says that PAIR is indicated for > interproc, so I cannot really expect it to work I guess. > > By the way what I am trying to do is > > > ------------ ---------- ------------- > | APPLICATION | ---ZeroMQ----> | LOGSTASH | -> |ElasticSearch| > ------------ ---------- ------------- > > Expecting ZeroMQ to implement some sort of reliable message shipping. ie. > I want > messages to be maintaned on a queue if the connection is no available or > the > remote machine crashed. Logstash is in a remote machine. > > Logstash only supports three communication types > > PAIR > PUSH/PULL > PUB/SUB > > Shall I try another type or shall I just give up this way? > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
