2012/7/4 David Montgomery <[email protected]>: > When I send a message, using the gevent loop, how do I send? > > import uwsgi > def init_mq_ctx(): > global context > global socket > context = zmq.Context() > socket = context.socket(zmq.PUSH) > socket.bind("tcp://127.0.0.1:5018") > > def mq_send(push): > socket.send(push) > > try: > uwsgi.post_fork_hook = init_mq_ctx > except: > pass > > > 1) Do I use: > import zmq > OR > import zmq.green as zmq > > 2) when I call mq_send('hello') do I call: > mq_send(push) > > OR > > gevent.spawn(mq_send,'hello') > > So...how do I make sending messages async without blocking the main loop? > > > I am utterly confused.....
Uh.. dont know about gevent i am using zmq + libev on receiving side (msg consumer). Looking at Your code - obviously you cant bind to the same port from multiple workers. "Address already in use" error is referring to this. But you can bind to constant_port + uwsgi worker numbler (there is also bind_to_random_port), or setup zmq forward device and connect to it from workers. I am using exactly the same: workers get connected to local zmq forwarder device listerning on unix socket (zmq ipc) and forwarder forwards messages to other listening sockets. That would be cool if zmq has an option to take socket file descriptor and use it - so that i can bind() in master and feed fd to zmq in workers, but i doubt it hash such an option. As for the blocking call - please refer to the http://api.zeromq.org/2-2:zmq-socket There is an option in pyzmq.socket.send() flags = NOBLOCK http://zeromq.github.com/pyzmq/api/generated/zmq.core.socket.html but i prefer using PUB/SUB sockets and set HWM to low numbers in wsgi processes. send() call on PUB/SUB sockets always not blocking - it is just putting message in thread send queue. > > Thanks > > > > > On Mon, Jul 2, 2012 at 2:04 PM, Evgeny Turnaev <[email protected]> wrote: >> Hello David, >> This (assertition error) is a feature-bug of zmq. Zmq context cant >> survive fork() call because zmq uses threads >> so when you call zmq.Context() a C threads is created behind the >> scenes. And when uwsgi forks worker - threads dont forked. So the >> correct way to use zmq with uwsgi - is to create context after fork() >> something like: >> >> ===== >> def init_mq_ctx(): >> global ctx >> ctx = zmq.Context() >> >> def mq_send(msg): >> global ctx >> >> uwsgi.post_fork_hook = init_mq_ctx >> ===== >> >> This feature-bug is badly documented in zmq and zmq mail list. And doc >> says that one should recreate context after fork but i did not managed >> to make pyzmq to work with >> ctx = zmq.Context() >> fork() >> ctx = zmq.ContexT() >> scenario (which makes pyzmq useless for logging for me.) >> >> As for gevent - dont know. I am using pyzmq + pyev + hiredis parser >> and custom client. >> >> Personally i think reporting such an error - is definatly a bug which >> should be reported. (Also i think a bug is - eating all memory if no >> HWT is set on socket) >> >> 2012/7/2 David Montgomery <[email protected]>: >>> Hi, >>> >>> When I am running in uWSGI mode and I use 0mq I get this error. >>> >>> (HTTP/1.1 200) 1 headers in 44 bytes (2 switches on core 9) >>> 2012-07-02 04:05:15 >>> Traceback (most recent call last): >>> File >>> "/usr/local/lib/python2.7/dist-packages/gevent-1.0b2-py2.7-linux-x86_64.egg/gevent/greenlet.py", >>> line 328, in run >>> result = self._run(*self.args, **self.kwargs) >>> File >>> "/home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiPixelServer/uwsgiPixelServer.py", >>> line 43, in sendthis >>> socket.send(push) >>> File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py", >>> line 173, in send >>> self._wait_write() >>> File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py", >>> line 108, in _wait_write >>> assert self.__writable.ready(), "Only one greenlet can be waiting >>> on this event" >>> AssertionError: Only one greenlet can be waiting on this event >>> <Greenlet at 0x2d41370: sendthis('2012-07-02 04:05:15')> failed with >>> AssertionError >>> >>> So...am I out of luck using 0mq with uWSGI? can it be resolved? >>> >>> Thanks >>> >>> >>> >>> >>> On Mon, Jul 2, 2012 at 9:52 AM, David Montgomery >>> <[email protected]> wrote: >>>> Hi, >>>> >>>> I am using the gevent loop for uWSGI with bottle. I am currently >>>> using redis-py to async writes to redis using the below. >>>> >>>> >>>> def RedisWrite(push): >>>> r.lpush('global',push) #most recent time user has seen add >>>> >>>> @get('/') >>>> def main(): >>>> yield 'Hello World' >>>> gevent.spawn(RedisWrite,'Hello World') >>>> >>>> >>>> Instead, I want to send data to workers via zeromq that will then >>>> process the data to write to redis. My main motivation for using >>>> gevent loop is for the redis reads that I have to do using redis-py. >>>> >>>> My question is, are there any issues with getting 0mq to work with the >>>> gevent loop? Rather then writing to a redis list, send messages to >>>> workers via 0mq. >>>> >>>> I am using the latest versions of everything thing, even 0mq 3.x RC. >>>> >>>> Thnaks >>> _______________________________________________ >>> uWSGI mailing list >>> [email protected] >>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> >> >> >> -- >> -------------------------------------------- >> Турнаев Евгений Викторович >> +7 906 875 09 43 >> -------------------------------------------- >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi -- -------------------------------------------- Турнаев Евгений Викторович +7 906 875 09 43 -------------------------------------------- _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
