The only way I sen see results in the worker is my using this design
but with random errors
0) Here is how I start uwsgi using the gevent loop. Using nginx and supervisord
/usr/local/bin/uwsgi --loop gevent --socket 127.0.0.1:8070 --processes
2 --pp /home/ubuntu/workspace/Server/ --wsgi-file
/home/ubuntu/workspace/Server.py -b 32768 --master --async 10
--enable-threads --listen 2048 --memory-report --logto2
/tmp/uwsgi_log.log
1) when I start uwsgi I get this error message in the logfile:
*** running gevent loop engine [addr:0x4515e0] ***
Traceback (most recent call last):
File
"/home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiPixelServer/uwsgiPixelServer.py",
line 72, in init_mq_ctx
socket.bind("tcp://127.0.0.1:5018")
File "socket.pyx", line 465, in zmq.core.socket.Socket.bind
(zmq/core/socket.c:4450)
zmq.core.error.ZMQError: Address already in use\
The worker is already listening on the port.
2) In the main loop (using bottle btw)
import gevent
import gevent.socket
from bottle import route, run, default_app, request, response, get,redirect
import zmq.green as zmq
import uwsgi
def init_mq_ctx():
global context
global socket
context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.bind("tcp://127.0.0.1:5018")
def mq_send(push):
socket.send(push)
uwsgi.post_fork_hook = init_mq_ctx
@get('/pixel/')
def pixel():
mq_send('test')
yield 'Hello World'
3) When I refresh the page, I get random errors and nothing is in the
log and no output is sent to the workers. If I randomly refresh, I
get a valid page, with results sent to the worker.
Error 500: Internal Server Error
Sorry, the requested URL 'http://127.0.0.1/pixel/' caused an error:
Unhandled exception
If I remove mq_send('test') from the loop, I get no errors.
On Wed, Jul 4, 2012 at 12:51 PM, David Montgomery
<[email protected]> wrote:
> When I send a message, using the gevent loop, how do I send?
>
> import uwsgi
> def init_mq_ctx():
> global context
> global socket
> context = zmq.Context()
> socket = context.socket(zmq.PUSH)
> socket.bind("tcp://127.0.0.1:5018")
>
> def mq_send(push):
> socket.send(push)
>
> try:
> uwsgi.post_fork_hook = init_mq_ctx
> except:
> pass
>
>
> 1) Do I use:
> import zmq
> OR
> import zmq.green as zmq
>
> 2) when I call mq_send('hello') do I call:
> mq_send(push)
>
> OR
>
> gevent.spawn(mq_send,'hello')
>
> So...how do I make sending messages async without blocking the main loop?
>
>
> I am utterly confused.....
>
> Thanks
>
>
>
>
> On Mon, Jul 2, 2012 at 2:04 PM, Evgeny Turnaev <[email protected]> wrote:
>> Hello David,
>> This (assertition error) is a feature-bug of zmq. Zmq context cant
>> survive fork() call because zmq uses threads
>> so when you call zmq.Context() a C threads is created behind the
>> scenes. And when uwsgi forks worker - threads dont forked. So the
>> correct way to use zmq with uwsgi - is to create context after fork()
>> something like:
>>
>> =====
>> def init_mq_ctx():
>> global ctx
>> ctx = zmq.Context()
>>
>> def mq_send(msg):
>> global ctx
>>
>> uwsgi.post_fork_hook = init_mq_ctx
>> =====
>>
>> This feature-bug is badly documented in zmq and zmq mail list. And doc
>> says that one should recreate context after fork but i did not managed
>> to make pyzmq to work with
>> ctx = zmq.Context()
>> fork()
>> ctx = zmq.ContexT()
>> scenario (which makes pyzmq useless for logging for me.)
>>
>> As for gevent - dont know. I am using pyzmq + pyev + hiredis parser
>> and custom client.
>>
>> Personally i think reporting such an error - is definatly a bug which
>> should be reported. (Also i think a bug is - eating all memory if no
>> HWT is set on socket)
>>
>> 2012/7/2 David Montgomery <[email protected]>:
>>> Hi,
>>>
>>> When I am running in uWSGI mode and I use 0mq I get this error.
>>>
>>> (HTTP/1.1 200) 1 headers in 44 bytes (2 switches on core 9)
>>> 2012-07-02 04:05:15
>>> Traceback (most recent call last):
>>> File
>>> "/usr/local/lib/python2.7/dist-packages/gevent-1.0b2-py2.7-linux-x86_64.egg/gevent/greenlet.py",
>>> line 328, in run
>>> result = self._run(*self.args, **self.kwargs)
>>> File
>>> "/home/ubuntu/workspace/rtbopsConfig/rtbServers/rtbUwsgiPixelServer/uwsgiPixelServer.py",
>>> line 43, in sendthis
>>> socket.send(push)
>>> File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py",
>>> line 173, in send
>>> self._wait_write()
>>> File "/usr/local/lib/python2.7/dist-packages/zmq/green/core.py",
>>> line 108, in _wait_write
>>> assert self.__writable.ready(), "Only one greenlet can be waiting
>>> on this event"
>>> AssertionError: Only one greenlet can be waiting on this event
>>> <Greenlet at 0x2d41370: sendthis('2012-07-02 04:05:15')> failed with
>>> AssertionError
>>>
>>> So...am I out of luck using 0mq with uWSGI? can it be resolved?
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>> On Mon, Jul 2, 2012 at 9:52 AM, David Montgomery
>>> <[email protected]> wrote:
>>>> Hi,
>>>>
>>>> I am using the gevent loop for uWSGI with bottle. I am currently
>>>> using redis-py to async writes to redis using the below.
>>>>
>>>>
>>>> def RedisWrite(push):
>>>> r.lpush('global',push) #most recent time user has seen add
>>>>
>>>> @get('/')
>>>> def main():
>>>> yield 'Hello World'
>>>> gevent.spawn(RedisWrite,'Hello World')
>>>>
>>>>
>>>> Instead, I want to send data to workers via zeromq that will then
>>>> process the data to write to redis. My main motivation for using
>>>> gevent loop is for the redis reads that I have to do using redis-py.
>>>>
>>>> My question is, are there any issues with getting 0mq to work with the
>>>> gevent loop? Rather then writing to a redis list, send messages to
>>>> workers via 0mq.
>>>>
>>>> I am using the latest versions of everything thing, even 0mq 3.x RC.
>>>>
>>>> Thnaks
>>> _______________________________________________
>>> uWSGI mailing list
>>> [email protected]
>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>>
>>
>>
>> --
>> --------------------------------------------
>> Турнаев Евгений Викторович
>> +7 906 875 09 43
>> --------------------------------------------
>> _______________________________________________
>> uWSGI mailing list
>> [email protected]
>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
