Hi Roberto! Thanks for picking up this thread. I was afraid that's what I'd hear - we're running on Ubuntu LTS 12.04.1, which is why we are using such an old version. I'll try to upgrade and re-test.
-Clemens On Tue, Apr 1, 2014 at 3:17 PM, Roberto De Ioris <[email protected]> wrote: > > > Hi guys, > > > > I've been struggling with a weird situation (I have to assume a bug) and > > cannot find any references to this on the Internet. If you could chime in > > with some advice, that'd be wonderful :) > > > > Please note that this whole situation is related to python-flask, and I > > have contacted their mailing list to see if anyone has seen this behavior > > before. They didn't and suggested to see if you guys know more :-/ > > > > The problem is that I'm not sure at what point uwsgi-plugin-python starts > > spawning new threads for handling requests. Maybe it's too early and > > python-flask does some optimizations that result in a bug. > > > > Here is the situation: We host a web-service through nginx, serving > > requests to uwsgi-plugin-python (version 1.0.3+dfsg-1ubuntu0.1) to our > > nice > > Python (2.7.3) flask app. Since we had garbage-collection issues with > many > > small objects on the heap in the past, we decided to respawn the uwsgi > > service every few k requests and use multiple processes to avoid a > > down-time in the respawn phase (and every process has multiple threads of > > course). > > > > Another important note is that we use the "lazy" option, because we use > > MySQL connections which cannot be forked after Python initialization > > (because all DB connections would share the same connection ID, which is > > not supported by the MySQL client). > > > > Now, it seems like each time the Python process respawns, that flask sets > > up our blueprints (mappings between Python handlers and URLs in case > > you're > > not familiar with flask) and then starts accepting requests, but > > *sometimes* responds with a 404 to clients early during this phase. This > > happens very rarely, but since there is considerable load on the server, > > it > > actually starts adding up so that we need to something about it. > > > > I have spent quite some time debugging the flask blueprints, and my > > findings - so far - show that at app creation time, only one thread is > > running in Python *after* the blueprints have been registered. However, > > when I add code to log details about the blueprints / URL-rules when the > > 404 happens, then the blueprints are not all/fully loaded yet. > > > > It seems to me like new threads are created and start accepting > > connections > > before all blueprints/rules have been fully registered. Here is some code > > to show what I mean: > > > > def create_app(...): > > app = flask.Flask(__name__) > > [...] > > > > @app.after_request > > def after_request(response): > > if response.status_code == 404: > > # DBG2 > > print 'dbg', response.status_code, request.base_url > > print threading.active_count(), "active threads" > > print app.url_map.__dict__.get('foo/mytesturl') > > return response > > > > app.register_blueprint(mymodel, url_prefix="/foo") > > # DBG1 > > print 'done loading', app.url_map.__dict__ > > print threading.active_count(), "active threads" > > return app > > > > While the DBG1 code shows everything correctly, the DBG2 code shows a > > sub-set of my URL-handlers (which is why the 404 gets executed). > > > > BTW, at DBG1 point, the logs usually show "1 active threads", while at > > DBG2 > > many more threads are loaded, making me assume there is some sort of > > lazy/on-demand loading of the rules within flask. > > > > Again, the code works, it's just that there is this race where I get > 404s. > > > > Is this a known bug in uwsgi / uwsgi-plugin-python? If so, it'd be worth > > investigating on a newer version, or is there a specific way to tell > flask > > to fully load all blueprints/rules before accepting connections? > > > > Sorry for the overly long email, I wanted to get all the details right :) > > > > Thanks! > > -Clemens > > > > -- > > Clemens Kolbitsch > > Security Researcher > > [email protected] > > Mobile +1 (206) 356-7745 > > Land +1 (805) 456-7076 > > > > Lastline, Inc. > > 6950 Hollister Avenue, Suite 101 > > Goleta, CA 93117 > > > > > > Hi Clemens, sorry but uWSGI 1.0 is unsupported since 2012 and know to be > broken (as well as 1.2). Only 1.4.x and 2.0.x are supported and actively > maintained. > > Upgrade to a stable version, and if you still have problems feel free to > repost. > > > -- > Roberto De Ioris > http://unbit.it > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > -- Clemens Kolbitsch Security Researcher [email protected] Mobile +1 (206) 356-7745 Land +1 (805) 456-7076 Lastline, Inc. 6950 Hollister Avenue, Suite 101 Goleta, CA 93117 www.lastline.com
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
