Well I believe we have conquered the 503 Reverse Proxy issues finally and this is my information on the causes and how to resolve this issue and perhaps other similar issues going forward. I'll submit this information to bug tracking and recommend we provide this information in the docs and/or cookbooks.
The dreaded 503 error as well as other unexplainable 5xx errors that occur after some run time are likely all linked directly to ulimit settings - specifically open file descriptors. This also applies to open files/file descriptors caused by use of I/O cache even if you are not using reverse proxy. This resolution wouldn't have been possible without Alvaro's long term assistance as well as Stefan and other folks on the list here; you folks are the best. This is our successful resolution after 48 hours of run time on a single server under a respectable amount of traffic. In Cherokee-Admin: Advanced > Resources File descriptors: by default I believe our install had an empty value - unsure if this value is required or optional. We had no value therein which essentially just left ulimit -n open file value at whatever default value the OS had for the given user Cherokee is run under. Everyone should set ulimit in Cherokee-admin ALWAYS. The environment variables from the OS are subject to change, might not be what you expect, etc. This is the only way to hard set the ulimit for Cherokee and have a predictable set point. File descriptor value needs to go as high as reasonable versus resources of the machine. If the server is dedicated and only you are on the machine, set it to something very high like 90000. This logic is especially true if you have a busy website. In Cherokee-Admin: Advanced > Nonces clean up time Nonces clean up time I believe was null in our installation. Unsure if this is a required value or not and what happens if no value is specified. In our experience leaving this value empty means open file descriptors are kept open artificially long - or perhaps infinitely, which creates a large number of idle but open descriptors on the server. Worst case is when under traffic load the number of open files become excessive until all descriptors are used and thus a 503 error is served. Nonces is another value that should be set in EVERY Cherokee installation. Cleanups need to happen in our experience otherwise eventually resource exhaustion occurs and 503s only get served. Nonces clean up time we have set intentionally low at value of 5. This means cleanup is done every 5 seconds. This has intended consequence of closing open file descriptors, which prevents them from piling up in unused heap. We have tested longer value of 10 seconds which works also just fine, just more open file descriptors being used. Clarification is needed on what intended task of this cleanup is and what meets qualification to be cleaned up each time it is ran. In Cherokee-Admin: Advanced > I/O Cache I/O Cache is another open file descriptor user. The more cacheable items you have the more open files there will be. When under traffic load this will add to consumed open file descriptors. Our solution with the 503 errors was to turn I/O cache off, which reduced 503 error frequency, but did not prevent such. Based on prior changes we made we reenabled I/O cache with no adverse effect. I want to note clearly that these items indicate no deficiency in Cherokee whatsoever. The same logic has been applied to fix problems someone was experiencing with Cherokee on SunOS as well as issues someone else was having with Varnish. If you Google any web server ulimit: apache ulimit nginx ulimit etc... You will find many discussions about the same issues with each and every pieces of web server software. Finally, as a disclaimer, setting ulimit for open descriptors high like I said above might have the side effect of consuming more RAM. I have yet to see anything scary from these settings, but we run Cherokee on a machine with 2GB of RAM that is mostly there for Cherokee to use. Your mileage might vary, so test always and feel free to ask me anything or recommend updates to any of this. -Paul _______________________________________________ Cherokee mailing list [email protected] http://lists.octality.com/listinfo/cherokee
