Ohad 1) I decreased it to 2 GB by default, but I don't want to go further down in case of many plugins installed.
Ohad 2) I tested it and it works just fine without scl, the non-scl passenger client can communicate with scl passenger as we install both passengers (one for puppet, one for foreman). I wrapped it with SCL tho, no problem. Ah wait, SCL shebang is not supported, I can create foreman-ruby wrapper for RHEL7 in the PR, it's Fedora only: https://bugzilla.redhat.com/show_bug.cgi?id=1058796 Tomer 1) It is possible to run this every minute, the script is simple and fast, but my goal is not to replace process frameworks like GodRB, I want something that notifies administrator something is wrong. I could also send a foreman UI notification if we have an API for that. If processes grows in minutes, they will be likely above threashold after an hour as well. And since it is very easy to signal termination and Passenger handles that very well (terminates the process and immediately starts respawning new one if there are not enough workers), why not to do it. Tomer 2) Well, in context of processes consuming 2 GB RSS memory which is essentially 24 GB RSS *private* memory (we default 12 passenger processes), user not seing response is not that relevant I think. If the server is powerful enough to terminate process gracefully (we have 2 minutes grace period), it will finish processing eventually. But if this is your concern, I added a configuration option to disable this feature, we can ship with disabled killing by default if you want to. Tomer 3) I added output of passenger-status which gives nice overview including stacktraces of all workers before termination is performed, but if I compare current state (we don't care) with what I propose (administrator gets emailed and excessive process gets terminated nicely) it's like night and day. We know more than we did before. Check out the new version I just pushed into the gist! I hope you will like it. On Wed, May 10, 2017 at 9:55 AM, Tomer Brisker <tbris...@redhat.com> wrote: > While this is an interesting idea, I have a few concerns about it: > 1. This won't take care of requests that balloon memory very rapidly (such > as the issue we recently found in katello which would cause OOM in a matter > of minutes). Waiting up to an hour to reclaim memory might not be fast > enough in those cases. > 2. If passenger fails to complete a response within the graceful shutdown > period, users will just have a request failed and no indication of what > happened. While most requests should finish within 2 minutes, for example a > large csv download can take much longer then that, and user will not know > why it failed (and it could be completely unrelated, leading to red herrings > - the request that failed won't necessarily be the one that caused the > leak). > 3. We might not know about places where we leak memory, since the process > will get killed without anyone noticing it. > > On Wed, May 10, 2017 at 10:43 AM, Ohad Levy <ohadl...@gmail.com> wrote: >> >> >> >> On Wed, May 10, 2017 at 10:28 AM, Lukas Zapletal <l...@redhat.com> wrote: >>> >>> Hello, >>> >>> I wrote a script that gracefully terminates Passenger processes >>> consuming more than 2.5 GB of private RSS memory. Typical consumption >>> of Foreman with Katello and other basic plugins is around 1.5 GB and >>> since only Passenger Enterprise allows to limit maximum amount of RSS >>> memory for processes, this simple script does exactly that: >>> >>> https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 >>> >>> Put it into your /etc/cron.hourly/ and make it executable, make sure >>> you read root emails or forward it properly, the script reports all >>> terminations performed on STDOUT. I would like you to test this script >>> in production and get back to me with feedback about what you think. >>> >>> Motivation is simple - we often introduce bugs in our Rails codebase >>> which performs some eager loading or there are memory leaks in our >>> code or dependencies and Passenger processes can grow up to dozens >>> gigabytes. Unfortunately, there is no other way of getting out other >>> than restarting httpd with passenger. This script could help to avoid >>> situations when production instance starts to swap hard thank to some >>> small regression we introduced. Also administrator will be noticed >>> early via email when this happens so we will keep track of these >>> regressions in production. >>> >>> http://projects.theforeman.org/issues/19496 >>> >>> Feedback appreciated. >>> >> >> I gave it a spin, two comments: >> >> 1. my default memory size was around 400m i had to change the script it in >> order to see it in action, maybe we should take into account avail memory >> and how many passenger processes are allowed - the default of 2.5gb seems a >> bit too high? >> 2. in order to use your script in an scl env, you would need to wrap it >> with scl enable <collection> passenger-recycler >> >> Thanks! >> Ohad >> >>> >>> -- >>> Later, >>> Lukas @lzap Zapletal >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Have a nice day, > Tomer Brisker > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
