Re: [foreman-dev] Re: 1.15.0 release status
Thanks, it seems like either I don't understand the combination filter or it didn't work (https://github.com/theforeman/foreman-infra/blob/cb03c58b7b2bf3b5c61a125e4c32841a07a6a274/puppet/modules/jenkins_job_builder/files/theforeman.org/yaml/jobs/release_test.yaml#L33) combination-filter: 'os == "el7" || os == "f24" || os == "jessie" || (os == "stretch" && version != "1.14" && version != "1.15") || os == "trusty" || os == "xenial"' I removed it from the job just to run it once and it passes fine. http://ci.theforeman.org/job/release_push_deb/ doesn't have a combination filter so it just publishes to the other 3 repos.(jessie trusty and xenial). http://ci.theforeman.org/view/Release%20pipeline/ On Thu, May 11, 2017 at 2:02 AM, Michael Moll wrote: > Hi, > > On Thu, May 11, 2017 at 01:57:26AM +0900, Daniel Lobato wrote: >> Currently the pipeline is stuck on tests for Debian Stretch (not sure >> if just an artifact or a real failure yet, investigating). > > Stretch is not supported with 1.15 (and known to fail). So that should > get overriden for 1.15 (or whatever the approach is with these jobs in > similar situations). > > Regards > -- > Michael Moll > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Daniel Lobato @dlobatog daniellobato.me GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Re: [Event] Foreman Community Demo Items - Thu 11 May 3pm [GMT]
Friendly reminder that the demo is *tomorrow* and I still see very little content on the agenda. Given we did an AMA last time, it's been 6 weeks since the last demo - is there really nothing to show since then? Or is it a case of people not having enough time to spare? Greg -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Re: 1.15.0 release status
Hi, On Thu, May 11, 2017 at 01:57:26AM +0900, Daniel Lobato wrote: > Currently the pipeline is stuck on tests for Debian Stretch (not sure > if just an artifact or a real failure yet, investigating). Stretch is not supported with 1.15 (and known to fail). So that should get overriden for 1.15 (or whatever the approach is with these jobs in similar situations). Regards -- Michael Moll -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Re: 1.15.0 release status
Update, RPMS are signed and signatures are uploaded. Currently the pipeline is stuck on tests for Debian Stretch (not sure if just an artifact or a real failure yet, investigating). Help investigating the issue would be appreciated :) After tests pass on Debian stretch both RPMs and debs will be pushed to their respective repos as you can see in the pipeline. http://ci.theforeman.org/view/Release%20pipeline/ On Tue, May 9, 2017 at 8:04 PM, Daniel Lobato wrote: > Hi Foreman devs, > > Our 1.15 release is almost there! Tagging is done. I kicked off the release > pipeline, which means the first release tarballs are on their way. > > After that I will sign them and continue so we can build rpms & debs. As > soon as that's ready I will sign and publish them. > > I am boarding a plane now, hopefully when I land the tarballs will be ready > to be signed :) Sorry for the slight delay! > > In case something goes wrong let me know on this thread. > > Best regards, -- Daniel Lobato @dlobatog daniellobato.me GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Katello Scenario tests and annotations merged
Justin Sherrill writes: > Hi All, > > I have just merged https://github.com/Katello/katello/pull/6716 which > adds some end to end testing for a small set of scenarios (that will > likely be expanded over time). It also adds annotations to these > requests and the ability to generated documentation around them. The > end goal being to a) have better testing for full scenarios and b) be > able to look up all the interactions between katello and pulp and > candlepin easily. (https://github.com/theforeman/theforeman.org/pull/845 > is the first iteration of that) > > Most of the time you should notice no difference. However sometimes if > you add or change an api call to pulp or candlepin you may start seeing > test failures (especially as we add requests). Once you re-record the > annotations (see README below), you may see that there are missing > annotations which need to be added to your PR. > > Feel free to read > https://github.com/Katello/katello/blob/master/test/scenarios/annotations/README.md > > for more information and let me know if you have questions. Interesting, BTW. for those of you lazy to actually run the rake, but still interested into how the output looks like, have a look at :) https://theforeman.org/plugins/katello/nightly/annotations/index.html -- Ivan > > -Justin > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Script to recycle passenger processes
On 05/10/2017 05:05 AM, Lukas Zapletal wrote: Check out the new version I just pushed into the gist! I hope you will like it. What is the process to go from gist to in 1.16? -- bk -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Script to recycle passenger processes
Ohad 1) I decreased it to 2 GB by default, but I don't want to go further down in case of many plugins installed. Ohad 2) I tested it and it works just fine without scl, the non-scl passenger client can communicate with scl passenger as we install both passengers (one for puppet, one for foreman). I wrapped it with SCL tho, no problem. Ah wait, SCL shebang is not supported, I can create foreman-ruby wrapper for RHEL7 in the PR, it's Fedora only: https://bugzilla.redhat.com/show_bug.cgi?id=1058796 Tomer 1) It is possible to run this every minute, the script is simple and fast, but my goal is not to replace process frameworks like GodRB, I want something that notifies administrator something is wrong. I could also send a foreman UI notification if we have an API for that. If processes grows in minutes, they will be likely above threashold after an hour as well. And since it is very easy to signal termination and Passenger handles that very well (terminates the process and immediately starts respawning new one if there are not enough workers), why not to do it. Tomer 2) Well, in context of processes consuming 2 GB RSS memory which is essentially 24 GB RSS *private* memory (we default 12 passenger processes), user not seing response is not that relevant I think. If the server is powerful enough to terminate process gracefully (we have 2 minutes grace period), it will finish processing eventually. But if this is your concern, I added a configuration option to disable this feature, we can ship with disabled killing by default if you want to. Tomer 3) I added output of passenger-status which gives nice overview including stacktraces of all workers before termination is performed, but if I compare current state (we don't care) with what I propose (administrator gets emailed and excessive process gets terminated nicely) it's like night and day. We know more than we did before. Check out the new version I just pushed into the gist! I hope you will like it. On Wed, May 10, 2017 at 9:55 AM, Tomer Brisker wrote: > While this is an interesting idea, I have a few concerns about it: > 1. This won't take care of requests that balloon memory very rapidly (such > as the issue we recently found in katello which would cause OOM in a matter > of minutes). Waiting up to an hour to reclaim memory might not be fast > enough in those cases. > 2. If passenger fails to complete a response within the graceful shutdown > period, users will just have a request failed and no indication of what > happened. While most requests should finish within 2 minutes, for example a > large csv download can take much longer then that, and user will not know > why it failed (and it could be completely unrelated, leading to red herrings > - the request that failed won't necessarily be the one that caused the > leak). > 3. We might not know about places where we leak memory, since the process > will get killed without anyone noticing it. > > On Wed, May 10, 2017 at 10:43 AM, Ohad Levy wrote: >> >> >> >> On Wed, May 10, 2017 at 10:28 AM, Lukas Zapletal wrote: >>> >>> Hello, >>> >>> I wrote a script that gracefully terminates Passenger processes >>> consuming more than 2.5 GB of private RSS memory. Typical consumption >>> of Foreman with Katello and other basic plugins is around 1.5 GB and >>> since only Passenger Enterprise allows to limit maximum amount of RSS >>> memory for processes, this simple script does exactly that: >>> >>> https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 >>> >>> Put it into your /etc/cron.hourly/ and make it executable, make sure >>> you read root emails or forward it properly, the script reports all >>> terminations performed on STDOUT. I would like you to test this script >>> in production and get back to me with feedback about what you think. >>> >>> Motivation is simple - we often introduce bugs in our Rails codebase >>> which performs some eager loading or there are memory leaks in our >>> code or dependencies and Passenger processes can grow up to dozens >>> gigabytes. Unfortunately, there is no other way of getting out other >>> than restarting httpd with passenger. This script could help to avoid >>> situations when production instance starts to swap hard thank to some >>> small regression we introduced. Also administrator will be noticed >>> early via email when this happens so we will keep track of these >>> regressions in production. >>> >>> http://projects.theforeman.org/issues/19496 >>> >>> Feedback appreciated. >>> >> >> I gave it a spin, two comments: >> >> 1. my default memory size was around 400m i had to change the script it in >> order to see it in action, maybe we should take into account avail memory >> and how many passenger processes are allowed - the default of 2.5gb seems a >> bit too high? >> 2. in order to use your script in an scl env, you would need to wrap it >> with scl enable passenger-recycler >> >> Thanks! >> Ohad >> >>> >>> -- >>> Later, >>> Lukas @lzap Zapletal >>> >>>
Re: [foreman-dev] Script to recycle passenger processes
While this is an interesting idea, I have a few concerns about it: 1. This won't take care of requests that balloon memory very rapidly (such as the issue we recently found in katello which would cause OOM in a matter of minutes). Waiting up to an hour to reclaim memory might not be fast enough in those cases. 2. If passenger fails to complete a response within the graceful shutdown period, users will just have a request failed and no indication of what happened. While most requests should finish within 2 minutes, for example a large csv download can take much longer then that, and user will not know why it failed (and it could be completely unrelated, leading to red herrings - the request that failed won't necessarily be the one that caused the leak). 3. We might not know about places where we leak memory, since the process will get killed without anyone noticing it. On Wed, May 10, 2017 at 10:43 AM, Ohad Levy wrote: > > > On Wed, May 10, 2017 at 10:28 AM, Lukas Zapletal wrote: > >> Hello, >> >> I wrote a script that gracefully terminates Passenger processes >> consuming more than 2.5 GB of private RSS memory. Typical consumption >> of Foreman with Katello and other basic plugins is around 1.5 GB and >> since only Passenger Enterprise allows to limit maximum amount of RSS >> memory for processes, this simple script does exactly that: >> >> https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 >> >> Put it into your /etc/cron.hourly/ and make it executable, make sure >> you read root emails or forward it properly, the script reports all >> terminations performed on STDOUT. I would like you to test this script >> in production and get back to me with feedback about what you think. >> >> Motivation is simple - we often introduce bugs in our Rails codebase >> which performs some eager loading or there are memory leaks in our >> code or dependencies and Passenger processes can grow up to dozens >> gigabytes. Unfortunately, there is no other way of getting out other >> than restarting httpd with passenger. This script could help to avoid >> situations when production instance starts to swap hard thank to some >> small regression we introduced. Also administrator will be noticed >> early via email when this happens so we will keep track of these >> regressions in production. >> >> http://projects.theforeman.org/issues/19496 >> >> Feedback appreciated. >> >> > I gave it a spin, two comments: > > 1. my default memory size was around 400m i had to change the script it in > order to see it in action, maybe we should take into account avail memory > and how many passenger processes are allowed - the default of 2.5gb seems a > bit too high? > 2. in order to use your script in an scl env, you would need to wrap it > with scl enable passenger-recycler > > Thanks! > Ohad > > >> -- >> Later, >> Lukas @lzap Zapletal >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Have a nice day, Tomer Brisker Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Script to recycle passenger processes
On Wed, May 10, 2017 at 10:28 AM, Lukas Zapletal wrote: > Hello, > > I wrote a script that gracefully terminates Passenger processes > consuming more than 2.5 GB of private RSS memory. Typical consumption > of Foreman with Katello and other basic plugins is around 1.5 GB and > since only Passenger Enterprise allows to limit maximum amount of RSS > memory for processes, this simple script does exactly that: > > https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 > > Put it into your /etc/cron.hourly/ and make it executable, make sure > you read root emails or forward it properly, the script reports all > terminations performed on STDOUT. I would like you to test this script > in production and get back to me with feedback about what you think. > > Motivation is simple - we often introduce bugs in our Rails codebase > which performs some eager loading or there are memory leaks in our > code or dependencies and Passenger processes can grow up to dozens > gigabytes. Unfortunately, there is no other way of getting out other > than restarting httpd with passenger. This script could help to avoid > situations when production instance starts to swap hard thank to some > small regression we introduced. Also administrator will be noticed > early via email when this happens so we will keep track of these > regressions in production. > > http://projects.theforeman.org/issues/19496 > > Feedback appreciated. > > I gave it a spin, two comments: 1. my default memory size was around 400m i had to change the script it in order to see it in action, maybe we should take into account avail memory and how many passenger processes are allowed - the default of 2.5gb seems a bit too high? 2. in order to use your script in an scl env, you would need to wrap it with scl enable passenger-recycler Thanks! Ohad > -- > Later, > Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Script to recycle passenger processes
Hello, I wrote a script that gracefully terminates Passenger processes consuming more than 2.5 GB of private RSS memory. Typical consumption of Foreman with Katello and other basic plugins is around 1.5 GB and since only Passenger Enterprise allows to limit maximum amount of RSS memory for processes, this simple script does exactly that: https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 Put it into your /etc/cron.hourly/ and make it executable, make sure you read root emails or forward it properly, the script reports all terminations performed on STDOUT. I would like you to test this script in production and get back to me with feedback about what you think. Motivation is simple - we often introduce bugs in our Rails codebase which performs some eager loading or there are memory leaks in our code or dependencies and Passenger processes can grow up to dozens gigabytes. Unfortunately, there is no other way of getting out other than restarting httpd with passenger. This script could help to avoid situations when production instance starts to swap hard thank to some small regression we introduced. Also administrator will be noticed early via email when this happens so we will keep track of these regressions in production. http://projects.theforeman.org/issues/19496 Feedback appreciated. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
