[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Any ETA on a port to Debian-ARM? Currently using Linux version 4.1.15+ (dc4@dc4-XPS13-9333) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #830 Tue Dec 15 16:58:28 GMT 2015 Thanks. Jeff Anttila -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
AFAICT, a fix to this has not yet been committed to Debian VCS so I have nothing to track or cherry-pick in Ubuntu yet. Is this accurate or have I missed something? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
@racb did you check the master-5.6 branch? I am quite sure that Debian jessie already has the new sessionclean script. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Ah sorry, thanks. I looked again. I saw the commits but assumed that 2014-08 was too old, but the comments do actually date from them. So I think that Wily contains the new script, but Trusty will need it cherry-picked. Setting bug tasks accordingly. ** Tags added: server-next ** Also affects: php5 (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: php5 (Ubuntu) Status: Confirmed => Fix Released ** Changed in: php5 (Ubuntu Trusty) Status: New => Triaged ** Changed in: php5 (Ubuntu Trusty) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
I just noticed 800+ processes on my raspberry pi 2 ubuntu 14.04, all were spawns from this cron. Although cron script is bad in terms of cpu use, I see it's set to run every 30 minutes. So I am also thinking something else is also going on, probably some hanging with lsof anyways. This script can be run once a day (not 48 times a day as currently). Session information is stored in /tmp so if you don't get it full, you may even disable it completely, especially if you don't use php much. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
My 14.10 laptop just froze at the moment when the cron.d job kicked in (9 minutes past the hour). There was not much else running at the time. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: php5 (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1356113] Re: PHP5 session clean cron job causes OOM
Sorry for the late reply. I've been out for the last couple of weeks. On Tue, Aug 19, 2014 at 08:13:55AM -, Ondřej Surý wrote: 2. Will these updated be present in 14.04.x? 3. Will these changes be present in PHP 5.5.x? I will merge these changes to master-5.5 branch in Debian git repo (and my PPA ppa:ondrej/php5). It's up to Ubuntu maintainer (@racb) to pick that or not. For updates to 14.04, I have to follow our SRU process for any updates, and can only push fixes for high-impact bugs with low regression risk. Each update is a separate effort and much more involved than a simple push. It isn't clear to me that this change qualifies, but I'm open to discussing this. The rationale, requirements and process is described here: https://wiki.ubuntu.com/StableReleaseUpdates By PHP 5.5.x, do you mean 14.10? We're past feature freeze for 14.10 now, but I can push bugfixes that aren't new features. The criteria isn't much different, but I am permitted to judge that for myself and can just upload. Again, I'm open to discussion and am happy to upload fixes that are justified. For anything else, we'll have to wait for the next release cycle. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
There's one more improvement the packages could do: Each SAPI could drop it's session snippet to /var/lib/php5/sapi/sapi and the sessionclean script would read those snippets and clean the directory according to really installed snippets. That would make the sessionclean script SAPI agnostic and could support HHVM out of the box... So if you talk to HHVM packagers and they show interest, I could implement this. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
That does seem more appropriate than putting all the SAPI methods in php-common. The HHVM guys (Paul Tarjan, etc) gave me the green light to do the session cleanup bit, so I'll take care of it once we settle on a script. Wouldn't /usr/lib/php5/sapi/sapi be a better location for scripts? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Wouldn't /usr/lib/php5/sapi/sapi be a better location for scripts? Yup... -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Either ``` for sapi in /usr/lib/php5/sapi/*; do . $sapi ``` and have the /usr/lib/php5/sapi/sapi contain shell snippet with definitions or ``` cat /usr/lib/php5/sapi/* | while IFS=: read -r VAR1 VAR2 VAR3 VAR4; do ``` and keep the current format... -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Btw, if we go the route of separate sapi folders inside /usr/lib/php5/sapi/ we should also move php5-fpm-checkconf into /usr/lib/php5/sapi/fpm/ (I used fpm instead of php5-fpm here since that how it is in /etc/php5/). -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
I prefer option #1 - sourcing the sessionclean scripts from /usr/lib/php5/sapi/* or perhaps /usr/lib/php5/sapi/sapi/sessionclean so we can make room for more sapi-specific scripts should the need arise. If we keep the logic in php-common, a con is that we can't use it for HHVM (not that HHVM should drive the decision). A pro for this method is that we can have /usr/lib/php5/sapi/sapi/sessionconfig scripts that simply spit out the params we need, then we feed them into the sessionclean script as you've described (sort of like /usr/lib/php5/maxlifetime is used now) Another pro for this method is that we can optimize cleanup by de-duplicating the session.save_paths and using max(session.gc_maxlifetime). -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
1. PROC_NAMES should be defined as proc_names or used as PROC_NAMES (inconsistent case) Good catch, fixed. 2. printf %s:%s:%s\n $save_path $gc_maxlifetime has 2 arguments but the definition takes 3, I would simplify it to this: echo $save_path:$gc_maxlifetime Good catch, fixed. 3. If apache2filter is installed, $proc_names will contains apache2 apache2, which, when passed to pidof, will return all the PIDs twice, so I've uniq'd them by delimiting them with \n and using sort -u | xargs pidof. apache2 and apache2filter cannot be co-installed. 4. I'm not entirely sure how cgi method would work, but I suppose it means the php command is running on the system by itself, so I've used the following bit to discover that pidof requires php, not php5 since that is the real name of the process: php -r 'sleep(60);echo Done;' The default package config points to /usr/lib/cgi-bin/php5 5. We've been using HHVM for a few years for one of our products, and I could easily add support for that (hhvm --php -r...), but things seem a bit out of order there, mainly, the php.ini is in /etc/hhvm instead of /etc/php5/hhvm It's up to the HHVM packages to sort this out. I doubt they will depend on php5-common at all. So I am skipping changes related to HHVM. It's start to feel like bikesheding. 1. Will PHP 5.6 be the default in Ubuntu 14.10? Nope, there was a separate bug report about this, and my recommendation was to stay with 5.5.latest 2. Will these updated be present in 14.04.x? 3. Will these changes be present in PHP 5.5.x? I will merge these changes to master-5.5 branch in Debian git repo (and my PPA ppa:ondrej/php5). It's up to Ubuntu maintainer (@racb) to pick that or not. 4. Are you going to be the Debian-side maintainer for the Debian-repo HHVM package? Nope, there are some Debian folks from fcbk who wanted to do the work. Since the name php5 is ambiguous with /usr/bin/php5, we can pass the full path to the executable, /usr/lib/cgi-bin/php5, to pidof. There's no need to do so. The worst case scenario is when /usr/bin/php5 session would get touched as well. it's a lot less expensive since PHP doesn't need to start up with all of its modules 3 times for each SAPI. Again it feels like over-optimizing, but I have merged a variant of this. session.save_path can be empty and empty string. It cannot be an empty string in Debian, since we patch the sources to have a default to point to /var/lib/php5/sessions/ Ondrej -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Indeed, I think we've passed the 80/20 mark on this one. I'll test your updated one today and we should be good. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Issue on line 8: echo -e $SAPIS | \ The -e is ending up in the output (perhaps dash's echo has no -e) resulting in this error: ./sessionclean: 10: [: /etc/php5/-e: unexpected operator I've removed the -e as our input is clean anyway. Also, I've run this to normalize the spacing: sed -i s/\t// sessionclean Other than that everything is working well. Thanks for the help here! ** Attachment added: sessionclean https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+attachment/4181650/+files/sessionclean -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Issue on line 8: Missed this in all the cruft generated by sh -x, and since the other output looked ok... Thanks for noticing that... Also thank you for the feedback, this was very valuable after all. I probably owe you an apology, so please accept my apology I should have been more calm from the beginning. I have probably met too many clueless people who have waved with expertness around and I am very glad this wasn't the case and this was very productive and expert interaction in the end. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Thanks Ondřej, no apology needed, this was indeed a constructive thread. Sometimes us open source geeks get all too used to being in charge and we forget how to communicate effectively. If I'm up in your neck of the woods some time I'll look you up and we can grab a beer :) Also, I'll do a PR on Facebook's HHVM repo which includes an HHVM- specific GC script (basically a non-iterative version of this one). -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
One problem with the solution you linked to is that it is extracting the gx_maxlifetime in seconds, then putting that in find's -cmin which expects minutes. Good catch. The code was there, but it got deleted accidentally. What do you think about running this for each SAPI? I am still not sure whether we are not breaking something with this change, but it's very good idea. I have incorporated the idea, but kept the one run per directory and improved it that it just takes the highest gc_maxlifetime value. There's also an optimization to run pidof only once, since iterating through /proc/ on servers with many processes would also take it's toll (although I doubt the common setups include more than one SAPI on the server...) Updated script is located in the git: http://anonscm.debian.org/cgit /pkg-php/php.git/tree/debian/sessionclean Could you please proofread it and test it on your servers? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Thanks, this is looking better now. Here are some notes: 1. PROC_NAMES should be defined as proc_names or used as PROC_NAMES (inconsistent case) 2. printf %s:%s:%s\n $save_path $gc_maxlifetime has 2 arguments but the definition takes 3, I would simplify it to this: echo $save_path:$gc_maxlifetime 3. If apache2filter is installed, $proc_names will contains apache2 apache2, which, when passed to pidof, will return all the PIDs twice, so I've uniq'd them by delimiting them with \n and using sort -u | xargs pidof. 4. I'm not entirely sure how cgi method would work, but I suppose it means the php command is running on the system by itself, so I've used the following bit to discover that pidof requires php, not php5 since that is the real name of the process: php -r 'sleep(60);echo Done;' 5. We've been using HHVM for a few years for one of our products, and I could easily add support for that (hhvm --php -r...), but things seem a bit out of order there, mainly, the php.ini is in /etc/hhvm instead of /etc/php5/hhvm I've attached an updated script. ** Attachment added: sessionclean https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+attachment/4180769/+files/sessionclean -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
I've added HHVM support and discovered another issue along the way: session.save_path can be empty and empty string. This is actually the default in stock PHP and HHVM. The PHP docs imply that an empty string is equal to /tmp (http://php.net/manual/en/session.configuration.php#ini.session.save- path), although that doesn't make much sense in Windows. Anyway, I am now checking for that as well, so if session.save_path==, we use /tmp. ** Attachment added: sessionclean https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+attachment/4181081/+files/sessionclean -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
It seems likely that HHVM will find its way into the Debian repo, so I'm going to see if they will set the session.save_path to /var/lib/php5: https://github.com/hhvm/packaging/pull/67 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
I have a feeling the CGI-mode process name is php5-cgi. I've installed it, but I haven't been able to get Apache 2.4 and mod_actions to play nice with it yet, so I haven't confirmed it. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
I noticed that you changed the default session.save_path to /var/lib/php5/sessions, which I think is a great idea (I have no idea what the modules/ dir is for). In order to align the HHVM packaging with the Debian PHP packaging I have some questions: 1. Will PHP 5.6 be the default in Ubuntu 14.10? 2. Will these updated be present in 14.04.x? 3. Will these changes be present in PHP 5.5.x? 4. Are you going to be the Debian-side maintainer for the Debian-repo HHVM package? The note to see /usr/lib/php5/maxlifetime is no longer needed as it doesn't exist http://anonscm.debian.org/cgit/pkg-php/php.git/tree/debian/php5-common.php5.cron.d?h=master-5.6 I was able to get php-cgi working, and it uses the binary named php5 (assuming the user follows the example in /etc/apache2/conf-available /serve-cgi-bin.conf). Since the name php5 is ambiguous with /usr/bin/php5, we can pass the full path to the executable, /usr/lib /cgi-bin/php5, to pidof. This latest script builds on the last one, but is over 2x faster by avoiding multiple calls to PHP per SAPI. Instead, it uses foreach(ini_get_all(session)) to get all the session config options, then uses simple commands to parse config options out. It seems like this is a little overkill, but I think it makes the code a bit more readable, and it's a lot less expensive since PHP doesn't need to start up with all of its modules 3 times for each SAPI. ** Attachment added: sessionclean https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+attachment/4181227/+files/sessionclean -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Thanks Ondřej, this is a much better script, and I agree that the lack of session.save_path support was rather annoying. This is still not optimal for high-performance servers, but it seems to be a good compromise for general use. Also, the fact that is honors SAPI-specific settings helps. Also, thanks for posting the thread regarding active sessions over the gc_maxlifetime, that is a strange issue. It would probably be more efficient to iterate /proc/x/fd for each of the SAPIs, but this seems like it would break easily, plus some instances of the process (php, for example) might be indistinguishable from the CLI version. I'll take a crack at it this weekend for the sake of curiosity. Good catch on the php-cli requirement - note that /usr/lib/php5/maxlifetime also uses php-cli in that way. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
What do you think about running this for each SAPI? It's actually *way* more efficient: for pid in $(pidof $sapi); do find /proc/$pid/fd -lname $session_save_path/* -exec touch {} \; done That will iterate all the PIDs of the running SAPI (we'll need to iterate them as well), then find all the files it has open and touch them. This is more efficient than lsof since it looks only at the files open by the SAPI in question, instead of every process. As an example, here's the timing of my proposed way and the current way: kamermans@steve-ubuntu:~$ time for pid in $(pidof apache2); do sudo find /proc/$pid/fd -lname /var/lib/php5/* -print -exec touch {} \; ; done /proc/4202/fd/21 real0m0.143s user0m0.069s sys 0m0.077s kamermans@steve-ubuntu:~$ time sudo /usr/bin/lsof -w -l +d /var/lib/php5 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME apache2 4202 33 21uW REG8,30 277620 /var/lib/php5/sess_bbn8sh6ljim484c4qt3inubhr3 real0m1.079s user0m0.514s sys 0m0.845s Notice that my version returns /proc/4202/fd/21, but that is the file descriptor which is a symbolic link to /var/lib/php5/sess_bbn8sh6ljim484c4qt3inubhr3. Touching the link touches the file, so the effect is the same. Also, since the find command finds and touches the file in (almost) one operation, so it's unlikely the file will disappear between the time that it's found and the time that it's touched. Yet another option is to alter the lsof call to restrict it to just processes that match the given name(s). This is also fast, although it not quite as fast as the aforementioned /proc method in my test: kamermans@steve-ubuntu:~$ time sudo /usr/bin/lsof -w -l -a +d /var/lib/php5 -c apache2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME apache2 4202 33 21uW REG8,30 266585 /var/lib/php5/sess_18and1t1ej6frr9kjai3m1dbj4 real0m0.182s user0m0.074s sys 0m0.091s Note that I added -a to AND the directory and command constraint, otherwise they would be OR'd. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
It just occurred to me that I was timing sudo in my last post, so here are the sudo-less numbers: root@steve-ubuntu:~# time for pid in $(pidof apache2); do find /proc/$pid/fd -lname /var/lib/php5/* -print -exec touch {} \; ; done /proc/30005/fd/21 real0m0.042s user0m0.016s sys 0m0.026s root@steve-ubuntu:~# time /usr/bin/lsof -w -l +d /var/lib/php5 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME apache2 30005 33 21uW REG8,30 268283 /var/lib/php5/sess_r5mhepbrhgmgsdnvkj72ppoc37 real0m1.076s user0m0.517s sys 0m0.846s root@steve-ubuntu:~# time /usr/bin/lsof -w -l -a +d /var/lib/php5 -c apache2 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME apache2 30005 33 21uW REG8,30 268283 /var/lib/php5/sess_r5mhepbrhgmgsdnvkj72ppoc37 real0m0.070s user0m0.032s sys 0m0.052s -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Ok, sorry for the repeated spamming tonight. One problem with the solution you linked to is that it is extracting the gx_maxlifetime in seconds, then putting that in find's -cmin which expects minutes. I've attached a script that incorporates the improvements of SAPI- specific session file cleaning, as well as my improved open-file touching code. Here's the difference in time on one of my production machines: # pidof php5-fpm | xargs php -r 'echo $argc running threads\n;' 102 running threads # time /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime) real0m10.631s user0m7.876s sys 0m2.738s # time /tmp/sessionclean_proposed.sh real0m0.341s user0m0.126s sys 0m0.241s ** Attachment added: sessionclean_propsed.sh https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+attachment/4178920/+files/sessionclean_propsed.sh -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
I was just trying to make a point You did make a point, but totally different from the one intended. Also your proposed fix is incomplete... # sed -n -e 's/^[[:space:]]*session.save_handler[[:space:]]*=[[:space:]]*\([[:alnum:]]\+\).*$/\1/p' /etc/php5/fpm/php.ini # php5 -c /etc/php5/fpm/php.ini -d error_reporting='~E_ALL' -r 'echo ini_get(session.save_handler).\n;' files The cron job is really meant for the default setup, since it also has other problems: * The cron job doesn't handle custom session.save_path * The cron job doesn't catch session files in subdirectories If you seriously think running lsof on a server every 30 minutes regardless of the necessity is perfectly safe, then I guess we can agree to disagree :) If you can come up with a better way how to skip and not delete session files that are still used by long running scripts (e.g. https://bugs.debian.org/626640), then I would gladly accept the new solution. I am also going to improve the wording of README.Debian to accent the need to modify the cron job if you modify the session handling. ** Bug watch added: Debian Bug tracker #626640 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626640 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Try this updated script that now requires php5-cli to be installed: http://anonscm.debian.org/cgit/pkg-php/php.git/tree/debian/sessionclean -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Steve, thank you for your report and for your time in helping to make Ubuntu better. I'm not really sure how to respond here. Ondřej is the Debian maintainer for PHP, and Ubuntu's packaging is derived from Debian's packaging, so his opinion matters. I appreciate Ondřej's participation in Ubuntu's bug tracker for this reason. I don't want Ubuntu's PHP packaging to deviate from Debian's. That's bad for everybody. Could you two figure out the best way of handling this for all users, please, so that Debian can be updated (if needed) and I can follow any changes in Ubuntu? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Thanks everyone for working on this. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
As an *expert* you surely heard of rm /etc/cron.d/php5 or even sed -i -e s/^/#/ /etc/cron.d/php5... -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Thanks for the completely non-constructive message. Google me if you want. Anyway, my point (and apparently your subpoint) is that the current script is not safe for production use. My improvements make it slightly more suitable out of the box. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
Should I be in awe or what? You can improve things even when not using the argument from authority. My point is that the default cron job works and is perfectly safe for most deployments and it's documented in README.Debian that you need to tweak the session cleanup if you do modification to session handling. And I don't think we need to add more knobs for the edge cases. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
You can improve things even when not using the argument from authority. I totally agree, I was just trying to make a point that I know what I'm talking about, as opposed to being a passive observer. You also know what you're talking about, and I can respect that as well. If you seriously think running lsof on a server every 30 minutes regardless of the necessity is perfectly safe, then I guess we can agree to disagree :) Prior to this bug report, my impression was that Ubuntu is developed in an open fashion where everybody is welcome to contribute. (http://community.ubuntu.com/contribute/developers/) This incident, however, has put a bad taste in my mouth. Had I not been an open-source developer already, I would certainly have turned away from the open- source community. Anyway, it is obvious that you are not going to consider my contribution, so I will stop pushing the issue. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1356113] Re: PHP5 session clean cron job causes OOM
On another note, why would we possibly need to run lsof every time we need to cleanup sessions? This is a very inefficient and expensive process. If someone knows how it came to be that this code was added, I would be very curious. Also, I see that recent versions of Ubuntu (14.04 for example) put a modules/ folder in /var/lib/php which is quite disconcerting since there are potentially thousands of session files in there as well. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to php5 in Ubuntu. https://bugs.launchpad.net/bugs/1356113 Title: PHP5 session clean cron job causes OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs