Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On 30/10/14 23:49, Itamar Heim wrote: On 10/28/2014 05:23 PM, Dan Kenigsberg wrote: On Tue, Oct 28, 2014 at 12:42:19PM +0100, Sven Kieske wrote: On 28/10/14 12:28, Daniel Helgenberger wrote: But it is a short term solution until the issue is fixed or El6 obsoleted (well, this might be quite soon for a protect like oVirt?) Well I doubt EL6 support will be dropped from ovirt, as downstream still requires it to work on EL6 in the future, EL6 is still in production phase. It won't happen in ovirt-3.6, but support for el6 would be dropped at actually: http://lists.ovirt.org/pipermail/users/2014-September/027421.html interesting what about engine side? when will support get dropped there? This is very important because it most likely requires to install a new server with el7 and recover engine db there, which needs some planning. some point in the future. Backporting packages an re-solving bugs that have been already solved in el7 is little fun, and drains our developement and QE resources. We should start thinking how and when to phase el6 out. Dan. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On 28/10/14 11:07, Daniel Helgenberger wrote: As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd This package is provided and installed from the base repo on el6. I really would discourage to blow the ovir repo up by yet-another-package. imho this should get fixed/backported in el6 base repo. just my 2 cents of course, but if ovirt ships this package you also need to maintain it. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
Il 28/10/2014 11:07, Daniel Helgenberger ha scritto: Hello Sandro, I just updated BZ [1] that m2crypto-0.21.1 from FC20 fixes the severe mem leak issue in VDSM for me on EL6 with Engine 3.4 and 3.5. As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd). Should I open an RFE or will this mail to the list suffice? For convenience, I added the rpm I am using as well as the SRPM I used to compile this version [2]. Let me see if I've understood. Fedora 20 already have m2crypto-0.21.1-13.fc20 Fedora 19 has m2crypto-0.21.1-11 Looking at the RPM changelog [1], 0.21.1-11 on F19 and your 0.21.1-12 on F20 are just mass rebuild so no update needed on Fedora. CentOS 7 has m2crypto-0.21.1-15 and don't need an update. CentOS 6.6 has m2crypto-0.20.2-9 (yes, centos 6.6 is out http://isoredirect.centos.org/centos/6.6/) which looks the same on 6.5. Are you really sure that the issue is on m2crypto side and not on vdsm side? If so, a bug against m2crytpo should be opened and quickly fixed on RHEL/CentOS. I don't think that just rebuilding m2crypto from F20 is a long term solution. Dan? [1] http://koji.fedoraproject.org/koji/buildinfo?buildID=485699 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 [2] ftp://rpmfind.net/linux/fedora-secondary/development/rawhide/source/SRPMS/m/m2crypto-0.21.1-12.fc20.src.rpm -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On 28.10.2014 11:55, Sandro Bonazzola wrote: Il 28/10/2014 11:07, Daniel Helgenberger ha scritto: Hello Sandro, I just updated BZ [1] that m2crypto-0.21.1 from FC20 fixes the severe mem leak issue in VDSM for me on EL6 with Engine 3.4 and 3.5. As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd). Should I open an RFE or will this mail to the list suffice? For convenience, I added the rpm I am using as well as the SRPM I used to compile this version [2]. Let me see if I've understood. Fedora 20 already have m2crypto-0.21.1-13.fc20 Fedora 19 has m2crypto-0.21.1-11 Looking at the RPM changelog [1], 0.21.1-11 on F19 and your 0.21.1-12 on F20 are just mass rebuild so no update needed on Fedora. I also think that EL7's version might be fine, too. I just used the version from fc20 because it was a known working one confirmed by Markus, see BZ (I confess I did not read through the changelogs but was only interested in solving the issue :) CentOS 7 has m2crypto-0.21.1-15 and don't need an update. CentOS 6.6 has m2crypto-0.20.2-9 (yes, centos 6.6 is out http://isoredirect.centos.org/centos/6.6/) which looks the same on 6.5. Are you really sure that the issue is on m2crypto side and not on vdsm side? Not quite. The issue is fixed however. It can still be a bug in VDSM with some special call or something. But I clearly lack knowledge here. My reasoning is, however, finding the root cause might be a time consuming process and the outcome in the end only be of academical value since the issue is not there with newer versions of m2crypto. If so, a bug against m2crytpo should be opened and quickly fixed on RHEL/CentOS. Thats what I am asking in the BZ. I am happy to open the issue in RHEL (CentOS is downstream...) but I have no subscription and cannot confirm the version RHEL is using. Also I need someone who can answer whatever this is an issue of m2crypto or VDSM (see above). I don't think that just rebuilding m2crypto from F20 is a long term solution. Me neither. But it is a short term solution until the issue is fixed or El6 obsoleted (well, this might be quite soon for a protect like oVirt?) Also, the new m2cypto version might be requirement of the VDSM rpm in the future; I think there are few people witch run VDSM apart from being an oVirt / RHEV host. This way the impact and possible side effects are minimized (if there are any). Dan? [1] http://koji.fedoraproject.org/koji/buildinfo?buildID=485699 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 [2] ftp://rpmfind.net/linux/fedora-secondary/development/rawhide/source/SRPMS/m/m2crypto-0.21.1-12.fc20.src.rpm -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On 28/10/14 12:28, Daniel Helgenberger wrote: But it is a short term solution until the issue is fixed or El6 obsoleted (well, this might be quite soon for a protect like oVirt?) Well I doubt EL6 support will be dropped from ovirt, as downstream still requires it to work on EL6 in the future, EL6 is still in production phase. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On 28.10.2014 11:51, Sven Kieske wrote: On 28/10/14 11:07, Daniel Helgenberger wrote: As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd This package is provided and installed from the base repo on el6. I really would discourage to blow the ovir repo up by yet-another-package. imho this should get fixed/backported in el6 base repo. I very much agree this is the best solution. But this might take a lot of time as we fist need to find the root cause (see below). I reasoned therefore adding m2crypto to oVirt EL6 repos should be a temporary measure. I think everybody running EL6 hosts is suffering from this bug. However, usually there are large amounts of RAM available in production hosts and VDSM might be restarted once in a while so this leak may as well be unnoticed - until the host is loaded and fails because of endless swapping. Further, it is totally unclear to me whatever this is a bug in VDSM or m2crypto. If this is more clear, I am happy to open a BZ against m2crypto EL6. just my 2 cents of course, but if ovirt ships this package you also need to maintain it. I would volunteer to provide the RPMs for oVirt for the time being; as I did already. I have to build them anyway and if this makes any sense (OTOH, changes in this package are very rare it seems). PS: May remark about EL6 being obsoleted was not meant seriously ;) -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On Tue, Oct 28, 2014 at 11:55:40AM +0100, Sandro Bonazzola wrote: Il 28/10/2014 11:07, Daniel Helgenberger ha scritto: Hello Sandro, I just updated BZ [1] that m2crypto-0.21.1 from FC20 fixes the severe mem leak issue in VDSM for me on EL6 with Engine 3.4 and 3.5. As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd). Should I open an RFE or will this mail to the list suffice? For convenience, I added the rpm I am using as well as the SRPM I used to compile this version [2]. Let me see if I've understood. Fedora 20 already have m2crypto-0.21.1-13.fc20 Fedora 19 has m2crypto-0.21.1-11 Looking at the RPM changelog [1], 0.21.1-11 on F19 and your 0.21.1-12 on F20 are just mass rebuild so no update needed on Fedora. CentOS 7 has m2crypto-0.21.1-15 and don't need an update. CentOS 6.6 has m2crypto-0.20.2-9 (yes, centos 6.6 is out http://isoredirect.centos.org/centos/6.6/) which looks the same on 6.5. Are you really sure that the issue is on m2crypto side and not on vdsm side? If so, a bug against m2crytpo should be opened and quickly fixed on RHEL/CentOS. I don't think that just rebuilding m2crypto from F20 is a long term solution. Dan? If indeed Bug 1147148 - M2Crypto usage in vdsm leaks memory does not manifest itself with m2crypto-0.21.1 and is limited to el6, it's great news. I've move the bug to RHEL6's m2crypto. I hope it can be solved there, properly. Until this takes place, we can place Daniel's m2crypto build somewhere and suggest people to use it (preferably, we'd take a koji scratch build). We should not drop it into the default ovirt repo just yet, as no package is error free, so we should better check this new m2crypto first. [1] http://koji.fedoraproject.org/koji/buildinfo?buildID=485699 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 [2] ftp://rpmfind.net/linux/fedora-secondary/development/rawhide/source/SRPMS/m/m2crypto-0.21.1-12.fc20.src.rpm -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On 28.10.2014 14:02, Dan Kenigsberg wrote: On Tue, Oct 28, 2014 at 11:55:40AM +0100, Sandro Bonazzola wrote: Il 28/10/2014 11:07, Daniel Helgenberger ha scritto: Hello Sandro, I just updated BZ [1] that m2crypto-0.21.1 from FC20 fixes the severe mem leak issue in VDSM for me on EL6 with Engine 3.4 and 3.5. As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd). Should I open an RFE or will this mail to the list suffice? For convenience, I added the rpm I am using as well as the SRPM I used to compile this version [2]. Let me see if I've understood. Fedora 20 already have m2crypto-0.21.1-13.fc20 Fedora 19 has m2crypto-0.21.1-11 Looking at the RPM changelog [1], 0.21.1-11 on F19 and your 0.21.1-12 on F20 are just mass rebuild so no update needed on Fedora. CentOS 7 has m2crypto-0.21.1-15 and don't need an update. CentOS 6.6 has m2crypto-0.20.2-9 (yes, centos 6.6 is out http://isoredirect.centos.org/centos/6.6/) which looks the same on 6.5. Are you really sure that the issue is on m2crypto side and not on vdsm side? If so, a bug against m2crytpo should be opened and quickly fixed on RHEL/CentOS. I don't think that just rebuilding m2crypto from F20 is a long term solution. Dan? If indeed Bug 1147148 - M2Crypto usage in vdsm leaks memory does not manifest itself with m2crypto-0.21.1 and is limited to el6, it's great news. That's an educated guess by me. At least I know for a fact: - Leak on EL6 running EL6 m2crypto - No leak on FC20 running FC20 m2crypto - No leak on EL6 running FC20 m2crypto Further I guess this issue is also gone with EL7 (I will test this soon as I migrate my hosts to EL7). I've move the bug to RHEL6's m2crypto. I hope it can be solved there, properly. Thanks for taking care of that. Can you please post the BZ here for me to keep track? Until this takes place, we can place Daniel's m2crypto build somewhere and suggest people to use it (preferably, we'd take a koji scratch build). We should not drop it into the default ovirt repo just yet, as no package is error free, so we should better check this new m2crypto first. True. A hint in Release Nots / known issues should do the trick in the mean time. [1] http://koji.fedoraproject.org/koji/buildinfo?buildID=485699 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 [2] ftp://rpmfind.net/linux/fedora-secondary/development/rawhide/source/SRPMS/m/m2crypto-0.21.1-12.fc20.src.rpm -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On Tue, Oct 28, 2014 at 01:13:36PM +, Daniel Helgenberger wrote: On 28.10.2014 14:02, Dan Kenigsberg wrote: On Tue, Oct 28, 2014 at 11:55:40AM +0100, Sandro Bonazzola wrote: Il 28/10/2014 11:07, Daniel Helgenberger ha scritto: Hello Sandro, I just updated BZ [1] that m2crypto-0.21.1 from FC20 fixes the severe mem leak issue in VDSM for me on EL6 with Engine 3.4 and 3.5. As I have seen no side effects, I propose adding a newer m2crypto to the ovirt EL6 repos; with fixes the issue immediately (well, you need to restart vdsmd). Should I open an RFE or will this mail to the list suffice? For convenience, I added the rpm I am using as well as the SRPM I used to compile this version [2]. Let me see if I've understood. Fedora 20 already have m2crypto-0.21.1-13.fc20 Fedora 19 has m2crypto-0.21.1-11 Looking at the RPM changelog [1], 0.21.1-11 on F19 and your 0.21.1-12 on F20 are just mass rebuild so no update needed on Fedora. CentOS 7 has m2crypto-0.21.1-15 and don't need an update. CentOS 6.6 has m2crypto-0.20.2-9 (yes, centos 6.6 is out http://isoredirect.centos.org/centos/6.6/) which looks the same on 6.5. Are you really sure that the issue is on m2crypto side and not on vdsm side? If so, a bug against m2crytpo should be opened and quickly fixed on RHEL/CentOS. I don't think that just rebuilding m2crypto from F20 is a long term solution. Dan? If indeed Bug 1147148 - M2Crypto usage in vdsm leaks memory does not manifest itself with m2crypto-0.21.1 and is limited to el6, it's great news. That's an educated guess by me. At least I know for a fact: - Leak on EL6 running EL6 m2crypto - No leak on FC20 running FC20 m2crypto - No leak on EL6 running FC20 m2crypto Further I guess this issue is also gone with EL7 (I will test this soon as I migrate my hosts to EL7). I've move the bug to RHEL6's m2crypto. I hope it can be solved there, properly. Thanks for taking care of that. Can you please post the BZ here for me to keep track? [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 I just moved your bug to the component that it belongs to (based on your analysis). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Proposing fix for EL6 m2crypto memory leak
On Tue, Oct 28, 2014 at 12:42:19PM +0100, Sven Kieske wrote: On 28/10/14 12:28, Daniel Helgenberger wrote: But it is a short term solution until the issue is fixed or El6 obsoleted (well, this might be quite soon for a protect like oVirt?) Well I doubt EL6 support will be dropped from ovirt, as downstream still requires it to work on EL6 in the future, EL6 is still in production phase. It won't happen in ovirt-3.6, but support for el6 would be dropped at some point in the future. Backporting packages an re-solving bugs that have been already solved in el7 is little fun, and drains our developement and QE resources. We should start thinking how and when to phase el6 out. Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
