Re: [ovirt-users] ?3.4: VDSM Memory consumption
On 30.09.2014 17:09, Sandro Bonazzola wrote: Il 30/09/2014 17:03, Dan Kenigsberg ha scritto: On Tue, Sep 30, 2014 at 10:23:47AM +, Daniel Helgenberger wrote: On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side. Please follow [1] section Configure ovirt
Re: [ovirt-users] ?3.4: VDSM Memory consumption
- Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please to set it to false and restart the engine. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ?3.4: VDSM Memory consumption
Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ?3.4: VDSM Memory consumption
- Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side. Please follow [1] section Configure ovirt-engine and vdsm to work in non-secure mode There is wrong name of the option and it should
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side. Indeed your beliefs were true; I was already running in this error while starting vdsm: vdsm: Running validate_configuration FAILED: conflicting vdsm and libvirt-qemu tls configuration. vdsm.conf with ssl=False
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side. Please follow [1] section Configure ovirt-engine and vdsm to work in non-secure mode There is wrong name of the option and it should be EncryptHostCommunication. [1] http://www.ovirt.org/Developers_All_In_One I
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On Tue, Sep 30, 2014 at 10:23:47AM +, Daniel Helgenberger wrote: On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On Tue, Sep 30, 2014 at 10:17:59AM +, Daniel Helgenberger wrote: On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side
Re: [ovirt-users] ?3.4: VDSM Memory consumption
Il 30/09/2014 17:03, Dan Kenigsberg ha scritto: On Tue, Sep 30, 2014 at 10:23:47AM +, Daniel Helgenberger wrote: On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl = false' this on on all hosts? Please to set it to false and restart the engine. I believe that you need to update a bit more on vdsm side. Please follow [1] section Configure ovirt-engine and vdsm to work in non-secure mode
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On Di, 2014-09-30 at 16:04 +0100, Dan Kenigsberg wrote: On Tue, Sep 30, 2014 at 10:17:59AM +, Daniel Helgenberger wrote: On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true
Re: [ovirt-users] ?3.4: VDSM Memory consumption
Hello Sandro, On Di, 2014-09-30 at 17:09 +0200, Sandro Bonazzola wrote: Il 30/09/2014 17:03, Dan Kenigsberg ha scritto: On Tue, Sep 30, 2014 at 10:23:47AM +, Daniel Helgenberger wrote: On 30.09.2014 11:57, Piotr Kliczewski wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Piotr Kliczewski pklic...@redhat.com, Dan Kenigsberg dan...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption Hello Piotr, On 30.09.2014 08:37, Piotr Kliczewski wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Daniel Helgenberger daniel.helgenber...@m-box.de, pklic...@redhat.com Cc: Francesco Romani from...@redhat.com, users@ovirt.org Sent: Tuesday, September 30, 2014 1:11:42 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct: 1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c update vdc_options set option_value = 'false' where option_name = 'EncryptHostCommunication'; engine=# SELECT * from vdc_options where option_name='EncryptHostCommunication'; option_id | option_name| option_value | version ---+--+--+- 335 | EncryptHostCommunication | false| general (1 row) 2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl = true ssl = false 4. restart VDSM I assume I have to set 'ssl
Re: [ovirt-users] 3.4: VDSM Memory consumption
Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, On 27.09.2014 13:18, Daniel Helgenberger wrote: Hallo Dan, I just opened a BZ aginst this [1]. Please let me know if I can be of further assistance. I stopped the cron script for now, so vdsm can 'grow nicely' (about 91.5 MB per 6hrs) Cheers, [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 On 24.09.2014 19:20, Dan Kenigsberg wrote: On 01.09.2014 18:08, Dan Kenigsberg wrote: On Mon, Sep 01, 2014 at 03:30:53PM +, Daniel Helgenberger wrote: Hello, in my LAB cluster I run into OOM conditions frequently because of a huge VDSM process. The memory stats from my nodes right now: Node A; running one VM: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 supervdsmServer Node B, running 3 VMs including HosedEngine: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm Basically VDSM consumes more then all my VMs. I thought of VDSM as a 'supervisor' process for qemu-kvm? I attached recend vdsm logs as well as a screen shot. Thanks for your report. It sounds like Bug 1130045 - Very high memory consumption which we believe is due to python-ioprocess. On Wed, Sep 24, 2014 at 03:39:25PM +, Daniel Helgenberger wrote: Hi Dan, just to get this right, you yourself pointed me to the BZ. I was looking at the duplicate, since the metadata tags 3.4. Sorry my lack of knowlage, I really have no idea whatever there is an ioprocess python binding in 3.4 or not - I just see vdsmd resident size growing in 3.4. The top output below was from 3.4.3; I just upgraded to 3.4.4. But clearly vdsmd should not use 10GB RAM? I'm sorry to have mislead you. The bug I refered to was indeed due to ioprocess, and caused a very dramatic memory leak in 3.5. We have yet another memory leak in 3.5.0, when managing gluster blocks Bug 1142647 - supervdsm leaks memory when using glusterfs 3.4.z does not use ioprocess, and do not have that gluster bug, so you are seeing completely different and much older. These leaks are not so easy to debug - but they are important. I'd love if you open a BZ about it. Please specify the rate of the leak, when does it happen (when a host has VMs? when a host is polled by Engine? On nfs? iscsi? What's `lsof -p pid-of-fat-vdsm`? I think that Francesco has some debug patches to help nail it down - he can provide smarter questions. Dan. -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] 3.4: VDSM Memory consumption
- Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. In absence of commands from Engine VDSM will just sit idle and it will not do anything to VMs except it's own sampling. If you are on IRC, on freenode/#vdsm or OFTC/#ovirt, please ping me anytime - see my sign below for my IRC nickname. -- Francesco Romani RedHat Engineering Virtualization R D Phone: 8261328 IRC: fromani ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On Mon, Sep 29, 2014 at 10:25:22AM +, Daniel Helgenberger wrote: Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? # iptables -A INPUT -p tcp --destination-port 54321 -j REJECT on the host would render it non-responsive in Engine. I wonder if the leak continues. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] 3.4: VDSM Memory consumption
Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users] 3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? In absence of commands from Engine VDSM will just sit idle and it will not do anything to VMs except it's own sampling. If you are on IRC, on freenode/#vdsm or OFTC/#ovirt, please ping me anytime - see my sign below for my IRC nickname. Thanks. -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] 3.4: VDSM Memory consumption
- Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users] 3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. -- Francesco Romani RedHat Engineering Virtualization R D Phone: 8261328 IRC: fromani ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] 3.4: VDSM Memory consumption
Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! -- Francesco Romani RedHat Engineering Virtualization R D Phone: 8261328 IRC: fromani smime.p7s Description: S/MIME cryptographic signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ?3.4: VDSM Memory consumption
On Mon, Sep 29, 2014 at 09:02:19PM +, Daniel Helgenberger wrote: Hello Francesco, -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv On 29.09.2014, at 22:19, Francesco Romani from...@redhat.com wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Francesco Romani from...@redhat.com Cc: Dan Kenigsberg dan...@redhat.com, users@ovirt.org Sent: Monday, September 29, 2014 2:54:13 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Hello Francesco, On 29.09.2014 13:55, Francesco Romani wrote: - Original Message - From: Daniel Helgenberger daniel.helgenber...@m-box.de To: Dan Kenigsberg dan...@redhat.com Cc: users@ovirt.org Sent: Monday, September 29, 2014 12:25:22 PM Subject: Re: [ovirt-users]3.4: VDSM Memory consumption Dan, I just reply to the list since I do not want to clutter BZ: While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, Hi Daniel, there is indeed a memory profiling patch under discussion: http://gerrit.ovirt.org/#/c/32019/ but for your case we'll need a backport to 3.4.x and clearer install instructions, which I'll prepare as soon as possible. I updated the BZ (and are now blocking 54321/tcp on one of my hosts). and verified it is not reachable. As general info: This system I am using is my LAB / Test / eval setup for a final deployment for ovirt (then 3.5) in production; so it will go away some time in the future (a few weeks / months). If I am the only one experiencing this problem then you might be better of allocating resources elsewhere ;) Thanks for your understanding :) Unfortunately it is true that developer resources aren't so abundant, but it is also true that memleaks should never be discarded easily and without due investigation, considering the nature and the role of VDSM. So, I'm all in for further investigation regarding this issue. As for your question: if I understood correctly what you are asking (still catching up the thread), if you are trying to rule out the stats polling made by Engine to this bad leak, one simple way to test is just to shutdown Engine, and let VDSMs run unguarded on hypervisors. You'll be able to command these VDSMs using vdsClient or restarting Engine. As I said in my BZ comment this is not an option right now, but if understand the matter correctly IPTABLES reject should ultimately do the same? Definitely yes! Just do whatever it is more convenient for you. As you might have already seen in the BZ comment the leak stopped after blocking the port. Though this is clearly no permanent option - please let me know if I can be of any more assistance! The immediate suspect in this situation is M2Crypto. Could you verify that by re-opening the firewall and setting ssl=False in vdsm.conf? You should disable ssl on Engine side and restart both Engine and Vdsm (too bad I do not recall how that's done on Engine: Piotr, can you help?). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] 3.4: VDSM Memory consumption
Hallo Dan, I just opened a BZ aginst this [1]. Please let me know if I can be of further assistance. I stopped the cron script for now, so vdsm can 'grow nicely' (about 91.5 MB per 6hrs) Cheers, [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 On 24.09.2014 19:20, Dan Kenigsberg wrote: On 01.09.2014 18:08, Dan Kenigsberg wrote: On Mon, Sep 01, 2014 at 03:30:53PM +, Daniel Helgenberger wrote: Hello, in my LAB cluster I run into OOM conditions frequently because of a huge VDSM process. The memory stats from my nodes right now: Node A; running one VM: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 supervdsmServer Node B, running 3 VMs including HosedEngine: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm Basically VDSM consumes more then all my VMs. I thought of VDSM as a 'supervisor' process for qemu-kvm? I attached recend vdsm logs as well as a screen shot. Thanks for your report. It sounds like Bug 1130045 - Very high memory consumption which we believe is due to python-ioprocess. On Wed, Sep 24, 2014 at 03:39:25PM +, Daniel Helgenberger wrote: Hi Dan, just to get this right, you yourself pointed me to the BZ. I was looking at the duplicate, since the metadata tags 3.4. Sorry my lack of knowlage, I really have no idea whatever there is an ioprocess python binding in 3.4 or not - I just see vdsmd resident size growing in 3.4. The top output below was from 3.4.3; I just upgraded to 3.4.4. But clearly vdsmd should not use 10GB RAM? I'm sorry to have mislead you. The bug I refered to was indeed due to ioprocess, and caused a very dramatic memory leak in 3.5. We have yet another memory leak in 3.5.0, when managing gluster blocks Bug 1142647 - supervdsm leaks memory when using glusterfs 3.4.z does not use ioprocess, and do not have that gluster bug, so you are seeing completely different and much older. These leaks are not so easy to debug - but they are important. I'd love if you open a BZ about it. Please specify the rate of the leak, when does it happen (when a host has VMs? when a host is polled by Engine? On nfs? iscsi? What's `lsof -p pid-of-fat-vdsm`? I think that Francesco has some debug patches to help nail it down - he can provide smarter questions. Dan. -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] 3.4: VDSM Memory consumption
Hi Dan, just to get this right, you yourself pointed me to the BZ. I was looking at the duplicate, since the metadata tags 3.4. Sorry my lack of knowlage, I really have no idea whatever there is an ioprocess python binding in 3.4 or not - I just see vdsmd resident size growing in 3.4. The top output below was from 3.4.3; I just upgraded to 3.4.4. But clearly vdsmd should not use 10GB RAM? Thanks! On 01.09.2014 18:08, Dan Kenigsberg wrote: On Mon, Sep 01, 2014 at 03:30:53PM +, Daniel Helgenberger wrote: Hello, in my LAB cluster I run into OOM conditions frequently because of a huge VDSM process. The memory stats from my nodes right now: Node A; running one VM: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 supervdsmServer Node B, running 3 VMs including HosedEngine: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm Basically VDSM consumes more then all my VMs. I thought of VDSM as a 'supervisor' process for qemu-kvm? I attached recend vdsm logs as well as a screen shot. Thanks for your report. It sounds like Bug 1130045 - Very high memory consumption which we believe is due to python-ioprocess. -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] 3.4: VDSM Memory consumption
On 01.09.2014 18:08, Dan Kenigsberg wrote: On Mon, Sep 01, 2014 at 03:30:53PM +, Daniel Helgenberger wrote: Hello, in my LAB cluster I run into OOM conditions frequently because of a huge VDSM process. The memory stats from my nodes right now: Node A; running one VM: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 supervdsmServer Node B, running 3 VMs including HosedEngine: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm Basically VDSM consumes more then all my VMs. I thought of VDSM as a 'supervisor' process for qemu-kvm? I attached recend vdsm logs as well as a screen shot. Thanks for your report. It sounds like Bug 1130045 - Very high memory consumption which we believe is due to python-ioprocess. On Wed, Sep 24, 2014 at 03:39:25PM +, Daniel Helgenberger wrote: Hi Dan, just to get this right, you yourself pointed me to the BZ. I was looking at the duplicate, since the metadata tags 3.4. Sorry my lack of knowlage, I really have no idea whatever there is an ioprocess python binding in 3.4 or not - I just see vdsmd resident size growing in 3.4. The top output below was from 3.4.3; I just upgraded to 3.4.4. But clearly vdsmd should not use 10GB RAM? I'm sorry to have mislead you. The bug I refered to was indeed due to ioprocess, and caused a very dramatic memory leak in 3.5. We have yet another memory leak in 3.5.0, when managing gluster blocks Bug 1142647 - supervdsm leaks memory when using glusterfs 3.4.z does not use ioprocess, and do not have that gluster bug, so you are seeing completely different and much older. These leaks are not so easy to debug - but they are important. I'd love if you open a BZ about it. Please specify the rate of the leak, when does it happen (when a host has VMs? when a host is polled by Engine? On nfs? iscsi? What's `lsof -p pid-of-fat-vdsm`? I think that Francesco has some debug patches to help nail it down - he can provide smarter questions. Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
