[ovirt-users] oVirt Weekly Meeting Minutes -- 2015-03-26
Minutes: http://ovirt.org/meetings/ovirt/2015/ovirt.2015-03-25-14.02.html Minutes (text): http://ovirt.org/meetings/ovirt/2015/ovirt.2015-03-25-14.02.txt Log: http://ovirt.org/meetings/ovirt/2015/ovirt.2015-03-25-14.02.log.html = #ovirt: oVirt Weekly Sync = Meeting started by ydary at 14:02:53 UTC. The full logs are available at http://ovirt.org/meetings/ovirt/2015/ovirt.2015-03-25-14.02.log.html . Meeting summary --- * Agenda and roll Call (ydary, 14:03:05) * infra update (ydary, 14:03:07) * 3.5.z updates (ydary, 14:03:08) * 3.6.0 status (ydary, 14:03:10) * conferences and workshops (ydary, 14:03:11) * other topics (ydary, 14:03:13) * infra update (ydary, 14:06:33) * CI breakage due to various missing deps in vdsm master is being fix now. (ydary, 14:08:31) * Jobs queue in jenkins is over 200 jobs, we need to find a way to process them in a faster way. Still waiting for new memory upgrades in phx lab. (ydary, 14:09:18) * Many jobs are starting for each patch causing load. Other than memory upgrade there is discssion on running some jobs in sequnce. mvk and dcaro are looking into this. (ydary, 14:12:47) * 3.5.z updates (ydary, 14:17:06) * 3.5.z status: http://lists.ovirt.org/pipermail/users/2015-March/032022.html (ydary, 14:19:41) * 5 blockers open, RC3 will be released next week, once they are all fixed. (ydary, 14:20:11) * 5 blockers open, oVirt 3.5.2 RC3 will be released next week, once they are all fixed. (ydary, 14:21:30) * ACTION: 3.5 branch tester, please use the CentOS 7 with CR enable, since we want to get early issues with this update. (ydary, 14:22:40) * 3.6 status (ydary, 14:23:12) * oVirt 3.6.0 Status: http://lists.ovirt.org/pipermail/users/2015-March/032019.html (ydary, 14:23:47) * Master branch broken on dependencies issues that is being worked on. (ydary, 14:24:19) * Integration 3.6 Status Update: Progress has been done this week for using cloud-init when deploying the engine using the ovirt-appliance. (ydary, 14:25:33) * Also on moving the hosted engine configuration to shared storage and supporting hyperconverged gluster. Several patches are currently under review. (ydary, 14:26:08) * No updates on Fedora 22 or Debian support this week. (ydary, 14:26:49) * FC22 support pending infra for wildfly / jdk 1.8. In Debian patches are pending review and there is a issue of libvirt package version available. (ydary, 14:29:41) * In CentOS, qemu-kvm-ev landed on Virt SIG for CentOS 7.1. (ydary, 14:30:13) * Storage 3.6.0 Status Update: Hosted Engine issues that may affect 3.5.0 is currently under investigation (#1205225). (ydary, 14:33:01) * No updates on feature status. (ydary, 14:35:55) * Gluster 3.6.0 Status Update: Geo-replication, volume snapshot management, brick provisioning and choosing gluster network for bricks - features merged and testing underway. (ydary, 14:37:06) * Waiting on testing issues for inital sanity. (ydary, 14:37:56) * REST API missing for these features and will be added in the next few weeks as well. (ydary, 14:38:25) * UX 3.6.0 Status Update: Patternfly tooltip infrastructure merged and Italian translation at 50% now. Other features are progressing nicely. (ydary, 14:41:37) * Node 3.6.0 Status Update: Big progress on the cockpit TUI migration. (ydary, 14:46:11) * rbarry added an initial backend for our configuration libraries (reusing the existing libraries from ovirt-node) to make them consumable from dbus and patches are been pushed. Currently pending builders and inclusion in some builds. Also progressed with refactoring to make the libaries consumable on regular hosts without breaking them. (ydary, 14:48:48) * Good progress on making the Engine appliance suitable for cloud-init usage with better integration with hosted-engine-setup. (ydary, 14:49:35) * Big effort is still in progress to clean up and align the node jenkins jobs better with the rest of oVirt CI. (ydary, 14:50:12) * Infra 3.6.0 Status Update: No updates from last week. (ydary, 14:51:48) * Virt 3.6.0 Status Update: V2V tool and passthrough devices work in progress. VM instance types images merged. (ydary, 14:56:59) * SLA 3.6.0 Status Update: No one attended from the team. (ydary, 14:57:42) * Network 3.6.0 Status Update: No one attended from the team. (ydary, 14:57:56) * conferences and workshops (ydary, 14:58:35) * Planning for KVM Forum is up and running. KVM- and oVirt-related talks should be submitted at http://events.linuxfoundation.org/events/kvm-forum/program/cfp before May 1. (ydary, 14:59:14) * There has been some discussion on the length of the oVirt Workshop track in this year's KVM Forum. Right now, it will be at least a full day, but
[ovirt-users] Hosted-Engine misc configuration
Hi everyone, i'm having this error during the deployment of VM for engine. im using ovirt 3.5 repo and freenas 9.3 as my NFS Server. vdsm-4.16.10-8.gitc937927.el6.x86_64 libvirt-0.10.2-46.el6_6.3.x86_64 sanlock-2.8-1.el6.x86_64 [ INFO ] Stage: Setup validation [WARNING] Host name ovirt1 has no domain suffix --== CONFIGURATION PREVIEW ==-- Engine FQDN : engine Bridge name : ovirtmgmt SSH daemon port : 22 Gateway address : 192.168.206.1 Host name for web application : hosted_engine_1 Host ID : 1 Image alias : hosted_engine Image size GB : 40 Storage connection : storage:/mnt/STORAGE-DOMAIN/OVIRT-ENGINE Console type : vnc Memory size MB : 4096 MAC address : 00:16:3e:4e:8c:cb Boot type : cdrom Number of CPUs : 2 ISO image (for cdrom boot) : /CentOS-6.6-x86_64-minimal.iso CPU Type : model_Penryn Please confirm installation settings (Yes, No)[Yes]: [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Configuring libvirt [ INFO ] Configuring VDSM [ INFO ] Starting vdsmd [ INFO ] Waiting for VDSM hardware info [ INFO ] Waiting for VDSM hardware info [ INFO ] Waiting for VDSM hardware info [ INFO ] Connected to Storage Domain [ INFO ] Connecting Storage Pool [ ERROR ] Failed to execute stage 'Misc configuration': Not SPM [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20150326181348.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination Thanks, Jaicel ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Hosted-Engine misc configuration
- Original Message - From: Jaicel jai...@asti.dost.gov.ph To: users@ovirt.org Sent: Thursday, March 26, 2015 12:22:53 PM Subject: [ovirt-users] Hosted-Engine misc configuration Hi everyone, i'm having this error during the deployment of VM for engine. im using ovirt 3.5 repo and freenas 9.3 as my NFS Server. Please post somewhere the setup log, from /var/log/ovirt-hosted-engine-setup. Thanks. -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VDSM memory consumption
On Wed, Mar 25, 2015 at 01:29:25PM -0500, Darrell Budic wrote:
On Mar 25, 2015, at 5:34 AM, Dan Kenigsberg dan...@redhat.com wrote:
On Tue, Mar 24, 2015 at 02:01:40PM -0500, Darrell Budic wrote:
On Mar 24, 2015, at 4:33 AM, Dan Kenigsberg dan...@redhat.com wrote:
On Mon, Mar 23, 2015 at 04:00:14PM -0400, John Taylor wrote:
Chris Adams c...@cmadams.net writes:
Once upon a time, Sven Kieske s.kie...@mittwald.de said:
On 13/03/15 12:29, Kapetanakis Giannis wrote:
We also face this problem since 3.5 in two different installations...
Hope it's fixed soon
Nothing will get fixed if no one bothers to
open BZs and send relevants log files to help
track down the problems.
There's already an open BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1158108
I'm not sure if that is exactly the same problem I'm seeing or not; my
vdsm process seems to be growing faster (RSS grew 952K in a 5 minute
period just now; VSZ didn't change).
For those following this I've added a comment on the bz [1], although in
my case the memory leak is, like Chris Adams, a lot more than the
300KiB/h
in the original bug report by Daniel Helgenberger .
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108
That's interesting (and worrying).
Could you check your suggestion by editing sampling.py so that
_get_interfaces_and_samples() returns the empty dict immediately?
Would this make the leak disappear?
Looks like you’ve got something there. Just a quick test for now, watching
RSS in top. I’ll let it go this way for a while and see what it looks in a
few hours.
System 1: 13 VMs w/ 24 interfaces between them
11:47 killed a vdsm @ 9.116G RSS (after maybe a week and a half running)
11:47: 97xxx
11:57 135544 and climbing
12:00 136400
restarted with sampling.py modified to just return empty set:
def _get_interfaces_and_samples():
links_and_samples = {}
return links_and_samples
Thanks for the input. Just to be a little more certain that the culprit
is _get_interfaces_and_samples() per se, would you please decorate it
with memoized, and add a log line in the end
@utils.memoized # add this line
def _get_interfaces_and_samples():
...
logging.debug('LINKS %s', links_and_samples) ## and this line
return links_and_samples
I'd like to see what happens when the function is run only once, and
returns a non-empty reasonable dictionary of links and samples.
Looks similar, I modified my second server for this test:
Thanks again. Would you be kind to search further?
Does the following script leak anything on your host, when placed in your
/usr/share/vdsm:
#!/usr/bin/python
from time import sleep
from virt.sampling import _get_interfaces_and_samples
while True:
_get_interfaces_and_samples()
sleep(0.2)
Something that can be a bit harder would be to:
# service vdsmd stop
# su - vdsm -s /bin/bash
# cd /usr/share/vdsm
# valgrind --leak-check=full --log-file=/tmp/your.log vdsm
as suggested by Thomas on
https://bugzilla.redhat.com/show_bug.cgi?id=1158108#c6
Regards,
Dan.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VDSM memory consumption
Hello Everyone, I did create the original BZ on this. In the mean time, lab system I used is dismantled and the production system is yet to deploy. As I wrote in BZ1147148 [1], I experienced two different issues. One, one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem unrelated. The larger leak was indeed related to SSL in some way; not necessarily M2Crypto. However, after disabling SSL this was gone leaving the smaller leak. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 On Mo, 2015-03-09 at 23:49 +0100, Matt . wrote: Hi, I also see this on the latest 3.5 version, I'm thinking about setting up a cronjob to restart vdsm every night. I did the same thing. In general, it seems to be a bad idea as it compromised system stability on the long run. While VMs seem to be fine, engine does not like this very much. I cannot believe that people say they don't have this issue. This was hard for me to accept as well. I know of Markus Stockhausen and Seven Kieske, both confirmed the small leak. This might also be some special other service; though I started out with a minimal install of Centos 6. Can someone of the devs dive in maybe ? Thanks! Matt 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg dan...@redhat.com: On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg dan...@redhat.com wrote: On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. https://bugzilla.redhat.com/show_bug.cgi?id=1158108 On Mar 6, 2015, at 10:23 AM, Chris Adams c...@cmadams.net wrote: Once upon a time, Federico Alberto Sayd fs...@uncu.edu.ar said: I am experiencing troubles with VDSM memory consuption. I am running Engine: ovirt 3.5.1 Nodes: Centos 6.6 VDSM 4.16.10-8 Libvirt: libvirt-0.10.2-46 Kernel: 2.6.32 When the host boots, memory consuption is normal, but after 2 or 3 days running, VDSM memory consuption grows and it consumes more memory that all vm's running in the host. If I restart the vdsm service, memory consuption normalizes, but then it start growing again. I have seen some BZ about vdsm and supervdsm about memory leaks, but I don't know if VDSM 4.6.10.8 is still affected by a related bug. Can't help, but I see the same thing with CentOS 7 nodes and the same version of vdsm. -- Chris Adams c...@cmadams.net ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I'm afraid that we are yet to find a solution for this issue, which is completly different from the horrible leak of supervdsm 4.16.7. Could you corroborate the claim of Bug 1147148 - M2Crypto usage in vdsm leaks memory ? Does the leak disappear once you start using plaintext transport? Regards, Dan. I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). This is from #ovirt around November 18th I think, Saggi thought he’d found something related: 9:58:43 AM saggi: YamakasY: Found the leak 9:58:48 AM saggi: YamakasY: Or at least the flow 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce 9:59:20 AM YamakasY: saggi: that's kewl! 9:59:25 AM YamakasY: saggi: what happens ? 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) djasa left the room (quit: Quit: Leaving). (10:00:24 AM) mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png I do recall what is the issue Saggi and YamakasY were dicussing (CCing the pair), or if it reached fruition as a patch. It is certainly something other than Bug 1158108, as the latter speak about a leak in a normal working state, with no getCapabilities calls. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Add network interface to self hosted engine
- Original Message - From: Jason Keltz j...@cse.yorku.ca To: users@ovirt.org Sent: Thursday, March 26, 2015 1:30:50 PM Subject: [ovirt-users] Add network interface to self hosted engine Hi. Is it possible to add a network interface from the node to self hosted engine after it is setup? Currently you cannot edit the engine VM form the engine itself but we are working to make you able to do that in 3.6. Currently the only way to do that is manually editing /etc/ovirt-hosted-engine/vm.conf on each involved host. Jason. Sent with AquaMail for Android http://www.aqua-mail.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Add network interface to self hosted engine
Hi. Is it possible to add a network interface from the node to self hosted engine after it is setup? Jason. Sent with AquaMail for Android http://www.aqua-mail.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VDSM memory consumption
Hi Daniel, Great! Thanks. I only see this issue happening on CentOS 7, Joop van de Wege also confirmed he didn't see it on CentOS 6. Cheers, Matt 2015-03-26 13:33 GMT+01:00 Daniel Helgenberger daniel.helgenber...@m-box.de: Hello Everyone, I did create the original BZ on this. In the mean time, lab system I used is dismantled and the production system is yet to deploy. As I wrote in BZ1147148 [1], I experienced two different issues. One, one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem unrelated. The larger leak was indeed related to SSL in some way; not necessarily M2Crypto. However, after disabling SSL this was gone leaving the smaller leak. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 On Mo, 2015-03-09 at 23:49 +0100, Matt . wrote: Hi, I also see this on the latest 3.5 version, I'm thinking about setting up a cronjob to restart vdsm every night. I did the same thing. In general, it seems to be a bad idea as it compromised system stability on the long run. While VMs seem to be fine, engine does not like this very much. I cannot believe that people say they don't have this issue. This was hard for me to accept as well. I know of Markus Stockhausen and Seven Kieske, both confirmed the small leak. This might also be some special other service; though I started out with a minimal install of Centos 6. Can someone of the devs dive in maybe ? Thanks! Matt 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg dan...@redhat.com: On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg dan...@redhat.com wrote: On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. https://bugzilla.redhat.com/show_bug.cgi?id=1158108 On Mar 6, 2015, at 10:23 AM, Chris Adams c...@cmadams.net wrote: Once upon a time, Federico Alberto Sayd fs...@uncu.edu.ar said: I am experiencing troubles with VDSM memory consuption. I am running Engine: ovirt 3.5.1 Nodes: Centos 6.6 VDSM 4.16.10-8 Libvirt: libvirt-0.10.2-46 Kernel: 2.6.32 When the host boots, memory consuption is normal, but after 2 or 3 days running, VDSM memory consuption grows and it consumes more memory that all vm's running in the host. If I restart the vdsm service, memory consuption normalizes, but then it start growing again. I have seen some BZ about vdsm and supervdsm about memory leaks, but I don't know if VDSM 4.6.10.8 is still affected by a related bug. Can't help, but I see the same thing with CentOS 7 nodes and the same version of vdsm. -- Chris Adams c...@cmadams.net ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I'm afraid that we are yet to find a solution for this issue, which is completly different from the horrible leak of supervdsm 4.16.7. Could you corroborate the claim of Bug 1147148 - M2Crypto usage in vdsm leaks memory ? Does the leak disappear once you start using plaintext transport? Regards, Dan. I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). This is from #ovirt around November 18th I think, Saggi thought he’d found something related: 9:58:43 AM saggi: YamakasY: Found the leak 9:58:48 AM saggi: YamakasY: Or at least the flow 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce 9:59:20 AM YamakasY: saggi: that's kewl! 9:59:25 AM YamakasY: saggi: what happens ? 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) djasa left the room (quit: Quit: Leaving). (10:00:24 AM) mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png I do recall what is the issue Saggi and YamakasY were dicussing (CCing the pair), or if it reached fruition as a patch. It is certainly something other than Bug 1158108, as the latter speak about a leak in a normal working state, with no getCapabilities calls. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht
Re: [ovirt-users] VDSM memory consumption
On 26/03/15 09:43, Matt . wrote: Hi Daniel, Great! Thanks. I only see this issue happening on CentOS 7, Joop van de Wege also confirmed he didn't see it on CentOS 6. Cheers, Matt I have experienced the same issue on Centos 6.6 and Centos 7 both managed by the same engine. Cheers Federico 2015-03-26 13:33 GMT+01:00 Daniel Helgenberger daniel.helgenber...@m-box.de: Hello Everyone, I did create the original BZ on this. In the mean time, lab system I used is dismantled and the production system is yet to deploy. As I wrote in BZ1147148 [1], I experienced two different issues. One, one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem unrelated. The larger leak was indeed related to SSL in some way; not necessarily M2Crypto. However, after disabling SSL this was gone leaving the smaller leak. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 On Mo, 2015-03-09 at 23:49 +0100, Matt . wrote: Hi, I also see this on the latest 3.5 version, I'm thinking about setting up a cronjob to restart vdsm every night. I did the same thing. In general, it seems to be a bad idea as it compromised system stability on the long run. While VMs seem to be fine, engine does not like this very much. I cannot believe that people say they don't have this issue. This was hard for me to accept as well. I know of Markus Stockhausen and Seven Kieske, both confirmed the small leak. This might also be some special other service; though I started out with a minimal install of Centos 6. Can someone of the devs dive in maybe ? Thanks! Matt 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg dan...@redhat.com: On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg dan...@redhat.com wrote: On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. https://bugzilla.redhat.com/show_bug.cgi?id=1158108 On Mar 6, 2015, at 10:23 AM, Chris Adams c...@cmadams.net wrote: Once upon a time, Federico Alberto Sayd fs...@uncu.edu.ar said: I am experiencing troubles with VDSM memory consuption. I am running Engine: ovirt 3.5.1 Nodes: Centos 6.6 VDSM 4.16.10-8 Libvirt: libvirt-0.10.2-46 Kernel: 2.6.32 When the host boots, memory consuption is normal, but after 2 or 3 days running, VDSM memory consuption grows and it consumes more memory that all vm's running in the host. If I restart the vdsm service, memory consuption normalizes, but then it start growing again. I have seen some BZ about vdsm and supervdsm about memory leaks, but I don't know if VDSM 4.6.10.8 is still affected by a related bug. Can't help, but I see the same thing with CentOS 7 nodes and the same version of vdsm. -- Chris Adams c...@cmadams.net ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I'm afraid that we are yet to find a solution for this issue, which is completly different from the horrible leak of supervdsm 4.16.7. Could you corroborate the claim of Bug 1147148 - M2Crypto usage in vdsm leaks memory ? Does the leak disappear once you start using plaintext transport? Regards, Dan. I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). This is from #ovirt around November 18th I think, Saggi thought he’d found something related: 9:58:43 AM saggi: YamakasY: Found the leak 9:58:48 AM saggi: YamakasY: Or at least the flow 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce 9:59:20 AM YamakasY: saggi: that's kewl! 9:59:25 AM YamakasY: saggi: what happens ? 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) djasa left the room (quit: Quit: Leaving). (10:00:24 AM) mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png I do recall what is the issue Saggi and YamakasY were dicussing (CCing the pair), or if it reached fruition as a patch. It is certainly something other than Bug 1158108, as the latter speak about a leak in a normal working state, with no getCapabilities calls. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ___ Users
Re: [ovirt-users] VDSM memory consumption
On Mar 26, 2015, at 6:42 AM, Dan Kenigsberg dan...@redhat.com wrote:
On Wed, Mar 25, 2015 at 01:29:25PM -0500, Darrell Budic wrote:
On Mar 25, 2015, at 5:34 AM, Dan Kenigsberg dan...@redhat.com wrote:
On Tue, Mar 24, 2015 at 02:01:40PM -0500, Darrell Budic wrote:
On Mar 24, 2015, at 4:33 AM, Dan Kenigsberg dan...@redhat.com wrote:
On Mon, Mar 23, 2015 at 04:00:14PM -0400, John Taylor wrote:
Chris Adams c...@cmadams.net writes:
Once upon a time, Sven Kieske s.kie...@mittwald.de said:
On 13/03/15 12:29, Kapetanakis Giannis wrote:
We also face this problem since 3.5 in two different installations...
Hope it's fixed soon
Nothing will get fixed if no one bothers to
open BZs and send relevants log files to help
track down the problems.
There's already an open BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1158108
I'm not sure if that is exactly the same problem I'm seeing or not; my
vdsm process seems to be growing faster (RSS grew 952K in a 5 minute
period just now; VSZ didn't change).
For those following this I've added a comment on the bz [1], although in
my case the memory leak is, like Chris Adams, a lot more than the
300KiB/h
in the original bug report by Daniel Helgenberger .
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108
That's interesting (and worrying).
Could you check your suggestion by editing sampling.py so that
_get_interfaces_and_samples() returns the empty dict immediately?
Would this make the leak disappear?
Looks like you’ve got something there. Just a quick test for now, watching
RSS in top. I’ll let it go this way for a while and see what it looks in a
few hours.
System 1: 13 VMs w/ 24 interfaces between them
11:47 killed a vdsm @ 9.116G RSS (after maybe a week and a half running)
11:47: 97xxx
11:57 135544 and climbing
12:00 136400
restarted with sampling.py modified to just return empty set:
def _get_interfaces_and_samples():
links_and_samples = {}
return links_and_samples
Thanks for the input. Just to be a little more certain that the culprit
is _get_interfaces_and_samples() per se, would you please decorate it
with memoized, and add a log line in the end
@utils.memoized # add this line
def _get_interfaces_and_samples():
...
logging.debug('LINKS %s', links_and_samples) ## and this line
return links_and_samples
I'd like to see what happens when the function is run only once, and
returns a non-empty reasonable dictionary of links and samples.
Looks similar, I modified my second server for this test:
Thanks again. Would you be kind to search further?
Does the following script leak anything on your host, when placed in your
/usr/share/vdsm:
#!/usr/bin/python
from time import sleep
from virt.sampling import _get_interfaces_and_samples
while True:
_get_interfaces_and_samples()
sleep(0.2)
Something that can be a bit harder would be to:
# service vdsmd stop
# su - vdsm -s /bin/bash
# cd /usr/share/vdsm
# valgrind --leak-check=full --log-file=/tmp/your.log vdsm
as suggested by Thomas on
https://bugzilla.redhat.com/show_bug.cgi?id=1158108#c6
Yes, this script leaks quickly. Started out at a RSS of 21000ish, already at
26744 a minute in, about 5 minutes later it’s at 39384 and climbing.
Been abusing a production server for those simple tests, but didn’t want to run
valgrind against it right this minute. Did run it against the test.py script
above though, got this (fpaste.org didn’t like, too long maybe?):
http://tower.onholyground.com/valgrind-test.log
To comment on some other posts in this thread, I also see leaks on my test
system which is running Centos 6.6, but it only has 3 VMs across 2 servers and
3 configured networks and it leaks MUCH slower. I suspect people don’t notice
this on test systems because they don’t have a lot of VMs/interfaces running,
and don’t leave them up for weeks at a time. That’s why I was running these
tests on my production box, to have more VMs up.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Windows sysprep format
I have made a sealed Win 2012 template and resolved the issues with quotes around unattend.xml. Now I need help/examples on how to run a powershell script from the blank sysprep field. Any formatting info or examples would be helpful. Thank you, Tim Macy ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] [moVirt] Failed to open console client.
I just started playing with moVirt after I first heard about it here on this list. Wow, that's so much easier to use on my Galaxy S5 than the regular web interface. :-) Of course, now I want the whole bag of chips but can't seem to get the console working. When I press the console button I see Failed to open console client. Check if aSPICE/bVNC is installed.. I had already installed aSPICE from the Play Store but when that didn't work installed bVNC in addition; still no luck. My engine.log shows this when I try: 2015-03-26 19:52:14,750 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-10) [7f6f1f09] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 6ca4e6ae-a201-432b-b22a-2c6517fbd92c Type: VMAction group CONNECT_TO_VM with role type USER 2015-03-26 19:52:14,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-10) [7f6f1f09] START, SetVmTicketVDSCommand(HostName = hosted_engine_1, HostId = e90e2ca0-e3b9-46d5-8fde-09fc1c5eed20, vmId=6ca4e6ae-a201-432b-b22a-2c6517fbd92c, ticket=pd2lTiu7w7Ra, validTime=7200,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6257f73 2015-03-26 19:52:14,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-10) [7f6f1f09] FINISH, SetVmTicketVDSCommand, log id: 6257f73 2015-03-26 19:52:14,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) [7f6f1f09] Correlation ID: 7f6f1f09, Call Stack: null, Custom Event ID: -1, Message: user admin@internal initiated console session for VM krypto_f21 What am doing wrong? ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Error during CanDoActionFailure.: java.lang.NullPointerException
Hi all, For backup purposes, I have created a python script which will create a snaphot (disk only) from the specified virtual machine, clone it, export the clone to export disk and afterwards remove both the clone and snapshot (we are using F20 with virt-preview enabled). Unfortunately our only Windows virtual machine (Windows Server 2012) is unable to create a clone from the just created snapshot. When trying to do this manually, the following message is visible within the oVirt GUI, when clicking on the Clone button: General command validation failure I have attached the error which is visible whitin the engine log. Directly after the lock is acquired a CanDoActionFailure is visible. I am going to try and do the same when the VM is powered down (unfortunately I cannot just turn this machine off at any time), but I'm wondering if somebody has experienced this before with a Windows machine; as this works perfectly with all our Linux virtual machines Many thanks, Roel ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (ajp--127.0.0.1-8702-8) [3c30897a] Error during CanDoActionFailure.: java.lang.NullPointerException at org.ovirt.engine.core.bll.AddVmFromSnapshotCommand.getDestintationDomainTypeFromDisk(AddVmFromSnapshotCommand.java:113) [bll.jar:] at org.ovirt.engine.core.bll.AddVmFromSnapshotCommand.adjustDisksImageConfiguration(AddVmFromSnapshotCommand.java:105) [bll.jar:] at org.ovirt.engine.core.bll.AddVmFromSnapshotCommand.getAdjustedDiskImagesFromConfiguration(AddVmFromSnapshotCommand.java:97) [bll.jar:] at org.ovirt.engine.core.bll.AddVmFromSnapshotCommand.canDoAction(AddVmFromSnapshotCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:768) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:347) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:369) [bll.jar:] at sun.reflect.GeneratedMethodAccessor319.invoke(Unknown Source) [:1.7.0_75] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_75] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_75] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source) [:1.7.0_75] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_75] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_75] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at
