Hi, 2012/8/26 Ruben S. Montero <rsmont...@opennebula.org>
> Hi > > If you want to try the cpu pinning suggested by Steven, simply add a > RAW attribute in your VM template. Something similar to: > > RAW=[ > TYPE="kvm", > DATA="<cputune><vcpupin vcpu=\"0\" cpuset=\"1\"/></cputune>" ] > Added to vm template. Running an expirement right now to observe availability. > > Cheers > > Ruben > > > On Sun, Aug 26, 2012 at 3:27 AM, Steven C Timm <t...@fnal.gov> wrote: > > I run high-availability squid servers on virtual machines although not > yet > > in OpenNebula. > > > > It can be done with very high availability. > > > > I am not familiar with Ubuntu Server 12.04 but if it has libvirt 0.9.7 or > > better, and you are > > > > Using KVM hypervisor, you should be able to use the cpu-pinning and > > numa-aware features of libvirt to pin > > > > > each virtual machine to a given physical cpu. That will beat the > migration > > issue you are seeing now. > Done. migration processes was beaten, as you sad. Thank you. > > > > With Xen hypervisor you can (and should) also pin. > > > > I think if you beat the cpu and memory pinning problem you will be OK. > > > > > > > > However, you did not say what network topology you are using for your > > virtual machine, > That is my vnet template: TYPE = RANGED BRIDGE = br0 NETWORK_SIZE = C NETWORK_ADDRESS = 192.168.15.0 NETMASK = 255.255.255.0 GATEWAY = 192.168.15.10 IP_START = 192.168.15.110 IP_END = 192.168.15.190 futhermore, there is one squid instance in execution on each VM, connected on a mesh architecture, with LRU replacement policy. Web-polygraph clients are configured to send requests on round-robin, to each of VMs (only three currently: 192.168.15.110-112). Web-polygraph server, that represents Internet, is running on 192.168.15.10, closing the topology. Remember that when same test is performed on Physical Machines, availability is 100% invariably. and what kind of virtual network drivers, > VMs are configured on VirtualBox, based on Ubuntu Server 12.04. Executing: ps ax | grep kvm on 192.168.15.110 cloud node: /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name one-609 -uuid a7115950-3f0e-0fab-ea20-3f829d835889 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-609.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -no-acpi -boot c -drive file=/srv3/cloud/one/var/datastores/0/609/disk.0,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/srv3/cloud/one/var/datastores/0/609/disk.1,if=none,id=drive-ide0-1-1,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=20,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:00:c0:a8:0f:70,bus=pci.0,addr=0x3 -usb -vnc 0.0.0.0:609 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 network driver used on VMs is rtl8139. > > > > > That is important too. Also—is your squid cache mostly disk-resident > or > > mostly RAM-resident? > disk-resident (from squid.conf): cache_mem 128 MB cache_dir aufs /var/spool/squid3 1024 16 256 Note that disk space is reduced to 1024MB for testing purposes. > If the former then the virtual disk drivers matter > > too, a lot. > Some reading suggestions? I'll appreciate a lot. One additional information that can be relevant. Monitoring VMs processes on D status, through command: ps -eo pid,tid,class,rtprio,psr,pcpu,stat,wchan:25,comm | grep D PID TID CLS RTPRIO PSR %CPU STAT WCHAN COMMAND 170 170 TS - 0 0.1 D sleep_on_page jbd2/sda1-8 1645 1645 TS - 0 11.4 Dsl get_write_access squid3 1647 1647 TS - 0 0.2 D get_write_access autExperiment.sh 1679 1679 TS - 0 0.8 D get_write_access autExperiment.sh 1680 1680 TS - 0 0.5 D get_write_access autExperiment.sh It reveals that journally ext4 process(jdb2/sda1-8), squid and my monitoring scripting are on D status, always waiting on sleep_on_page and get_write_access kernel functions. I am researching the relevance/influence of these WCHANs. I performed tests without my monitoring scripts and availability is as bad as with them. Thanks for Steve and Ruben, for your time and answers. Erico. > > > > > > > > Steve Timm > > > > > > > > > > > > > > > > From: users-boun...@lists.opennebula.org > > [mailto:users-boun...@lists.opennebula.org] On Behalf Of Erico Augusto > > Cavalcanti Guedes > > Sent: Saturday, August 25, 2012 6:33 PM > > To: users@lists.opennebula.org > > Subject: [one-users] Very high unavailable service > > > > > > > > Dears, > > > > I 'm running Squid Web Cache Proxy server on Ubuntu Server 12.04 VMs > (kernel > > 3.2.0-23-generic-pae), OpenNebula 3.4. > > My private cloud is composed by one frontend and three nodes. VMs are > > running on that 3 nodes, initially one by node. > > Outside cloud, there are 2 hosts, one working as web clients and another > as > > web server, using Web Polygraph Benchmakring Tool. > > > > The goal of tests is stress Squid cache running on VMs. > > When same test is executed outside the cloud, using the three nodes as > > Physical Machines, there are 100% of cache service availability. > > Nevertheless, when cache service is provided by VMs, nothing better than > 45% > > of service availability is reached. > > Web clients do not receive responses from squid when it is running on > VMs in > > 55% of the time. > > > > I have monitored load average of VMs and PMs where VMs are been executed. > > First load average field reaches 15 after some hours of tests on VMs, > and 3 > > on physical machines. > > Furthermore, there is a set of processes, called migration/X, that are > > champions in CPU TIME when VMs are in execution. A sample: > > > > top - 20:01:38 up 1 day, 3:36, 1 user, load average: 5.50, 5.47, 4.20 > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ TIME > COMMAND > > 13 root RT 0 0 0 0 S 0 0.0 408:27.25 408:27 > > migration/2 > > 8 root RT 0 0 0 0 S 0 0.0 404:13.63 404:13 > > migration/1 > > 6 root RT 0 0 0 0 S 0 0.0 401:36.78 401:36 > > migration/0 > > 17 root RT 0 0 0 0 S 0 0.0 400:59.10 400:59 > > migration/3 > > > > > > It isn't possible to offer web cache service via VMs in the way the > service > > is behaving, with so small availability. > > > > So, my questions: > > > > 1. Does anybody has experienced a similar problem of unresponsive > service? > > (Whatever service). > > 2. How to state the bootleneck that is overloading the system, so that it > > can be minimized? > > > > Thanks a lot, > > > > Erico. > > > > > > _______________________________________________ > > Users mailing list > > Users@lists.opennebula.org > > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > > > > > > -- > Ruben S. Montero, PhD > Project co-Lead and Chief Architect > OpenNebula - The Open Source Solution for Data Center Virtualization > www.OpenNebula.org | rsmont...@opennebula.org | @OpenNebula >
_______________________________________________ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
