[pve-devel] kernel BUG at ./include/linux/swapops.h:129!

[Eric Blevins]

I believe 4.10.8-6 is affected by this bug:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1674838

My server just experienced this problem, had to power cycle to recover.
If there is any info I can provide that might help fix this let me
know but I mostly mailed pve-devel to make you aware of this upstream
bug.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] pve-kernel-4.4.10-1-pve woes

[Eric Blevins]

Might not be your problem but there is a known issue with IPoIB and NFS
resulting in a deadlock.


Originally only the mlx4 driver was fixed:
https://lkml.org/lkml/2014/5/11/50
I noticed qib was patched earlier this year:
http://comments.gmane.org/gmane.linux.drivers.rdma/32914



On Mon, Jul 4, 2016 at 1:39 PM, Michael Rasmussen  wrote:

> On Mon, 4 Jul 2016 07:42:51 +0200
> Fabian Grünbichler  wrote:
>
> >
> > could you give the 4.4.13 kernel from pve-no-subscription a try? More
> I have installed 4.4.13 on a node.
> 1) Created a Debian Jessie container and a Ubuntu-15-10 container on a
> IB NFS share
> 2) Installed bonnie++ on both
> 3) Run bonnie++ -d /tmp -r2048 -u mir on both in parallel
> 4) Run some backup to same NFS share and another NFS share over bonded
> igb. Both sequential and in parallel
>
> No nic freeze and nothing logged to syslog on the node what so ever;-)
>
> I will keep this node running on this kernel for a couple of days to see
> whether the nic should freeze or other NFS related issue before I will
> make my final conclusion.
>
> I will keep you posted.
>
> --
> Hilsen/Regards
> Michael Rasmussen
>
> Get my public GnuPG keys:
> michael  rasmussen  cc
> http://pgp.mit.edu:11371/pks/lookup?op=get=0xD3C9A00E
> mir  datanom  net
> http://pgp.mit.edu:11371/pks/lookup?op=get=0xE501F51C
> mir  miras  org
> http://pgp.mit.edu:11371/pks/lookup?op=get=0xE3E80917
> --
> /usr/games/fortune -es says:
> The real reason large families benefit society is because at least
> a few of the children in the world shouldn't be raised by beginners.
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] Running KVM as root is a security issue

[Eric Blevins]

I have no idea if CVE-2015-5154 that Stephan inquired about affests Proxmox.

But when I see exploits like that the first thought in my mind is how
easy it would be for such an exploit to get root on the Proxmox host.

I've done some experimenting. If I take the KVM command as generated
by Proxmox and simply add -runas nobody the VM starts up and runs
without a problem.

However when I try to open a console the KVM process fails.
I suspect this is just some permissions in creating the socket but not
investidated.

A patch exists to prevent a crash when a socket cannot be opened.
https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg00577.html

Any chance this security issue can be fixed before the 4.0 release?

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Running KVM as root is a security issue

[Eric Blevins]

Having only PCI passthrough VMs running as root would be a huge improvement.
Maybe cgroups could be used to reduce the risk.

Exit scripts could be suid if needed.
An exploted VM could potentially use the suid pve-bridgedown script to
destroy bridges of other VMs.

Long term I think a better idea is needed.

The exit scripts could simply notify some other privlidged process
that they are shutting down.
Privlidged process would then verify that VM is down and do whatever
cleanup is necessary.




On Mon, Jul 27, 2015 at 10:07 AM, Alexandre DERUMIER
aderum...@odiso.com wrote:
Yes, that much I've tested, too. I'm worried about the shutdown scripts
though (bridgedown). They might lack permissions if qemu doesn't keep a
privileged parent process around for those.

 I think that pci passthrough need root access too. (maybe not with vfio).

 Not sure about disks with /dev/ mapping ?



 - Mail original -
 De: Wolfgang Bumiller w.bumil...@proxmox.com
 À: Eric Blevins ericlb...@gmail.com
 Cc: pve-devel pve-devel@pve.proxmox.com
 Envoyé: Lundi 27 Juillet 2015 15:53:00
 Objet: Re: [pve-devel] Running KVM as root is a security issue

 A patch exists to prevent a crash when a socket cannot be opened.
 https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg00577.html

 Included in the current 2.4 devel build.

 I've done some experimenting. If I take the KVM command as generated
 by Proxmox and simply add -runas nobody the VM starts up and runs
 without a problem.

 Yes, that much I've tested, too. I'm worried about the shutdown scripts
 though (bridgedown). They might lack permissions if qemu doesn't keep a
 privileged parent process around for those.

 Ideally the VM can be started directly as a user, though, rather than
 using the -runas switch. That will be some work though.

 ___
 pve-devel mailing list
 pve-devel@pve.proxmox.com
 http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Running KVM as root is a security issue

[Eric Blevins]

 I guess it would also help if we add a reasonable apparmor profile?


apparmor profiles would be greatly appreciated
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] backup : add pigz compressor

[Eric Blevins]

 But pigz needs all you CPU power do be that fast! If you have running VMs,
 it will be as slow as normal gzip, or it will massively slow down the VMs.
 I am really afraid this will trigger many support calls ...

Make the default number of pigz CPUs 2, this should not have much of a
CPU impact even on servers with small number of cores.
Those of us who have dozens of cores can edit vzdump.conf and set an
option to use more cores for pigz.

You could even make it so using pigz requires a setting in
vzdump.conf. So in GUI you still can only select gzip or lzop.
If set to gzip and vzdump.conf has pigz:on then pigz is used instead of gzip.
Most novice users are only going to use the GUI, this would reduce the
likelyhood of them turning on pigz and then complaiing about their
decision.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Default cache mode for VM hard drives

[Eric Blevins]

Suggestion for another test case.

Create a linux VM that has tons of swap and just enough RAM to boot.
Run a single threaded program that allocates and modifies more RAM
than the VM has ensuring many of the modifications are to swap.
This should result in out of sync blocks in the swap area.

Test case inspired by Linbit:
http://lists.linbit.com/pipermail/drbd-user/2008-May/009282.html

We already know that the swap code has similar behavior. In case
a page gets touched while it is under write out IO, the swap allows
the modification to the page although it is under IO by the block
layer. Therefore the swap code does not know which version actually
got written to disk, but it does not care, since it knows that it
has the up to date version in core memory.





On Fri, May 29, 2015 at 9:25 AM, Dietmar Maurer diet...@proxmox.com wrote:
  1) On any guest OS (Linux or Windows) where swap is used actively

 So maybe the bug can be triggered using mmap? Unfortunately, I have no real 
 idea
 how mmap work internally, but I can imagine that it produces a similar 
 pattern?
 Any mmap expert here?

 ___
 pve-devel mailing list
 pve-devel@pve.proxmox.com
 http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] CVE-2015-3456

[Eric Blevins]

Is Proxmox vulnerable to CVE-2015-3456?

https://securityblog.redhat.com/tag/cve-2015-3456/
From the article:
It can result in guest controlled execution of arbitrary code in, and
with privileges of, the corresponding QEMU process on the host. Worst
case scenario this can be guest to host exit with the root privileges.


Can we expect Proxmox to stop running KVM processes as root in the near future?
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] Stop running KVM as root - bug 547

[Eric Blevins]

This bug has been sitting around for a year, any chance it can be
bumped up in priority?

https://bugzilla.proxmox.com/show_bug.cgi?id=547
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] 3.4: Changes in the way quorum is formed?

[Eric Blevins]

 On Fri, 20 Feb 2015 00:29:36 +0100
 Michael Rasmussen m...@datanom.net wrote:

 I am giving up for now and have reverted back to
 pve-kernel-2.6.32-34-pve.

Doing the Same


 I forgot to mention that my conclusion is that the infiniband drivers
 delivered with pve-kernel-2.6.32-37-pve is broken.

I can confirm this, IPOIB multicast seems to only work intermittently
with pve-kernel-2.6.32-37-pve

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] iothread support

[Eric Blevins]

Any plans to support iothreads in the future?

It makes a significant difference in IOPS on my windows 2012R2 guest.

cache=none, virtio, discard enabled:
Random Read 4KB (QD=32) :   190.467 MB/s [ 46500.7 IOPS]
Random Write 4KB (QD=32) :   170.091 MB/s [ 41526.2 IOPS]

iothread, cache=none, discard enabled:
Random Read 4KB (QD=32) :   349.899 MB/s [ 85424.6 IOPS] +38923.9
Random Write 4KB (QD=32) :   221.866 MB/s [ 54166.6 IOPS] +12640.4

To enable iothreads I added this to my vm.conf:
args: -object iothread,id=iothread0 -drive
file=/dev/drbd2-vm7-vm8/vm-109-disk-5,if=none,id=drive-virtio4,cache=none,discard=on,aio=native,detect-zeroes=unmap
-device 
virtio-blk-pci,drive=drive-virtio4,id=virtio4,bus=pci.0,addr=0xe,iothread=iothread0

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Default cache mode for VM hard drives

[Eric Blevins]

I have been looking into this issue too.

cache=none and cache=directsync does not prevent the out of sync blocks on DRBD.

I have seen this on windows and linux guests.
The latest windows virtio driver from fedora should have fix to
implement flushes properly, need to test if it helps.

https://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/

Eric

On Thu, Dec 4, 2014 at 11:48 AM, Stanislav German-Evtushenko
ginerm...@gmail.com wrote:
 On Thu, Dec 4, 2014 at 7:42 PM, Dietmar Maurer diet...@proxmox.com wrote:

   Hi Dietmar, hi Alexandre,
   I did manage to reproduce the issue without hardware dependency and
  without
   LVM to reduce any possible influence from other components.
 
  IMHO this is a DRBD problem. So I think you should report that to the
  DRBD list
  instead.
 
  I did and I have mentioned that.

 You already reported a reproducible test case?


 No, but he didn't want this.
 He said:
 ***
 Misbehaving upper layer results in potentially divergent blocks on the DRBD
 peers. Or would result in potentially divergent blocks on a local software
 RAID 1.
 ...
 Anyways.
 Point being: Either have those upper layers stop modifying buffers
 while they are in-flight (keyword: stable pages).
 Kernel upgrade within the VMs may do it. Changing something in the
 virtual IO path configuration may do it. Or not.

 Or live with the results, which are
 potentially not identical blocks on the DRBD peers.
 ***

 Stanislav

 ___
 pve-devel mailing list
 pve-devel@pve.proxmox.com
 http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Task List Migration

[Eric Blevins]

Migrate the VM then migrate the logs for that VM.
Or use some sort of central logging system/distributed database.
A distributed NoSQL database seems like a good place to store the task
logs, each task can simply be a document.

Three ways to deal with task logs when a VM is deleted:
a) never reuse a VMID
or
b) Delete all task logs for that VM when it is deleted
or
c)generate a GUID on VM creation, store GUID in the VM conf file.
Include GUID as part of the log index eg VMID+GUID
If VM is deleted and VMID reused, the tasks with the old GUID would
not be displayed for the same VMID with new GUID



On Thu, Oct 2, 2014 at 3:18 PM, Dietmar Maurer diet...@proxmox.com wrote:
 I also like the rsync logs on migration. The only missing part is HA 
 migration here.
 Can i ignore this one?

 So you would also need to sync the log on VM start. But from which node ??

 What do you think regarding deletion of VM?

 no opinion

 ___
 pve-devel mailing list
 pve-devel@pve.proxmox.com
 http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Security

[Eric Blevins]

I think that direct access to /dev/... don't work


Could Proxmox simply chown resources used by KVM before starting KVM?
This would make transition to non-root KVM easier for most people too.


 I am also unsure if there is a way to pass auth info to iscsi/glusterfs/ceph 
 libraries (without
 exposing that info to non-root users).


Could this info be provided using environment variables?
Maybe make the file read only for the KVM process group?

I created a bugzilla item for this:
https://bugzilla.proxmox.com/show_bug.cgi?id=547
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Two-Factor Authentication

[Eric Blevins]

 https://git.proxmox.com/?p=pve-access-control.git;a=commitdiff;h=ab652a80189a1498caba8c7f3f2641affe9ec3bf

 I also ordered some keys to play around with, but this will take some time ...


Good things come to those who wait.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Two-Factor Authentication

[Eric Blevins]

 I have made some initial test and uploaded that code:

 https://git.proxmox.com/?p=pve-access-control.git;a=commitdiff;h=ab652a80189a1498caba8c7f3f2641affe9ec3bf

The URL should default to https but allow configuring it in
/etc/pve/datacenter.cfg

If an attacker was able to intercept the request they could utilize
the OTP to gain access or trick Proxmox into thinking an invalid OTP
is valid.
https cert validation will (theoretically) prevent such attacks.

I have not had issues with certificate validation provided the
ca-certificates package is installed.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Two-Factor Authentication

[Eric Blevins]

 Interesting, but what happens if yubiko network is down?

Users can setup their own authentication server for yubikeys if that
is a concern.
I'd rather be locked out of my console during an authentication server
outage than have all of my data deleted when my password is
compromised.

There are also other ways to implement two-factor authentication such
as OATH TOTP (Google Authenticator) which is supported by many
applications.
The trend seems to be using apps or smartphones to implement TOTP.
Malware will just start stealing the keys from phones making that
method useless.

Ideally Proxmox would implement Yubikeys and OATH TOTP as two-factor
authentication options.
Yubikey: API is down, your locked out
OATH TOTP: Clocks out of sync, your locked out
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] pve-kernel 2.6.32-28 crashes..?

[Eric Blevins]

Also no crashes on 11 servers.
One is an AMD Phenom II x6 but it is not doing much.
Rest are Xeon 55xx, E3s and E5s

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] KVM Security

[Eric Blevins]

Why does Proxmox run KVM process as root?

Running KVM as a non-root user would be much more secure, a flaw allowing
code execution on the host would be limited by the user account.

For added security running each KVM process as a unique user would prevent
an exploit in one guest from accessing virtual disks of another guest
provided proper permissions were also applied to the vm disk files/devices.

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Seg faults during backup

[Eric Blevins]

Are you able to reproduce the segfault?


I am unable to reproduce it.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] pve-firewall: using NFLOG

[Eric Blevins]

I'm thinking about log centralisation in kibana webinterface, like this:

https://home.regit.org/2014/03/suricata-ulogd-splunk-logstash/


Well, looks like we just need to write a format those tools can read?


logstash can read just about anything, it can also listen on UDP or TCP 
and accept data in a format you specify.


Logstash uses ElasticSearch to store the data, a scalable document 
oriented search engine. Very easy to create a redundant HA ElasticSearch 
cluster too. You could also just put the data directly into ES and save 
resources by not using logstash.


Kibana is an awesome UI for logstash data stored in ES, it can store 
pre-configured dashboards. Proxmox could create a dashboard for each 
VM/Node then simply link to them:

https://logserver/#/dashboard/elasticsearch/VM101

This might not be a good fit for all Proxmox users.
I would prefer to tell Proxmox to send data to my existing logstash cluster.



___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] pve-firewall: using NFLOG

[Eric Blevins]

So you think we can use whatever format we like?

And use nxlog to feed logstash?



I do not know much about nxlog but I believe it can feed log data as 
JSON into logstash


http://logstash.net/docs/1.3.3/codecs/json

logstash has a large number of inputs and some inputs can also use codecs.
http://logstash.net/docs/1.3.3/
eg. input udp + codec json will allow you to send JSON over UDP into 
logstash.


My experience is limited to using the file,  gelf, udp, and tcp inputs.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] PVE with support of QEMU Incremental Backup

[Eric Blevins]

Just wanted to throw this idea out there.
With incremental backups we could also implement a near real time VM 
replication system.


Full backup created, shipped to remote server and restored.
Every X minutes grab changed blocks, ship to remote server, restore.


On 02/04/2014 08:28 AM, Cesar Peschiera wrote:
And will be possible (when the code will be ready) add differential 
Backup?  This is the more important to me for restore a backup 
more quick.


- Original Message - From: Alexandre DERUMIER 
aderum...@odiso.com

To: Lindsay Mathieson lindsay.mathie...@gmail.com
Cc: pve-devel@pve.proxmox.com
Sent: Tuesday, February 04, 2014 4:04 AM
Subject: Re: [pve-devel] PVE with support of QEMU Incremental Backup


tts not part of qemu yet. Once (or if) its accepted by the qemu 
project
maintainers I imagine it will become integrated into a following 
release of

proxomox.




yes, this wiki is bit old.

But good news, some prelimary patches have been send to qemu mailing 
list this month :)

https://www.mail-archive.com/qemu-devel@nongnu.org/msg212874.html

So, we just need to wait a little more :)


- Mail original -

De: Lindsay Mathieson lindsay.mathie...@gmail.com
À: pve-devel@pve.proxmox.com
Envoyé: Lundi 3 Février 2014 22:33:52
Objet: Re: [pve-devel] PVE with support of QEMU Incremental Backup

On Mon, 3 Feb 2014 05:29:12 PM Cesar Peschiera wrote:
In this link you will see that qemu offers support to Livebackup for 
making

incremental Backups
http://wiki.qemu.org/Features/Livebackup#Livebackup_-_A_Complete_Solution_f 


or_making_Full_and_Incremental_Disk_Backups_of_Running_VMs
The questions:
1- Is possible add it to PVE GUI with the options of full or 
incremental and
of an orderly fashion and easy to use for the next release (3.2)? 
... :-)

2- Disadvantages? (may be for restore: more steps?)

If you can not add this support to PVE GUI , please explain why not?




Its not part of qemu yet. Once (or if) its accepted by the qemu project
maintainers I imagine it will become integrated into a following 
release of

proxomox.
--
Lindsay
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel



___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Serious problems with the PVE Cluster

[Eric Blevins]

What you describe does not seem like a Proxmox specific problem to me.

You turned off the NFS server without dismounting the volumes from the 
nodes.

This causes IO to those NFS volumes to stall.

Proxmox does periodically check the backup directories, so there is 
consistent IO to them.


Since that IO cannot complete, it causes processes to hang.
I've even seen Linux not perform IO to local disks when IO to NFS is 
stalled for a period of time.

I am sure you can envision the horrible problems this can cause.

Using NFS soft mount might help prevent this problem but that can also 
cause corrupted data.


My suggestion to help avoid this is to use a vzdump hook script to mount 
the NFS volume only when performing a backup then dismount it at 
completion of backup. Better yet, setup HA NFS.





On 01/29/2014 10:33 PM, Cesar Peschiera wrote:

Serious problems with the PVE Cluster


@any developer that can help in the code:

I had problems with 2 of 5 PVE Hosts in a PVE cluster when the NFS 
Backup Server  was shutdown manually (without  that KVM Live Backup 
is running in the PVE Hosts),


The symptom was:
2 PVE Nodes were disconnected suddenly of PVE Cluster
The PVE GUI shows leds in red for the nodes without connection to PVE 
Cluster


To return to normal operation:
Only was necessary start the NFS Backup Server

I mean two stuff:
1- If KVM Live Backup is running in the PVE hosts while that NFS 
Backup Server is shutdown suddenly, the problem would have been more 
serious.
2- The PVE Cluster not must depend of NFS Backup Server to run 
correctly, this situation is VERY SERIOUS


For these reasons i think it will be necessary to correct the code of 
PVE Cluster


Awaiting a answer, i say see you soon

Best regards
Cesar


- Part of Original Message - From: Alexandre DERUMIER 
aderum...@odiso.com

To: Cesar Peschiera br...@click.com.py
Cc: pve-devel@pve.proxmox.com
Sent: Wednesday, January 29, 2014 2:31 AM
Subject: Re: [pve-devel] KVM Live Backup performance



And the fifth question:
What will happen if this NFS Server suddenly decomposes while KVM Live
Backup is running?


mmm, good questionI don't known what happen when backup job is 
hanging because of unavailable storage...


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

Anyways, I will try to upgrade KVM to 1.7 first (many backup related 
changes).

We can then test again and try to optimize further.


Cesar, from my testing KVM 1.7 fixed the backup related performance issues.
See archive: 
http://pve.proxmox.com/pipermail/pve-devel/2013-December/009296.html


The buffers related to Live backup are for the data sent to the backup, 
not the data sent to the VMs disk.
The data sent to the disk will still use whatever cache policy you have 
set for that disk.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] New Backup Strategy - pve-manager patch

[Eric Blevins]

When I was looking at this my first thought was 'why not use a hook script?'
Well there is no way, that I know of, for the hook script to return a 
status telling vzdump to 'skip' a particular VM.


My suggestion would be to:

1. Add a 'pre-backup' hook, if the hook script responds with non-zero 
exit status vzdump would simply skip that VM and move onto the next one.
2. Allow the hook script to alter the storage used when 'pre-backup' is 
called


It would then be possible to implement the same functionality Jeff is 
proposing using a hook script. This gives the users the ability to 
implement whatever they need without Proxmox dictating a one sized fits 
all solution.





On 01/20/2014 05:16 AM, Jeff Moskow wrote:

See my responses inline.

On Mon, Jan 20, 2014 at 06:30:43AM +, Dietmar Maurer wrote:

   1.  Allows idle machines to be skipped if they have already been backed up

This looks dangerous to me.

It shouldn't be.  If a VM hasn't been turned on (idle), then nothing should 
have changed
and the backup should be fine.  I guess that some of the configuration 
information could
have changed, but if that's concern, I could also check the date on the .conf 
file (probably
not a bad idea).


   2.  Allows multiple storages to be assigned to a backup job and for vzdump to
pick the best place to store a backup

Why do we want/need that? An what exactly is the 'best' place?

We want it because we rotate backups across multiple backup storage devices and 
if/when
a backup fails, we used to end up removing our most recent backup rather than 
the oldest
one.

The best place is defined as follows (calculated per VM):

 1) for all strategies, if there is a storage with no backup, use that
 2) for distribute, aggressive, fill find a storage where the most 
recent backup on that storage is older than the most recent backup on all others
 3) for always, replace the oldest backup no matter which storage it's 
on

Jeff
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Proxmox compatible Android client (Opaque) available for beta testing

[Eric Blevins]

2) Login to Proxmox's web interface with a mobile browser.
Proxmox GUI worked ok in Firefox but was a bit slow and clunky to 
navigate on a phone.


3) Obtain a .vv file for a VM, and tap it to open it. Opaque will 
launch automatically and use the .vv file. I've tested Firefox and 
UCBrowser with oVirt, and they can download the .vv files from there 
and open them with one tap or with a swipe of the notification area 
and a tap.

Here is where I run into problems
The file downloaded from Proxmox but is named spiceproxy not 
spiceproxy.vv


I quickly switch to a file manager, rename the file adding the .vv 
extension.

Click it, associate with Opaque Beta
Opaque opens, nothing happens.

I've tried this several times with the same results.


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

I just uploaded the qemu 1.7 package with new backup patches:

You should be able to install with:

# wget ftp://download.proxmox.com/tmp/pve-libspice-server1_0.12.4-3_amd64.deb
# wget ftp://download.proxmox.com/tmp/pve-qemu-kvm_1.7-2_amd64.deb

# dpkg -i pve-libspice-server1_0.12.4-3_amd64.deb pve-qemu-kvm_1.7-2_amd64.deb

Please can you re-run your tests?


Same host and guest as before.

Backup command, X was replaced with various values:
vzdump 108 -stdout|cstream -t X  /dev/null

Used dd to read in the VM:
dd if=/dev/vdb of=/dev/null bs=1M count=8192 skip=24000

cstream   | dd read  | improvement
14000 | 128 MB/s |  +19.0 MB/s
12000 | 130 MB/s |  +34.4 MB/s
1 | 129 MB/s |  +48.7 MB/s
8000  | 129 MB/s |  +63.4 MB/s
6000  | 135 MB/s |  +87.8 MB/s
4000  | 140 MB/s | +108.2 MB/s

Used dd to write in the VM:
dd if=/dev/zero of=/dev/vdb bs=1M count=8192 seek=24000

cstream   | dd write
14000 | 106 MB/s
12000 | 123 MB/s
1 | 125 MB/s
8000  | 126 MB/s
6000  | 128 MB/s
4000  | 123 MB/s

It seems that the backup process is greatly improved, many thanks!
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

There is also a small possibility that we have a bug ;-) I will debug that
when I update that code for 1.7.

Looking at the code, it seems that we also backup read blocks immediately. That 
way we
can avoid re-reads.

I am not sure if that is good or bad.


This would explain the degraded read performance I am seeing.
The writes are also limited in this same manner but this was already 
well known.


I have a suggestion that would help alleviate the read and write 
downsides to this.


Create a memory buffer where the reads/writes from the VM are placed.
When buffer is over a certain percentage, stop the backup read 
operations and flush the buffer.
The VM can perform IO up to the limit of the memory buffer with little 
loss in performance.


mbuffer demonstrates that adding a buffer has a positive impact on the 
VM performance:

vzdump 108 -stdout|mbuffer -m 10G |cstream -t 4000  /dev/null

In the VM I ran the same read operation as before:
dd if=/dev/vdb of=/dev/null bs=1M count=8192 skip=24000

I was able to read at 83.7MB/sec, previously with the same cstream limit 
I only got 31.8MB/sec


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

That is how it works already.

Is the size of the buffer configurable?
I would like to use 4-8G of RAM


Anyways, I will try to upgrade KVM to 1.7 first (many backup related changes).
We can then test again and try to optimize further.


Sounds like a plan
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

No, it is hard coded and quite small.

But that mbuffer looks promising - maybe we can use much larger
buffers (same size as LVM snapshot size), maybe mmap'ed?
To clarify, are you are suggesting to make the existing hard coded 
buffer larger/configurable?
If so, I like this idea. It seems like the most IO efficient way to 
perform the backup while having the least impact on the guest IO 
performance.


The LVM snapshot size and buffer size should be two different settings 
since snapshots are still used for openvz.  The settings should be 
configurable in the same manner.


The host OS storage might be slower than the VMs backing storage.
With a memory mapped file this could make the problem worse in some 
instances.



Would be great if you can run a few test with such setups.


Let me know if you need something tested.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

I am unable to reproduce that - for me LVM and Live backup are about the same 
speed.

Can you see the effect if you dump backup output directly to /dev/null?

# /usr/lib/qemu-server/vmtar  '/etc/pve/qemu-server/108.conf' 'qemu-server.conf' 
'/dev/vmdisks/test-snapshot' 'vm-disk'/dev/null

# vzdump VMID -stdout/dev/null


There is little difference reading inside the VM when writing to 
/dev/null, 117MB/sec vs 131MB/sec with LVM.


Here is the best way I have to demonstrate the problem, simulate slow 
backup media using cstream.

Slower backup media results in greater IO performance loss in the VM.

vzdump vmid -stdout|cstream -t 6000  /dev/null

In the VM I ran:
dd if=/dev/vdb of=/dev/null bs=1M count=8192 skip=2400

140MB/sec Limit, 109 MB/s Read speed in VM
120MB/sec Limit,  95.6 MB/s Read speed in VM
100MB/sec Limit,  80.3 MB/s Read speed in VM
 80MB/sec Limit,  65.6 MB/s Read speed in VM
 60MB/sec Limit,  47.2 MB/s Read speed in VM
 40MB/sec Limit,  31.8 MB/s Read speed in VM

With LVM snapshot backups the VM does not suffer performance problems if 
the backup media is slow, the backup simply takes longer.


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

I have identified one use-case where KVM Live Backup causes a 
significant decrease in IO read performance.


Start a KVM Live Backup
Inside the VM immediately run:
dd if=/dev/disk_being_backed_up of=/dev/null bs=1M count=8192

Repeated same test but used LVM snapshot and vmtar:
lvcreate -L33000M -s -n test-snapshot /dev/vmdisks/vm-108-disk-2
/usr/lib/qemu-server/vmtar  '/etc/pve/qemu-server/108.conf' 
'qemu-server.conf' '/dev/vmdisks/test-snapshot' 'vm-disk'|lzop -o 
/backup1/dump/backup.tar.lzop


KVM Live Backup: 120 seconds or more
LVM Snapshot backup: 55 seconds
With no backup: 45 seconds

Even worse was to read from an area far away from where the backup 
process is reading.

I started the backup, in the guest I ran:
dd if=/dev/disk_being_backed_up of=/dev/null bs=1M count=8192 skip=24000

KVM Live Backup: 298 seconds
LVM Snapshot Backup: 58 seconds

I have not tested writes yet and doubt I will have time to get to that 
this week.


On a positive note, CPU intensive tasks seems to be slightly faster 
using KVM Live Backup vs LVM Snapshot.

Used: sysbench --test=cpu --cpu-max-prime=5 run

Averages from 10 trials:
No backup: 92.57103s
KVM Live Backup: 93.21844s
LVM Snapshot Backup: 93.51216s

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

KVM Live Backup: 120 seconds or more
LVM Snapshot backup: 55 seconds
With no backup: 45 seconds

Why does that show a decrease in IO read performance?
I guess the dd inside the VM is much faster with live backup?

No, it took dd 120 seconds to read 8GB of data when using live backup 
and only took 55 seconds when using LVM snapshot backup.




___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

No, it took dd 120 seconds to read 8GB of data when using live backup and only
took 55 seconds when using LVM snapshot backup.

OK.

But your test dose not issue a single write?


Right, I mentioned that I had not tested writes yet.

Live backup had such a significant impact on sequential read inside the 
VM it seemed appropriate to post those results so others can also 
investigate this.





___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

Sure, I will investigate further. How large is the VM disk? What 
backup speed do you get MB/s? 


Guest was debian wheezy, the OS disk was not used for testing and marked 
as no backup.

The 2nd disk used for testing backups was 32GB, virtio cache=none
I filled that disk with data from /dev/urandom before performing any 
backup tests


The backup destination was ext4 on a single dedicated SATA disk.

I got varying MB/sec speeds from both vmtar and live backup.
Slowest for both was 41MB/sec
Fastest live backup was 61MB/sec
Fastest vmtar backup was 51MB/sec


___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

Besides, live backup uses the same IO thread as KVM, so it looks like using
one thread (with aio) perform less than using 2 thread.
But this can also be an advantage if you run more than one VM. Or you can backup
multiple VM at same time.



I agree, limiting IO from the VM during backup can have advantages.
On the flip side loosing 50% of the IO bandwidth in a database or mail 
spool VM can be a significant problem.


If storage is really fast then IO in the VM is already limited by KVM, 
limiting it even more has no advantage.



___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] KVM Live Backup performance

[Eric Blevins]

On the forum there are a number of people who are complaining about high 
load averages on the host and/or in the VM being backed up when using 
the new KVM Live Backup feature.


My suspicion is that having the KVM process move the backup data around 
the performance of the VM is negatively affected. With the KVM process 
using more CPU, performing more IO the VM is starved for resources 
possibly caused by the CPU cache being filled with backup data instead 
of VMs executing code.


I would like to perform some benchmarks where CPU/IO/RAM intensive tasks 
are run inside the VM while performing a LVM Snapshot backup and then a 
KVM Live Backup.  Comparing the completion times of the CPU/IO/RAM tasks 
would allow us to assess what subsystems are affected, good or bad, by 
KVM Live Backup.


Since all of the LVM Snapshot code was removed I am unable to perform 
the above benchmarks, anyone have a suggestion how we could perform such 
tests easily?


Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] KVM Live Backup performance

[Eric Blevins]

You said that you have some VMs which behave badly with new backup? May I ask
what you run inside those VMs?

Windows 2008 R2 servers running MSSQL
Windows 2003 servers running MSSQL and a Java based application

I doubt this is a Windows problem, loosing performance of SQL is more 
noticeable than other services.





Since all of the LVM Snapshot code was removed I am unable to perform the
above benchmarks, anyone have a suggestion how we could perform such tests
easily?

Simpyl make a lvm snapshot manually - that is quite easy.
Sure I can make an LVM Snapshot manually (suspend - snapshot - resume 
- backup)
I want to ensure I am comparing apples to apples, what would the proper 
command be to create an lzo backup archive from the LVM Snapshot?



___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] tasks / ressources

[Eric Blevins]

Or setting this value in a new the user account field ?
This makes the most sense to me, each user having their own preference 
that works regardless of what browser they log in from.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] default I/O scheduler - CFQ or Deadline?

[Eric Blevins]

I have been using deadline for years, KVM machines seem to perform best
with it.

Eric

On 05/06/2013 08:21 AM, Martin Maurer wrote:
 Hi all,

 We want to discuss the changing of the pve default I/O scheduler from CFQ to 
 Deadline. I want to collect feedback, pros and cons here.

 CFQ:
 - only CFQ support priorities (ionice)

 Deadline:
 - No support for priorities but generally faster for KVM in some (or a lot 
 of?) systems.

 The scheduler can always be changed manually if the default is not suited 
 (e.g. in /etc/default/grub).

 What do you think about this? Please report your thoughts!

 Martin





 ___
 pve-devel mailing list
 pve-devel@pve.proxmox.com
 http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] some spice news

[Eric Blevins]

Could you just use Apache?

http://httpd.apache.org/docs/2.2/mod/mod_proxy_connect.html

Eric

On 04/09/2013 07:23 AM, Alexandre DERUMIER wrote:
 But we can implement a connect method ourselves? 
 I don't known, they are HTTP::Proxy, but I don't think it can handle connect 
 tunnels.

 here another small proxy with connect support:
 https://banu.com/tinyproxy/


 - Mail original - 

 De: Dietmar Maurer diet...@proxmox.com 
 À: Alexandre DERUMIER aderum...@odiso.com 
 Cc: pve-devel@pve.proxmox.com 
 Envoyé: Mardi 9 Avril 2013 06:13:36 
 Objet: RE: [pve-devel] some spice news 

 s/mandatory/optionnal 

 But we can implement a connect method ourselves? 
 ___
 pve-devel mailing list
 pve-devel@pve.proxmox.com
 http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] kernel panic from Infiniband driver in 2.6.32-19

[Eric Blevins]

Running this patched kernel for over a week, no problems.

Eric

On 03/29/2013 03:46 AM, Dietmar Maurer wrote:
 uploaded a fix to pvetest repository - please test



___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] kernel panic from Infiniband driver in 2.6.32-19

[Eric Blevins]

After upgrading to pve-kernel-2.6.32-19 I have had a few machines stop
responding and get fenced.

Confirmed bug in IB drivers:
https://bugzilla.redhat.com/show_bug.cgi?id=913645

Thought those of you running Infiniband would want to be aware of this.

Eric



___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH] qemu-server: add support for unsecure migration (setting in datacenter.cfg)

[Eric Blevins]

On 01/07/2013 10:42 AM, Dietmar Maurer wrote:

 So you can transfer an VM with 4GB within 10 seconds? IMHO not that bad.

But 4 seconds is even better, when you have dozens of machines to
migrate seconds add up to minutes.
Besides, networks will get faster over time but SSH will still have
limitations due to its design.

 
 Anyways, would It be faster if we open a tls connection to the remote host 
 (instead of the ssh tunnel)?

Maybe this would be a good compromise allowing faster than SSH migration
while still maintaining encryption and the security that comes with it.

I would like to saturate my 10G cluster network when live migrating to
get the most speed possible, with SSH that is impossible.

Eric
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel