Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

> Thats virtio-blk, not virtio-scsi?

Hi Lindsey.

This should be correct. If i understood it correctly virtio-SCSI should be:
BUS: SCSI
SCSI-Controller-Type: virtIO

Right Dmitry?

Kind regards
Michael

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

Hi Lindsay.

 

Due to dual port intel NICs onboard (Two Intel® 82574L controllers,
Dual-Port Gigabit Ethernet) we use the default e1000 driver.

 

Kind regards

Michael

 

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

> Since you use RAID controllers (I assume with BBU) try using proxmox
> cache setting 'write through' or 'directsync'. Read here:
> https://pve.proxmox.com/wiki/Performance_Tweaks

> Default mount options for ext4 in debian 8.2: type ext4
> (rw,noatime,data=ordered)

> You could try: rw,noatime,data=ordered,barrier=1

Hi Michael.

Thanks for your advices.

If i understand barriers correctly barrier=1 is for more data safety and on
the other side with  ap. 3% loss in speed.
VMs are tested with write-through and none -> resulting in same errors.

Ok now i think write-through is the speed worst cache due to perma flush

As mentioned from RedHat
This nobarrier or barrier=0 setting should only be set on host systems and
should never be used inside the VMs itself:
->
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/htm
l/Storage_Administration_Guide/writebarrieronoff.html

Ok i now got that in ext4 barrier is enabled by default if nothing is
specified in mount options:
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt

"When comparing versus ext3, note that ext4 enables write barriers by
default, while ext3 does not enable write barriers by default."

So to answer your question, barrier should be on already

Sorry for my wired thinking. It´s to much for me today :P



Due to ZMCP (non-volatile Flash covered cache) following cache settings
should be best and safe together

Settings on Raid-Controller (default):
Write-cache mode   : Enabled (write-back)
Write-cache setting : Enabled (write-back) when protected by battery/ZMM

Settings in host-system:
Barrier on by default (could be turned off, due to non-volatile raid-cache)

Settings for guest:
!!! iothread = on !!!   ->
https://wiki.mikejung.biz/KVM_/_Xen#virtio-blk_iothreads_.28x-data-plane.29
cache = writeback
barrier per default on
BUS=virtio
SCSI-Controller-Type: default (LSI 53C895A)

Next test could be switching to:
BUS=SCSI
SCSI-Controller-Type: virtIO

PS: iothread is off per default

I´ll give you an update as soon as possible.

@ Emmanuel
If i don´t find the bug, be prepared that we´ll start a ticket ;)

Kind regards
Michael



___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

Hi Davel.

 

Thanks for the info. Due to non-shared storage / no SAN we don´t use HA
setup.

We killed our cluster to test if the problem would be solved by using
proxmox 4.1 instead of 3.4, but the errors are the same :(

 

Kind regards

Michael

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Christian Kivalo]

On 2016-01-07 14:33, Michael Pöllinger wrote:


Has anyone running ext4 running stable in the VMs?


I'm running multiple Debian 8.2 VMs on proxmox 4.1 host with ext4 as 
filesystem inside the VMs. No problems.



kind regards
Michael


--
 Christian Kivalo
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

Hi Michael R.

That would be a interesting thing.

I´ve spoken with a friend some minutes ago and he mentioned an idea
about filesystems.
Could it be that there is a bug / problem with ext4 and kvm?

All other VMs, which are still running fine, are on ext3.

Has anyone running ext4 running stable in the VMs?

kind regards
Michael
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Rasmussen]

On Thu, 7 Jan 2016 15:48:08 +0100
Michael Pöllinger  wrote:

> 
> Only for info i can only repeat that this problems occour only on new VMs 
> we´ve isntalled on debian 8.x with ext4.
> Older Systems with ext3 run and run and run without any errors.
> 
> So i guess no hardware errors, cause there should be any error information or 
> different behave in those VMs
> 
Since you use RAID controllers (I assume with BBU) try using proxmox
cache setting 'write through' or 'directsync'. Read here:
https://pve.proxmox.com/wiki/Performance_Tweaks

Default mount options for ext4 in debian 8.2: type ext4
(rw,noatime,data=ordered)

You could try: rw,noatime,data=ordered,barrier=1

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael  rasmussen  cc
http://pgp.mit.edu:11371/pks/lookup?op=get=0xD3C9A00E
mir  datanom  net
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE501F51C
mir  miras  org
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE3E80917
--
/usr/games/fortune -es says:
***
  ***
 *
 ** Confucious say: "Is stuffy inside fortune cookie."
  ***
***


pgpWBBnsSs0Ds.pgp
Description: OpenPGP digital signature
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[David Lawley]

sorry have not completely followed your thread but the only time I had 
this issue was on ver 3.x of prox and when I enabled HA on vms without 
shutting them down first.  Shutdown VM, enable HA, then restart. I ended 
up having multiple or identical vm running competing for resources.  Nasty


Not sure how prox 4 is handling this now, at the time I felt it should 
not allow me to do it, but it did.


I eventually caught it pinging a vm that I knew was off.  Then following 
up with watching with htop.


BTW, I'm still running 3.4 without HA.  HA always seems to bite me in 
the butt.






On 1/7/2016 8:58 AM, Christian Kivalo wrote:

On 2016-01-07 14:33, Michael Pöllinger wrote:


Has anyone running ext4 running stable in the VMs?


I'm running multiple Debian 8.2 VMs on proxmox 4.1 host with ext4 as
filesystem inside the VMs. No problems.


kind regards
Michael


--
  Christian Kivalo
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user


___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

Hi Emmanuel.

Sounds like hardeware, yes, but on two nodes at the same time?
That would be a chance ;)

kind regards
Michael
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Michael Pöllinger]

> First a question: Is the file system hosting the VM disks also using ext4?
We are talking about two different nodes now, due to the test of running the vm 
on a proxmox 4.1 also.
All nodes are installed with the iso installer, so:

Node 1 (latest 3.4. proxmox): 
/dev/pve/root / ext3 
/dev/pve/data /var/lib/vz ext3 

Node 2 (latest 4.1 proxmox):
/dev/pve/root / ext4 
/dev/pve/data /var/lib/vz ext4 



> And is the disk connected through a raid controller?
Both nodes with same raid-controller: Adaptec 5405Z with ZMCP-Module

Node 1: Raid 1 over 2 HDN724030AL
Node 2: Raid 5 over 4 ST31000424SS



> Are the disks connected through a SAS backplane, and if so do you use SATA 
> disks instead of SAS disks in this backplane?
All systems are on SAS basis.
The disks are with SAS backplanes.


> Ideas for further inquiry is testing various combinations of host file
> system cache configuration and VM file system cache configuration as
> well as the used cache configuration in Proxmox.
Thats what we´ve already started with (proxmox configs)
We switched the VMs-Configs from IDE to virtIO with LSI setting

On IDE the errors where:
[So Dez 27 05:17:44 2015] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 
0x6 frozen

On virtIO:
task xxx blocked for more than 120 seconds.

===

Only for info i can only repeat that this problems occour only on new VMs we´ve 
isntalled on debian 8.x with ext4.
Older Systems with ext3 run and run and run without any errors.

So i guess no hardware errors, cause there should be any error information or 
different behave in those VMs


@All: I´m really deeply grateful that you all listen and try to help me with 
this. 

kind regards
Michael

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HDD errors in VMs

[Lindsay Mathieson]

What network card/drivers are you using?

ethtool -i eth0 etc

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] Change node IP addresses

[Frederic Van Espen]

Hi,

Is it possible to change the node IP addresses without breaking the cluster
for an extended amount of time? Are there any caveats?

I would like to move the PVE nodes in a separate dedicated network for
better security.

Cheers,

Frederic
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user