Re: [PVE-User] NFS Storage issue
Might also be the number of nfsd started on your Synology. -- Mark Schouten Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208 - Originele bericht - Van: Gerald Brandt (g...@majentis.com) Datum: 25-04-2019 14:51 Naar: pve-user@pve.proxmox.com Onderwerp: Re: [PVE-User] NFS Storage issue On 2019-04-25 7:34 a.m., Gianni Milo wrote: > Hello > > I have a Synology with High Availability serving NFS storage to Proxmox. >> A couple of months ago, this started happening: >> >> Apr 24 12:32:51 proxmox-1 pvestatd[3298]: unable to activate storage >> 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable >> Apr 24 13:08:00 proxmox-1 pvestatd[3298]: unable to activate storage >> 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable >> >> > It appears that the NFS storage target was unreachable during that time... > > >> It affecting backups and VM speeds. Any idea what is causing this? > > I would guess that the network link used for accessing the NFS target is > getting saturated. This sometimes can be caused by running backup jobs, > and/or other tasks like VMs demanding high i/o to that destination. > > There are no errors in the Synology logs, and the storage stays mounted and >> accessible. >> > You mean that it stays accessible after the backups are finished (or > failed) and after the i/o operations return to the normal levels ? > Do you recall if there was any of the above happening during that time ? > > >> Any ideas what's going on? >> > I would set bandwidth limits either on the storage target or on the backup > jobs. See documentation for details on how to achieve this. > Even better, separate VMs from the backups if that's possible (i.e use > different storage/network links). > > G. > ___ Thanks for the reply. Although it does happen during backups, causing serious issues that require Proxmox server restart, it also happens during the day. I don't see any saturation issues during the day, but I'll start monitoring that. So you're saying to set network bandwidth limits on the storage network (I do keep a separate network for storage). I guess upgrading the synology's with something that has 10GE would help as well. Thanks, Gerald ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] NFS Storage issue
On 2019-04-25 7:34 a.m., Gianni Milo wrote: Hello I have a Synology with High Availability serving NFS storage to Proxmox. A couple of months ago, this started happening: Apr 24 12:32:51 proxmox-1 pvestatd[3298]: unable to activate storage 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable Apr 24 13:08:00 proxmox-1 pvestatd[3298]: unable to activate storage 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable It appears that the NFS storage target was unreachable during that time... It affecting backups and VM speeds. Any idea what is causing this? I would guess that the network link used for accessing the NFS target is getting saturated. This sometimes can be caused by running backup jobs, and/or other tasks like VMs demanding high i/o to that destination. There are no errors in the Synology logs, and the storage stays mounted and accessible. You mean that it stays accessible after the backups are finished (or failed) and after the i/o operations return to the normal levels ? Do you recall if there was any of the above happening during that time ? Any ideas what's going on? I would set bandwidth limits either on the storage target or on the backup jobs. See documentation for details on how to achieve this. Even better, separate VMs from the backups if that's possible (i.e use different storage/network links). G. ___ Thanks for the reply. Although it does happen during backups, causing serious issues that require Proxmox server restart, it also happens during the day. I don't see any saturation issues during the day, but I'll start monitoring that. So you're saying to set network bandwidth limits on the storage network (I do keep a separate network for storage). I guess upgrading the synology's with something that has 10GE would help as well. Thanks, Gerald ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] NFS Storage issue
Hello I have a Synology with High Availability serving NFS storage to Proxmox. > A couple of months ago, this started happening: > > Apr 24 12:32:51 proxmox-1 pvestatd[3298]: unable to activate storage > 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable > Apr 24 13:08:00 proxmox-1 pvestatd[3298]: unable to activate storage > 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable > > It appears that the NFS storage target was unreachable during that time... > It affecting backups and VM speeds. Any idea what is causing this? I would guess that the network link used for accessing the NFS target is getting saturated. This sometimes can be caused by running backup jobs, and/or other tasks like VMs demanding high i/o to that destination. There are no errors in the Synology logs, and the storage stays mounted and > accessible. > You mean that it stays accessible after the backups are finished (or failed) and after the i/o operations return to the normal levels ? Do you recall if there was any of the above happening during that time ? > Any ideas what's going on? > I would set bandwidth limits either on the storage target or on the backup jobs. See documentation for details on how to achieve this. Even better, separate VMs from the backups if that's possible (i.e use different storage/network links). G. ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] NFS Storage issue
Storage conf file: nfs: NAS export /volume1/Storage path /mnt/pve/NAS server 192.168.50.100 content images,vztmpl,rootdir,iso,backup maxfiles 2 options vers=3,async,noatime,fsc,nodiratime Sorry, that should be: options vers=4,async,noatime,fsc,nodiratime I'm using NFSv4 Gerald ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] NFS Storage issue
Hi, I have a Synology with High Availability serving NFS storage to Proxmox. A couple of months ago, this started happening: Apr 24 12:32:51 proxmox-1 pvestatd[3298]: unable to activate storage 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable Apr 24 13:08:00 proxmox-1 pvestatd[3298]: unable to activate storage 'NAS' - directory '/mnt/pve/NAS' does not exist or is unreachable It affecting backups and VM speeds. Any idea what is causing this? There are no errors in the Synology logs, and the storage stays mounted and accessible. Any ideas what's going on? # pveversion --verbose proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve) pve-manager: 5.4-3 (running version: 5.4-3/0a6eaa62) pve-kernel-4.15: 5.3-3 pve-kernel-4.15.18-12-pve: 4.15.18-35 pve-kernel-4.15.18-11-pve: 4.15.18-34 pve-kernel-4.15.18-10-pve: 4.15.18-32 pve-kernel-4.15.18-7-pve: 4.15.18-27 pve-kernel-4.15.18-5-pve: 4.15.18-24 pve-kernel-4.15.18-4-pve: 4.15.18-23 pve-kernel-4.15.17-3-pve: 4.15.17-14 pve-kernel-4.13.13-6-pve: 4.13.13-42 pve-kernel-4.13.13-4-pve: 4.13.13-35 pve-kernel-4.13.13-2-pve: 4.13.13-33 pve-kernel-4.13.4-1-pve: 4.13.4-26 corosync: 2.4.4-pve1 criu: 2.11.1-1~bpo90 glusterfs-client: 3.8.8-1 ksm-control-daemon: 1.2-2 libjs-extjs: 6.0.1-2 libpve-access-control: 5.1-8 libpve-apiclient-perl: 2.0-5 libpve-common-perl: 5.0-50 libpve-guest-common-perl: 2.0-20 libpve-http-server-perl: 2.0-13 libpve-storage-perl: 5.0-41 libqb0: 1.0.3-1~bpo9 lvm2: 2.02.168-pve6 lxc-pve: 3.1.0-3 lxcfs: 3.0.3-pve1 novnc-pve: 1.0.0-3 proxmox-widget-toolkit: 1.0-25 pve-cluster: 5.0-36 pve-container: 2.0-37 pve-docs: 5.4-2 pve-edk2-firmware: 1.20190312-1 pve-firewall: 3.0-19 pve-firmware: 2.0-6 pve-ha-manager: 2.0-9 pve-i18n: 1.1-4 pve-libspice-server1: 0.14.1-2 pve-qemu-kvm: 2.12.1-3 pve-xtermjs: 3.12.0-1 qemu-server: 5.0-50 smartmontools: 6.5+svn4324-1 spiceterm: 3.0-5 vncterm: 1.5-3 zfsutils-linux: 0.7.13-pve1~bpo2 Storage conf file: # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content vztmpl,backup,iso zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 nfs: NAS export /volume1/Storage path /mnt/pve/NAS server 192.168.50.100 content images,vztmpl,rootdir,iso,backup maxfiles 2 options vers=3,async,noatime,fsc,nodiratime nfs: weekly_snapshots export /volume2/weekly_snapshots path /mnt/pve/weekly_snapshots server 192.168.50.100 content backup maxfiles 1 options vers=3 nfs: daily_snapshots export /volume2/daily_snapshots path /mnt/pve/daily_snapshots server 192.168.50.100 content backup maxfiles 3 options vers=3 ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
