Hello,
On 2025-04-28 12:17, Fiona Ebner wrote:
As reported in the community forum [0] and the virtio-win project [1],
virtiofsd will run into its open file limit when used with a Windows
guest that reads too many files. It's also reported that the issue
does not occur with Linux guests and a workaround is using
'--inode-file-handles=mandatory' on virtiofsd command line.
I was able to reproduce the issue with a directory containing a million files.
The virtiofs share became unusable when I tried to list the files with 'ls'.
This patch allows to list all the files without problems.
The option is described as follows in the vritiofsd help:
typo: virtiofsd
When to use file handles to reference inodes instead of O_PATH file
descriptors (never, prefer, mandatory)
and the default is 'never'.
Fix the above issue by using 'prefer' rather than 'mandatory', because
that should not break other edge cases:
prefer: Attempt to generate file handles, but fall back to O_PATH
file descriptors where the underlying filesystem does not support
file handles. Useful when there are various different filesystems
under the shared directory and some of them do not support file
handles.
[0]: https://forum.proxmox.com/threads/165565/
[1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
Signed-off-by: Fiona Ebner <f.eb...@proxmox.com>
Tested-by: Markus Frank <m.fr...@proxmox.com>
---
Didn't get around to measure the performance impact yet, so feel free
to check that if you test this patch. While not being broken is more
important than good performance, it would still be good to know for
completeness.
I made a few read/write tests with fio in a Windows 11 guest.
without this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share
--announce-submounts --syslog):
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=11.0MiB/s (11.5MB/s), 11.0MiB/s-11.0MiB/s (11.5MB/s-11.5MB/s),
io=665MiB (698MB), run=60667-60667msec
WRITE: bw=6054KiB/s (6200kB/s), 6054KiB/s-6054KiB/s (6200kB/s-6200kB/s),
io=359MiB (376MB), run=60667-60667msec
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=11.1MiB/s (11.6MB/s), 11.1MiB/s-11.1MiB/s (11.6MB/s-11.6MB/s),
io=665MiB (698MB), run=59933-59933msec
WRITE: bw=6129KiB/s (6276kB/s), 6129KiB/s-6129KiB/s (6276kB/s-6276kB/s),
io=359MiB (376MB), run=59933-59933msec
with this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share
--announce-submounts --inode-file-handles=prefer --syslog):
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=9.77MiB/s (10.2MB/s), 9.77MiB/s-9.77MiB/s (10.2MB/s-10.2MB/s),
io=665MiB (698MB), run=68094-68094msec
WRITE: bw=5394KiB/s (5524kB/s), 5394KiB/s-5394KiB/s (5524kB/s-5524kB/s),
io=359MiB (376MB), run=68094-68094msec
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=10.8MiB/s (11.3MB/s), 10.8MiB/s-10.8MiB/s (11.3MB/s-11.3MB/s),
io=665MiB (698MB), run=61461-61461msec
WRITE: bw=5976KiB/s (6120kB/s), 5976KiB/s-5976KiB/s (6120kB/s-6120kB/s),
io=359MiB (376MB), run=61461-61461msec
(/share is a directory on an ext4 filesystem on the host)
PVE/QemuServer/Virtiofs.pm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/PVE/QemuServer/Virtiofs.pm b/PVE/QemuServer/Virtiofs.pm
index cfde92c9..5a91b23a 100644
--- a/PVE/QemuServer/Virtiofs.pm
+++ b/PVE/QemuServer/Virtiofs.pm
@@ -130,14 +130,17 @@ sub start_all_virtiofsd {
next if !$conf->{$opt};
my $virtiofs = parse_property_string('pve-qm-virtiofs', $conf->{$opt});
- my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs);
+ # See
https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
+ my $prefer_inode_fh =
PVE::QemuServer::Helpers::windows_version($conf->{ostype}) ? 1 : 0;
+
+ my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs,
$prefer_inode_fh);
push @$virtiofs_sockets, $virtiofs_socket;
}
return $virtiofs_sockets;
}
sub start_virtiofsd {
- my ($vmid, $fsid, $virtiofs) = @_;
+ my ($vmid, $fsid, $virtiofs, $prefer_inode_fh) = @_;
mkdir $socket_path_root;
my $socket_path = "$socket_path_root/vm$vmid-fs$fsid";
@@ -175,6 +178,7 @@ sub start_virtiofsd {
push @$cmd, '--announce-submounts';
push @$cmd, '--allow-direct-io' if $virtiofs->{'direct-io'};
push @$cmd, '--cache='.$virtiofs->{cache} if $virtiofs->{cache};
+ push @$cmd, '--inode-file-handles=prefer' if $prefer_inode_fh;
push @$cmd, '--syslog';
exec(@$cmd);
} elsif (!defined($pid2)) {
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
