RosiKyu opened a new issue, #12415:
URL: https://github.com/apache/cloudstack/issues/12415
### problem
When secstorage.nfs.version is SQL NULL, CloudStack incorrectly generates an
NFS mount option vers=null when mounting Secondary Storage on the Secondary
Storage VM (SSVM).
This results in mount.nfs failing with a parsing error, even though the
configuration description states that NULL should trigger default NFS version
negotiation.
**Expected Behavior:** If secstorage.nfs.version is NULL, CloudStack should
omit the vers= mount option entirely and allow the OS to auto-negotiate the NFS
version.
**Actual Behavior:** CloudStack passes vers=null to mount.nfs, which results
in a mount failure.
**Evidence**
1. Configuration value (in DB)
```
mysql> SELECT name,value,description
-> FROM configuration
-> WHERE name RLIKE
'secstorage.*nfs.*version|(^|\\.)nfs\\..*version|(^|\\.)nfs\\.version'
-> ORDER BY name;
+------------------------+-------+-------------------------------------------------------------------------------------------------------+
| name | value | description
|
+------------------------+-------+-------------------------------------------------------------------------------------------------------+
| secstorage.nfs.version | NULL | Enforces specific NFS version when
mounting Secondary Storage. If NULL default selection is performed |
+------------------------+-------+-------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
```
2. SSVM mount command generated by CloudStack
From SSVM logs:
```
/var/log/cloud.log:192:2026-01-13T07:34:23,658 WARN
[storage.resource.NfsSecondaryStorageResource] (AgentRequest-Handler-7:[])
Execution of process [4146] for command [mount -t nfs -o
soft,timeo=133,retrans=2147483647,tcp,acdirmax=0,acdirmin=0,vers=null
10.0.32.4:/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3
/mnt/SecStorage/6ff0eb2d-ffed-31c8-8f8c-e2c5c1fb2db5 ] failed.
```
3. Error produced:
```
/var/log/cloud.log:214:2026-01-13T07:34:23,684 ERROR
[storage.resource.NfsSecondaryStorageResource] (AgentRequest-Handler-7:[])
failed to copy filetemplate/tmpl/1/4/0c364cce-8288-3a92-9234-4ffde372aec7.qcow2
com.cloud.utils.exception.CloudRuntimeException: GetRootDir for
nfs://10.0.32.4/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3
failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to mount
10.0.32.4:/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3
at /mnt/SecStorage/6ff0eb2d-ffed-31c8-8f8c-e2c5c1fb2db5 due to mount.nfs:
parsing error on 'vers=' option
```
4. SSVM environment confirmation:
```
root@s-4-VM:~# uname -a
Linux s-4-VM 6.1.0-40-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.153-1
(2025-09-20) x86_64 GNU/Linux
root@s-4-VM:~#
root@s-4-VM:~#
root@s-4-VM:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/";
SUPPORT_URL="https://www.debian.org/support";
BUG_REPORT_URL="https://bugs.debian.org/";
root@s-4-VM:~#
root@s-4-VM:~#
root@s-4-VM:~# mount.nfs -V
mount.nfs: (linux nfs-utils 2.6.2)
```
5. NFS works for the same Secondary Storage export when an invalid vers=
option is not supplied
The same NFS export path used for sec3 mounts successfully on the SSVM with
negotiated NFSv4.2, as shown by existing mounts:
```
root@s-4-VM:~# mount | egrep -i 'nfs|sec' || true
securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl
(rw,nosuid,nodev,noexec,relatime)
10.0.32.4:/acs/secondary/ref-trl-10584-k-Mol9-rositsa-kyuchukova/ref-trl-10584-k-Mol9-rositsa-kyuchukova-sec3
on /mnt/SecStorage/6ff0eb2d-ffed-31c8-8f8c-e2c5c1fb2db5 type nfs4
(rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=133,retrans=2147483647,sec=sys,clientaddr=10.0.42.148,local_lock=none,addr=10.0.32.4)
```
**Impact**
- Adding a new Secondary Storage may fail intermittently or fully
- Template copy operations can fail with misleading “copy failed” errors
- Behavior contradicts documented configuration semantics
### versions
**Environment**
- Apache CloudStack: 4.20.3.0-SNAPSHOT
- Hypervisor: KVM
- Secondary Storage: NFS
- SSVM OS: Debian GNU/Linux 12 (bookworm)
- Kernel: Linux 6.1.0-40-amd64
- nfs-utils: mount.nfs (linux nfs-utils 2.6.2)
### The steps to reproduce the bug
1. Deploy Apache CloudStack 4.20.3.0-SNAPSHOT with:
- KVM hypervisor
- NFS-based Secondary Storage
- Secondary Storage VM (SSVM) running Debian 12
2. Ensure the following global configuration is unset (SQL NULL):
```
mysql> SELECT name,value FROM configuration WHERE
name='secstorage.nfs.version';
+------------------------+-------+
| name | value |
+------------------------+-------+
| secstorage.nfs.version | NULL |
+------------------------+-------+
1 row in set (0.00 sec)
```
3. Add a new Secondary Storage (image store) backed by NFS
(or trigger an operation that causes the SSVM to mount a Secondary Storage
path, e.g. template copy/sync).
4. Observe the SSVM attempting to mount the Secondary Storage (check SSVM
logs)
5. Observe that CloudStack generates the following mount command:
```
mount -t nfs -o
soft,timeo=133,retrans=2147483647,tcp,acdirmax=0,acdirmin=0,vers=null \
<nfs-server>:/acs/secondary/...-sec3 \
/mnt/SecStorage/<uuid>
```
6. Observe the mount failure: `mount.nfs: parsing error on 'vers=' option`
### What to do about it?
**Workaround**
Explicitly set the NFS version: `update configuration
name=secstorage.nfs.version value=4.2`
After this, CloudStack correctly mounts Secondary Storage using: `vers=4.2`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
