RosiKyu opened a new issue, #14726:
URL: https://github.com/apache/cloudstack/issues/14726

   ### problem
   
   After `patchSystemVm` on the secondary storage VM reports "Successfully 
patched systemVM", creating a new download link fails with "mkdir: cannot 
create directory '/var/www/html/userdata/{id}/': Permission denied". A reboot 
of the SSVM makes downloads work again.
   
   
   ### versions
   
   **Build:** Apache CloudStack 24.0.0-SNAPSHOT (main), nightly 
24.0.0-SNAPSHOT.20261118, management server and KVM agents.
   
   - CloudStack: 24.0.0-SNAPSHOT (cloudstack-management and cloudstack-agent 
24.0.0-SNAPSHOT.20261118)
   - System VM: "SystemVM Template (KVM)", system VM version 24.0.0-SNAPSHOT, 
one secondary storage VM
   - Management server: Oracle Linux 9, MySQL 8.0.46
   - Hypervisor: KVM on Oracle Linux 9, libvirt 9.0.0, qemu-kvm 7.2.0
   - Storage: NFS primary storage (cluster scope), NFS secondary storage
   - Network: advanced zone
   
   ### The steps to reproduce the bug
   
   1. Make sure the SSVM is Running and its agent is Up (Infrastructure > 
System VMs).
   2. Patch the SSVM: Infrastructure > System VMs > SSVM S > Patch, or `cmk 
patch systemvm id={ssvm id} forced=true`. The result is "Successfully patched 
systemVM".
   3. Download a volume that has not been downloaded before: Storage > Volumes 
> volume V (of a stopped instance) > Download Volume, or `cmk extract volume 
id={volume id} zoneid={zone id} mode=HTTP_DOWNLOAD`.
   4. Reboot the SSVM: `cmk reboot systemvm id={ssvm id}`.
   5. Download volume V again.
   
   ### Expected Result
   
   A new download link is created right after the patch, the same as after a 
reboot.
   
   ### Actual Result
   
   Downloading a volume after the patch fails with error 530 "Unable to create 
a link for entity at volumes/... on ssvm, Error in creating directory =mkdir: 
cannot create directory '/var/www/html/userdata/{id}/': Permission denied". 
After a reboot of the SSVM the same download creates the link and the file can 
be downloaded.
   
   ### Evidence
   
   Lab mapping: `SSVM S = s-1-VM` 
(**ref-trl-12930-k-Mol9-rositsa-kyuchukova**), `volume V = ROOT-20`, `secondary 
storage public IP = 10.0.60.221`.
   
   **Management server, step 2: the patch reports success and the SSVM is Up.**
   
   ```
   [root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk patch systemvm 
id=79217cb8-2146-48b6-964e-11b370fa1044 forced=true
   {
     "details": "Successfully patched systemVM s-1-VM ",
     "success": true
   }
   [root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list systemvms 
systemvmtype=secondarystoragevm filter=name,state,agentstate,version
   {
     "count": 1,
     "systemvm": [
       {
         "agentstate": "Up",
         "name": "s-1-VM",
         "state": "Running",
         "version": "24.0.0-SNAPSHOT"
       }
     ]
   }
   ```
   
   **Management server, step 3: downloading volume V right after the patch 
fails with Permission denied.**
   
   ```
   [root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk extract volume 
id=80d39571-1693-40af-99c6-22e7b2847a15 
zoneid=fe97e06a-c522-41e0-8a01-40aa677d0562 mode=HTTP_DOWNLOAD 
filter=state,url,status
   {
     "account": "admin",
     "accountid": "4d6f477b-dec8-11f1-a260-1e00580003b2",
     "cmd": "org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd",
     "completed": "2026-11-18T10:43:19+0000",
     "created": "2026-11-18T10:42:58+0000",
     "domainid": "99f0ba66-dec7-11f1-a260-1e00580003b2",
     "domainpath": "ROOT",
     "jobid": "6664e9c7-fe37-4930-9dfd-3dedb47e0ab0",
     "jobinstanceid": "80d39571-1693-40af-99c6-22e7b2847a15",
     "jobinstancetype": "Volume",
     "jobprocstatus": 0,
     "jobresult": {
       "errorcode": 530,
       "errortext": "Unable to create a link for entity at 
volumes/8/18/1be61cc8-ff48-4632-b9dc-99eaaa66f2e9.qcow2 on ssvm, Error in 
creating directory =mkdir: cannot create directory 
‘/var/www/html/userdata/21c835f3-030e-4a29-9d16-1c228d839f2e/’: Permission 
denied"
     },
     "jobresultcode": 530,
     "jobresulttype": "object",
     "jobstatus": 2,
     "userid": "4d710ca5-dec8-11f1-a260-1e00580003b2"
   }
   🙈 Error: async API failed for job 6664e9c7-fe37-4930-9dfd-3dedb47e0ab0
   ```
   
   **Management server, steps 4 and 5: after a reboot of the SSVM the same 
download works and the file is served.**
   
   ```
   [root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk reboot systemvm 
id=79217cb8-2146-48b6-964e-11b370fa1044 filter=name,state
   {
     "systemvm": {
       "name": "s-1-VM",
       "state": "Running"
     }
   }
   [root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk extract volume 
id=80d39571-1693-40af-99c6-22e7b2847a15 
zoneid=fe97e06a-c522-41e0-8a01-40aa677d0562 mode=HTTP_DOWNLOAD 
filter=state,url,status | tee /root/ssvm-t4.json
   {
     "volume": {
       "state": "DOWNLOAD_URL_CREATED",
       "url": 
"http://10.0.60.221/userdata/189b7b21-1530-47ae-8b29-6598e6464a08/ROOT-20.qcow2";
     }
   }
   [root@ref-trl-12930-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# curl -s -o /dev/null 
-w '%{http_code} %{size_download}\n' --max-time 30 -r 0-1048575 
'http://10.0.60.221/userdata/189b7b21-1530-47ae-8b29-6598e6464a08/ROOT-20.qcow2'
   206 1048576
   ```
   
   ### What to do about it?
   
   Downloads (volumes, templates, ISOs, backup files) should keep working right 
after a successful patch of the SSVM, without a reboot. Until this is fixed, 
rebooting the SSVM after a patch makes downloads work again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to