Re: Linstor volume boot issues

[Antoine Boucher]

Hello Nathan,

We were experimenting with Linstor and we have the same issue when a VM is 
directly created from a Template on a Linstor Primary Storage but not when 
created on a non Linstor PS and then Migrated to Linstor PS.  Have you found 
the culprit ?

Regards,
Antoine

On Aug 9, 2023, at 15:56, Nathan Gleason  wrote:

Hello,

We are working to get Linstor integrated into our environment and have
noticed a few issues.

When we spawn a VM everything works as expected.  Cloudstack reports
success, we see the volume(s) are "InUse" in Linstor.  But when we attach
to the console we see the SeaBIOS screen with the message "Boot failed: Not
a bootable disk" "No bootable device". The strange thing is that sometimes
it works and the VM will boot.  We have tried many different templates that
all work just fine on local storage.  We also see that sometimes the drbd
primary volume is assigned to the "DfltDisklessStorPool", sometimes it is
assigned to the pool we created "cloudstack_pool"

Things we have checked include:
- Make sure the block device is indeed attached to the host machine
- virsh dumpxml the VM to make sure it's looking at the correct block device
- numerous "--place-count N" resource group configurations

Has anyone run into this boot issue?

Versions:
Ubuntu 22.04
Cloudstack 4.17.1.0
linstor-server-1.23.0
drbd-9.2.4
drbd-utils-9.25

Thanks,
Nathan

Primary Storage Migration UI

[Antoine Boucher]

In ACS version 4.18.1, when conducting a VM Primary Storage Migration, should 
the list of potential destinations exclude any Primary Storages that are 
disabled?

It currently shows them all.

Regards,
Antoine

Re: Console Proxy Stopped Working after 4.17.2 -> 4.18.1 upgrade on KVM Hosts

[Antoine Boucher]

Thank you again, Stephan; all is resolved after re-importing the certificates 
one more time.


-Antoine


> On Jan 15, 2024, at 4:19 AM, Stephan Bienek  wrote:
> 
> Hello Antoine,
> 
> we had exactly the same issue.
> 
> Re-uploading and re-applying all certificates according to the documentation 
> solved the issue, they were lost during the update.
> 
> Best regards,
> Stephan
> 
>> Antoine Boucher  hat am 15.01.2024 06:32 CET 
>> geschrieben:
>> 
>> 
>> Hello,
>> 
>> The console access now fails on KVM hosts after upgrading from ACS 4.17.2 to 
>> ACS 4.18.1
>> The console has been working flawlessly on 4.17.2 with SSL and 
>> www-xxx-yyy-zzz.domain.com configuration.
>> I see some authentication and loading resource issues in the following Proxy 
>> logs.  Everything else works well, including the console access on Xen hosts.
>> 
>> The firewall on the host is disabled, times are synced, the management 
>> server is using the local time zone, and the KVM hosts are using UTC.
>> 
>> What am I missing?  
>> 
>> Regards,
>> Antoine
>> 
>> 
>> Note:  The KVM host IP has been changed to 10.xx.xx.xx
>> 
>> 
>> MANAGEMENT SERVER LOGS
>> 
>> 2024-01-14 23:57:20,027 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-4:null) (logid:) SeqA 104-3099: Processing Seq 
>> 104-3099:  { Cmd , MgmtId: -1, via: 104, Ver: v1, Flags: 11, 
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"493","_loadInfo":"{
>>  "connections": [],
>>  "removedSessions": []
>> }","wait":"0","bypassHostMaintenance":"false"}}] }
>> 2024-01-14 23:57:20,032 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-4:null) (logid:) SeqA 104-3099: Sending Seq 104-3099:  
>> { Ans: , MgmtId: 130593671224, via: 104, Ver: v1, Flags: 100010, 
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
>>  }
>> 2024-01-14 23:57:21,363 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-5:null) (logid:) SeqA 102-2964: Processing Seq 
>> 102-2964:  { Cmd , MgmtId: -1, via: 102, Ver: v1, Flags: 11, 
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"488","_loadInfo":"{
>>  "connections": [],
>>  "removedSessions": []
>> }","wait":"0","bypassHostMaintenance":"false"}}] }
>> 2024-01-14 23:57:21,367 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-5:null) (logid:) SeqA 102-2964: Sending Seq 102-2964:  
>> { Ans: , MgmtId: 130593671224, via: 102, Ver: v1, Flags: 100010, 
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
>>  }
>> 2024-01-14 23:57:23,558 DEBUG [c.c.a.ApiServlet] 
>> (qtp279593458-695:ctx-2e31e696) (logid:a3f76229) ===START===  104.28.133.19 
>> -- GET  
>> virtualmachineid=bca11a2a-f642-4ded-a8d8-809a046de56b=createConsoleEndpoint=json
>> 2024-01-14 23:57:23,559 DEBUG [c.c.a.ApiServlet] 
>> (qtp279593458-695:ctx-2e31e696) (logid:a3f76229) Two factor authentication 
>> is already verified for the user 2, so skipping
>> 2024-01-14 23:57:23,574 DEBUG [c.c.a.ApiServer] 
>> (qtp279593458-695:ctx-2e31e696 ctx-ced9fb8e) (logid:a3f76229) CIDRs from 
>> which account 'Account 
>> [{"accountName":"admin","id":2,"uuid":"2cee75f9-8bc4-11ec-9c43-001e67fd4838"}]'
>>  is allowed to perform API calls: 0.0.0.0/0,::/0
>> 2024-01-14 23:57:23,579 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] 
>> (qtp279593458-695:ctx-2e31e696 ctx-ced9fb8e) (logid:a3f76229) Account 
>> [Account 
>> [{"accountName":"admin","id":2,"uuid":"2cee75f9-8bc4-11ec-9c43-001e67fd4838"}]]
>>  is Root Admin or Domain Admin, all APIs are allowed.
>> 2024-01-14 23:57:23,581 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] 
>> (qtp279593458-695:ctx-2e31e696 ctx-ced9fb8e) (logid:a3f76229) Project is 
>> null, ProjectRoleBasedApiAccessChecker only applies to projects, returning 
>> API [createConsoleEndpoint] for user [User 
>> {"username":"admin","uuid":"2cefb708-8bc4-11ec-9c43-001e67fd4838"}.] as 
>> allowed.
>> 2024-01-14 23:

Re: Console Proxy Stopped Working after 4.17.2 -> 4.18.1 upgrade on KVM Hosts

[Antoine Boucher]

Thank you Stephan.  We also have re-uploaded the certificates, but we still 
have the issue.

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Jan 15, 2024, at 4:19 AM, Stephan Bienek  wrote:
> 
> Hello Antoine,
> 
> we had exactly the same issue.
> 
> Re-uploading and re-applying all certificates according to the documentation 
> solved the issue, they were lost during the update.
> 
> Best regards,
> Stephan
> 
>> Antoine Boucher  hat am 15.01.2024 06:32 CET 
>> geschrieben:
>> 
>> 
>> Hello,
>> 
>> The console access now fails on KVM hosts after upgrading from ACS 4.17.2 to 
>> ACS 4.18.1
>> The console has been working flawlessly on 4.17.2 with SSL and 
>> www-xxx-yyy-zzz.domain.com configuration.
>> I see some authentication and loading resource issues in the following Proxy 
>> logs.  Everything else works well, including the console access on Xen hosts.
>> 
>> The firewall on the host is disabled, times are synced, the management 
>> server is using the local time zone, and the KVM hosts are using UTC.
>> 
>> What am I missing?  
>> 
>> Regards,
>> Antoine
>> 
>> 
>> Note:  The KVM host IP has been changed to 10.xx.xx.xx
>> 
>> 
>> MANAGEMENT SERVER LOGS
>> 
>> 2024-01-14 23:57:20,027 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-4:null) (logid:) SeqA 104-3099: Processing Seq 
>> 104-3099:  { Cmd , MgmtId: -1, via: 104, Ver: v1, Flags: 11, 
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"493","_loadInfo":"{
>>  "connections": [],
>>  "removedSessions": []
>> }","wait":"0","bypassHostMaintenance":"false"}}] }
>> 2024-01-14 23:57:20,032 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-4:null) (logid:) SeqA 104-3099: Sending Seq 104-3099:  
>> { Ans: , MgmtId: 130593671224, via: 104, Ver: v1, Flags: 100010, 
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
>>  }
>> 2024-01-14 23:57:21,363 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-5:null) (logid:) SeqA 102-2964: Processing Seq 
>> 102-2964:  { Cmd , MgmtId: -1, via: 102, Ver: v1, Flags: 11, 
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"488","_loadInfo":"{
>>  "connections": [],
>>  "removedSessions": []
>> }","wait":"0","bypassHostMaintenance":"false"}}] }
>> 2024-01-14 23:57:21,367 DEBUG [c.c.a.m.AgentManagerImpl] 
>> (AgentManager-Handler-5:null) (logid:) SeqA 102-2964: Sending Seq 102-2964:  
>> { Ans: , MgmtId: 130593671224, via: 102, Ver: v1, Flags: 100010, 
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
>>  }
>> 2024-01-14 23:57:23,558 DEBUG [c.c.a.ApiServlet] 
>> (qtp279593458-695:ctx-2e31e696) (logid:a3f76229) ===START===  104.28.133.19 
>> -- GET  
>> virtualmachineid=bca11a2a-f642-4ded-a8d8-809a046de56b=createConsoleEndpoint=json
>> 2024-01-14 23:57:23,559 DEBUG [c.c.a.ApiServlet] 
>> (qtp279593458-695:ctx-2e31e696) (logid:a3f76229) Two factor authentication 
>> is already verified for the user 2, so skipping
>> 2024-01-14 23:57:23,574 DEBUG [c.c.a.ApiServer] 
>> (qtp279593458-695:ctx-2e31e696 ctx-ced9fb8e) (logid:a3f76229) CIDRs from 
>> which account 'Account 
>> [{"accountName":"admin","id":2,"uuid":"2cee75f9-8bc4-11ec-9c43-001e67fd4838"}]'
>>  is allowed to perform API calls: 0.0.0.0/0,::/0
>> 2024-01-14 23:57:23,579 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] 
>> (qtp279593458-695:ctx-2e31e696 ctx-ced9fb8e) (logid:a3f76229) Account 
>> [Account 
>> [{"accountName

Re: Console Proxy Stopped Working after 4.17.2 -> 4.18.1 upgrade on KVM Hosts

[Antoine Boucher]

We have recreated the CPVM

-Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Jan 15, 2024, at 2:56 AM, Wei ZHOU  wrote:
> 
> Hi,
> 
> Have you patched the CPVM, or recreated  CPVM ?
> 
> -Wei
> 
> On Mon, 15 Jan 2024 at 06:34, Antoine Boucher  <mailto:antoi...@haltondc.com>> wrote:
> 
>> Hello,
>> 
>> The console access now fails on KVM hosts after upgrading from ACS 4.17.2
>> to ACS 4.18.1
>> The console has been working flawlessly on 4.17.2 with SSL and
>> www-xxx-yyy-zzz.domain.com configuration.
>> I see some authentication and loading resource issues in the following
>> Proxy logs.  Everything else works well, including the console access on
>> Xen hosts.
>> 
>> The firewall on the host is disabled, times are synced, the management
>> server is using the local time zone, and the KVM hosts are using UTC.
>> 
>> What am I missing?
>> 
>> Regards,
>> Antoine
>> 
>> 
>> Note:  The KVM host IP has been changed to 10.xx.xx.xx
>> 
>> 
>> MANAGEMENT SERVER LOGS
>> 
>> 2024-01-14 23:57:20,027 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-4:null) (logid:) SeqA 104-3099: Processing Seq
>> 104-3099:  { Cmd , MgmtId: -1, via: 104, Ver: v1, Flags: 11,
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"493","_loadInfo":"{
>>  "connections": [],
>>  "removedSessions": []
>> }","wait":"0","bypassHostMaintenance":"false"}}] }
>> 2024-01-14 23:57:20,032 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-4:null) (logid:) SeqA 104-3099: Sending Seq
>> 104-3099:  { Ans: , MgmtId: 130593671224, via: 104, Ver: v1, Flags: 100010,
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
>> }
>> 2024-01-14 23:57:21,363 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-5:null) (logid:) SeqA 102-2964: Processing Seq
>> 102-2964:  { Cmd , MgmtId: -1, via: 102, Ver: v1, Flags: 11,
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"488","_loadInfo":"{
>>  "connections": [],
>>  "removedSessions": []
>> }","wait":"0","bypassHostMaintenance":"false"}}] }
>> 2024-01-14 23:57:21,367 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-5:null) (logid:) SeqA 102-2964: Sending Seq
>> 102-2964:  { Ans: , MgmtId: 130593671224, via: 102, Ver: v1, Flags: 100010,
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
>> }
>> 2024-01-14 23:57:23,558 DEBUG [c.c.a.ApiServlet]
>> (qtp279593458-695:ctx-2e31e696) (logid:a3f76229) ===START===  104.28.133.19
>> -- GET
>> virtualmachineid=bca11a2a-f642-4ded-a8d8-809a046de56b=createConsoleEndpoint=json
>> 2024-01-14 23:57:23,559 DEBUG [c.c.a.ApiServlet]
>> (qtp279593458-695:ctx-2e31e696) (logid:a3f76229) Two factor authentication
>> is already verified for the user 2, so skipping
>> 2024-01-14 23:57:23,574 DEBUG [c.c.a.ApiServer]
>> (qtp279593458-695:ctx-2e31e696 ctx-ced9fb8e) (logid:a3f76229) CIDRs from
>> which account 'Account
>> [{"accountName":"admin","id":2,"uuid":"2cee75f9-8bc4-11ec-9c43-001e67fd4838"}]'
>> is allowed to perform API calls: 0.0.0.0/0,::/0
>> 2024-01-14 <http://0.0.0.0/0,::/02024-01-14> 23:57:23,579 INFO
>> [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp279593458-695:ctx-2e31e696
>> ctx-ced9fb8e) (logid:a3f76229) Account [Account
>> [{"accountName":"admin","id":2,"uuid":"2cee75f9-8bc4-11ec-9c43-001e67fd4838"}]]
>> is Root Admin or Domain Admin, all APIs are allowed.
>> 2024-01-14 23:57:23,581 WARN  [o.a.c.a.Pr

Console Proxy Stopped Working after 4.17.2 -> 4.18.1 upgrade on KVM Hosts

[Antoine Boucher]

44803,
  "sessionUuid": "d7e6f73b-230a-43b7-95dc-dfc20c2f8992"
}
  ]
}","wait":"0","bypassHostMaintenance":"false"}}] }
2024-01-14 23:57:24,855 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-6:null) (logid:) SeqA 102-2966: Sending Seq 102-2966:  { 
Ans: , MgmtId: 130593671224, via: 102, Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
 }
2024-01-14 23:57:30,026 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-14:null) (logid:) SeqA 104-3100: Processing Seq 104-3100: 
 { Cmd , MgmtId: -1, via: 104, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"493","_loadInfo":"{
  "connections": [],
  "removedSessions": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2024-01-14 23:57:30,030 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-14:null) (logid:) SeqA 104-3100: Sending Seq 104-3100:  { 
Ans: , MgmtId: 130593671224, via: 104, Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
 }
2024-01-14 23:57:31,364 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-12:null) (logid:) SeqA 102-2967: Processing Seq 102-2967: 
 { Cmd , MgmtId: -1, via: 102, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"488","_loadInfo":"{
  "connections": [
{
  "id": 13,
  "clientInfo": "",
  "host": "10.xxx.xxx.xxx",
  "port": 5905,
  "tag": "bca11a2a-f642-4ded-a8d8-809a046de56b",
  "createTime": 1705294644803,
  "lastUsedTime": 1705294644803,
  "sessionUuid": "d7e6f73b-230a-43b7-95dc-dfc20c2f8992"
}
  ],
  "removedSessions": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2024-01-14 23:57:31,368 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-12:null) (logid:) SeqA 102-2967: Sending Seq 102-2967:  { 
Ans: , MgmtId: 130593671224, via: 102, Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
 }



CONSOLE PROXY LOGS

2024-01-15 04:57:24,497 INFO  [cloud.consoleproxy.ConsoleProxyResourceHandler] 
(Thread-175:null) Get resource request for /resource/noVNC/vnc.html
2024-01-15 04:57:24,506 INFO  [cloud.consoleproxy.ConsoleProxyResourceHandler] 
(Thread-175:null) Sent file /resource/noVNC/vnc.html with content type text/html
2024-01-15 04:57:24,630 INFO  [cloud.consoleproxy.ConsoleProxyResourceHandler] 
(Thread-176:null) Get resource request for /resource/noVNC/app/sounds/bell.mp3
2024-01-15 04:57:24,635 INFO  [cloud.consoleproxy.ConsoleProxyResourceHandler] 
(Thread-176:null) Sent file /resource/noVNC/app/sounds/bell.mp3 with content 
type application/octet-stream
2024-01-15 04:57:24,735 INFO  [cloud.consoleproxy.ConsoleProxyNoVNCHandler] 
(qtp1640959236-39:null) Get websocket connection request from remote IP : 
104.28.133.19
2024-01-15 04:57:24,810 INFO  [cloud.consoleproxy.ConsoleProxyNoVncClient] 
(Thread-177:null) Connect to VNC server directly. host: 10.xxx.xxx.xxx, port: 
5905
2024-01-15 04:57:24,815 INFO  [consoleproxy.vnc.NoVncClient] (Thread-177:null) 
Connecting to VNC server 10.xxx.xxx.xxx:5905 ...
2024-01-15 04:57:25,028 INFO  [vnc.security.VncAuthSecurity] (Thread-177:null) 
VNC server requires password authentication
2024-01-15 04:57:25,041 INFO  [vnc.security.VncAuthSecurity] (Thread-177:null) 
Finished VNCAuth security
 



Antoine Boucher

Re: Changeing the IP of an NFS Secondary Storage

[Antoine Boucher]

Yes - thanks again



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Dec 19, 2023, at 11:20 AM, Wido den Hollander  wrote:
> 
> (Back to ML)
> 
> Op 19/12/2023 om 17:18 schreef Antoine Boucher:
>> Thank you, Wido. That's a great suggestion. Do you define all your ACS using 
>> DNS names?
> 
> Yes, I always use DNS everywhere. Never use static IPs in any of the 
> configuration.
> 
> If it's Ceph Monitors for primary storage, NFS hostnames for Primary Storage 
> or Secondary Storage.
> 
> Make sure you have a proper domain name to use for your infrastructure and 
> then just use DNS.
> 
> Wido
> 
>> I’m a fan - Regards,
>> Antoine
>> *Antoine Boucher*
>> antoi...@haltondc.com
>> [o] +1-226-505-9734
>> www.haltondc.com
>> “Data security made simple”
>> HDClogo7-small.png
>> Confidentiality Warning: This message and any attachments are intended only 
>> for the use of the intended recipient(s), are confidential, and may be 
>> privileged. If you are not the intended recipient, you are hereby notified 
>> that any review, retransmission, conversion to hard copy, copying, 
>> circulation or other use of this message and any attachments is strictly 
>> prohibited. If you are not the intended recipient, please notify the sender 
>> immediately by return e-mail, and delete this message and any attachments 
>> from your system.
>>> On Dec 19, 2023, at 10:18 AM, Wido den Hollander  wrote:
>>> 
>>> 
>>> 
>>> Op 15/12/2023 om 18:10 schreef Antoine Boucher:
>>>> Hello, When I initially adopted ACS a few years ago, I inadvertently 
>>>> placed my secondary storage on the storage network. Surprisingly, 
>>>> everything functioned well in the first year.
>>>> However, issues arose when I updated the system to the next version. The 
>>>> SSVM failed to manage and operate the storage correctly, and I encountered 
>>>> difficulties migrating templates and snapshots to a different SS.
>>>> My Question:
>>>> To permanently resolve my problem, could I change the SS IP to the 
>>>> management network by:
>>>>  1. Modifying the address of my SS.
>>>>  2. Updating the address in the URL field of the image_store table.
>>>>  3. Deleting the existing SSVM in the zone.
>>>>  4. Reboot the Management Server (?)
>>> 
>>> I think that works. While you are at it, I suggest using a DNS name 
>>> pointing to the SS. Should you ever need to change the IP again, you can 
>>> just change the DNS without needing to touch CloudStack.
>>> 
>>> Wido
>>> 
>>>> Any suggestions would be greatly appreciated.
>>>> Regards,
>>>> Antoine
>>>> Antoine Boucher
>>>> antoi...@haltondc.com
>>>> [o] +1-226-505-9734
>>>> www.haltondc.com

Re: Changeing the IP of a NFS Secondary Storage

[Antoine Boucher]

Can anyone provide insights into my proposed solution, helping me determine 
whether it is advisable to proceed cautiously with modifications to the 
management server database?

Regards,
Antoine


> On Dec 15, 2023, at 12:10 PM, Antoine Boucher  wrote:
> 
> Hello, When I initially adopted ACS a few years ago, I inadvertently placed 
> my secondary storage on the storage network. Surprisingly, everything 
> functioned well in the first year. 
> 
> However, issues arose when I updated the system to the next version. The SSVM 
> failed to manage and operate the storage correctly, and I encountered 
> difficulties migrating templates and snapshots to a different SS.
> 
> My Question: 
> 
> To permanently resolve my problem, could I change the SS IP to the management 
> network by: 
> 
>  1. Modifying the address of my SS. 
>  2. Updating the address in the URL field of the image_store table. 
>  3. Deleting the existing SSVM in the zone. 
>  4. Reboot the Management Server (?) 
> 
> Any suggestions would be greatly appreciated.
> 
> Regards,
> Antoine
> 
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
> 

Changeing the IP of an NFS Secondary Storage

[Antoine Boucher]

Hello, When I initially adopted ACS a few years ago, I inadvertently placed my 
secondary storage on the storage network. Surprisingly, everything functioned 
well in the first year. 

However, issues arose when I updated the system to the next version. The SSVM 
failed to manage and operate the storage correctly, and I encountered 
difficulties migrating templates and snapshots to a different SS.

My Question: 

To permanently resolve my problem, could I change the SS IP to the management 
network by: 

 1. Modifying the address of my SS. 
 2. Updating the address in the URL field of the image_store table. 
 3. Deleting the existing SSVM in the zone. 
 4. Reboot the Management Server (?) 

Any suggestions would be greatly appreciated.

Regards,
Antoine


Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

Re: SSVM routing issue

[Antoine Boucher]

HI Jithin,

Thank you for your response.  I’m not sure why more people are not hitting the 
same issue?  My storage network is also used for my primary storage.  If I 
remove the tag I’m assuming that host will fail to reach my primary storage on 
vlan 53.   

We are going to migrate to a new zone using vxlan soon.  What are the best 
practices for the cloud bridges? 0, 1 and cloudbrx for storage?  We use 
cloudbr0 for private and 1 for guest.  I’m assuming that is the kvm host cs 
agent is properly configure all should be fine?

I will file a new bug report soon.


Regards,
Antoine


> On Jun 21, 2023, at 12:12 AM, Jithin Raju  wrote:
> 
> Hi Antonine,
> 
>> I also tried to define the storage traffic type with VLAN 53; the VLAN/VNI 
>> column shows blank,
> 
> I see the same issue with CS 4.18 too, this appears to be a bug could you 
> report this in github? In my testing the VLAN is present in cloud. 
> dc_storage_network_ip_range record and the same is present in the API 
> response for API listStorageNetworkIpRange as well. Just that it is not 
> displayed in the UI.
> 
> Could you try removing the VLAN tag 53 from the bridge Cloudbr53 and let 
> cloudstack configure the storage VLAN?
> 
>> Is the storage definition only or mainly used for the SSVM?
> 
> Only for SSVM.
> 
> -Jithin
> 
> From: Antoine Boucher 
> Date: Wednesday, 21 June 2023 at 12:14 AM
> To: stanley.bur...@gmail.com , users 
> 
> Subject: Re: SSVM routing issue
> Hi Stanley,
> 
> You will find the answers below.
> 
> 
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com<http://www.haltondc.com>
> 
> “Data security made simple”
> 
> 
> 
> 
> 
>> On May 24, 2023, at 8:59 PM, Stanley Burkee  wrote:
>> 
>> Hi Antoine,
>> 
>> 
>> Please share the cloudstack version you are using. Also check if you have
>> connectivity between your management network & storage network.
> 
> 4.17.2.0
> 
> 
> 
> 
>> 
>> Please share the management server logs & your zone cloudbr0 & other
>> interfaces configurations.
> 
> Here is my CentOS network config: (Management Server and Some Clusters)
> 
> [root@nimbus network-scripts]# cat ifcfg*
> DEVICE=bond0
> ONBOOT=yes
> BONDING_OPTS="mode=6"
> BRIDGE=cloudbr0
> NM_CONTROLLED=no
> 
> 
> DEVICE=bond0.53
> VLAN=yes
> BOOTPROTO=static
> ONBOOT=yes
> TYPE=Unknown
> BRIDGE=cloudbr53
> 
> 
> DEVICE=bond1
> ONBOOT=yes
> BONDING_OPTS="mode=6"
> BRIDGE=cloudbr1
> NM_CONTROLLED=no
> 
> 
> DEVICE=cloudbr0
> ONBOOT=yes
> TYPE=Bridge
> IPADDR=10.101.2.40
> NETMASK=255.255.252.0
> GATEWAY=10.101.0.1
> DOMAIN="haltondc.net"
> DEFROUTE=yes
> NM_CONTROLLED=no
> DELAY=0
> 
> 
> DEVICE=cloudbr1
> ONBOOT=yes
> TYPE=Bridge
> NM_CONTROLLED=no
> DELAY=0
> 
> 
> DEVICE=cloudbr53
> ONBOOT=yes
> TYPE=Bridge
> VLAN=yes
> IPADDR=10.101.6.40
> #GATEWAY=10.101.6.1
> NETMASK=255.255.254.0
> NM_CONTROLLED=no
> DELAY=0
> 
> 
> DEVICE=eno1
> TYPE=Ethernet
> USERCTL=no
> MASTER=bond1
> SLAVE=yes
> BOOTPROTO=none
> NM_CONTROLLED=no
> ONBOOT=yes
> 
> DEVICE=eno2
> TYPE=Ethernet
> USERCTL=no
> MASTER=bond1
> SLAVE=yes
> BOOTPROTO=none
> NM_CONTROLLED=no
> ONBOOT=yes
> DEVICE=eno3
> TYPE=Ethernet
> USERCTL=no
> MASTER=bond1
> SLAVE=yes
> BOOTPROTO=none
> NM_CONTROLLED=no
> ONBOOT=yes
> DEVICE=eno4
> TYPE=Ethernet
> USERCTL=no
> #MASTER=bond1
> #SLAVE=yes
> #BOOTPROTO=none
> NM_CONTROLLED=no
> ONBOOT=no
> DEVICE=ens2f0
> TYPE=Ethernet
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
> NM_CONTROLLED=no
> ONBOOT=yes
> 
> 
> DEVICE=ens2f1
> TYPE=Ethernet
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
> NM_CONTROLLED=no
> ONBOOT=yes
> 
> 
> DEVICE=lo
> IPADDR=127.0.0.1
> NETMASK=255.0.0.0
> NETWORK=127.0.0.0
> # If you're having problems with gated making 127.0.0.0/8 a martian,
> # you can change this to something else (255.255.255.255, for example)
> BROADCAST=127.255.255.255
> ONBOOT=yes
> NAME=loopback
> 
> 
> Here is my Ubuntu 20/22 network config: (Most Clusters)
> 
> root@cs-kvm01:~# cat /etc/netplan/00-installer-config.yaml
> network:
>  version: 2
>  ethernets:
>eno1: {}
>eno2: {}
>ens2f0:
>  mtu: 1500
>ens2f1:
>  mtu: 1500
>  bonds:
>bond0:
>  interfaces:
>  - ens2f0
>  - ens2f1
>  mtu: 1500
>  parameters:
>mode: balance-alb
>bond1:
>  inter

Re: SSVM routing issue

[Antoine Boucher]

Hi Stanley,

You will find the answers below.



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”


> On May 24, 2023, at 8:59 PM, Stanley Burkee  wrote:
> 
> Hi Antoine,
> 
> 
> Please share the cloudstack version you are using. Also check if you have
> connectivity between your management network & storage network.

4.17.2.0




> 
> Please share the management server logs & your zone cloudbr0 & other
> interfaces configurations.

Here is my CentOS network config: (Management Server and Some Clusters)

[root@nimbus network-scripts]# cat ifcfg*
DEVICE=bond0
ONBOOT=yes
BONDING_OPTS="mode=6"
BRIDGE=cloudbr0
NM_CONTROLLED=no


DEVICE=bond0.53
VLAN=yes
BOOTPROTO=static
ONBOOT=yes
TYPE=Unknown
BRIDGE=cloudbr53


DEVICE=bond1
ONBOOT=yes
BONDING_OPTS="mode=6"
BRIDGE=cloudbr1
NM_CONTROLLED=no


DEVICE=cloudbr0
ONBOOT=yes
TYPE=Bridge
IPADDR=10.101.2.40
NETMASK=255.255.252.0
GATEWAY=10.101.0.1
DOMAIN="haltondc.net"
DEFROUTE=yes
NM_CONTROLLED=no
DELAY=0


DEVICE=cloudbr1
ONBOOT=yes
TYPE=Bridge
NM_CONTROLLED=no
DELAY=0


DEVICE=cloudbr53
ONBOOT=yes
TYPE=Bridge
VLAN=yes
IPADDR=10.101.6.40
#GATEWAY=10.101.6.1
NETMASK=255.255.254.0
NM_CONTROLLED=no
DELAY=0


DEVICE=eno1
TYPE=Ethernet
USERCTL=no
MASTER=bond1
SLAVE=yes
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes

DEVICE=eno2
TYPE=Ethernet
USERCTL=no
MASTER=bond1
SLAVE=yes
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes
DEVICE=eno3
TYPE=Ethernet
USERCTL=no
MASTER=bond1
SLAVE=yes
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes
DEVICE=eno4
TYPE=Ethernet
USERCTL=no
#MASTER=bond1
#SLAVE=yes
#BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=no
DEVICE=ens2f0 
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes


DEVICE=ens2f1
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes


DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback


Here is my Ubuntu 20/22 network config: (Most Clusters)

root@cs-kvm01:~# cat /etc/netplan/00-installer-config.yaml 
network:
  version: 2
  ethernets:
eno1: {}
eno2: {}
ens2f0:
  mtu: 1500
ens2f1:
  mtu: 1500
  bonds:
bond0:
  interfaces:
  - ens2f0
  - ens2f1
  mtu: 1500
  parameters:
mode: balance-alb
bond1:
  interfaces:
  - eno1
  - eno2
  nameservers:
addresses: []
search: []
  parameters:
mode: balance-alb
  vlans:
bond0.53:
  id: 53
  link: bond0
  mtu: 1500
  bridges:
cloudbr0:
  interfaces: [bond0]
  mtu: 1500
  addresses:
  - 10.101.2.42/22
  gateway4: 10.101.0.1
  nameservers:
addresses:
- 10.101.0.1
search:
- haltondc.net
- haltondc.com
  dhcp4: no
  dhcp6: no
cloudbr1:
  interfaces: [bond1]
  mtu: 1500
  dhcp4: no
  dhcp6: no
cloudbr53:
  interfaces: [bond0.53]
  mtu: 1500
  addresses:
  - 10.101.6.42/23
  dhcp4: no
  dhcp6: no



> 
> Thanks.
> 
> Best Regards
> Stanley
> 
> On Mon, 15 May 2023, 6:00 pm Antoine Boucher,  <mailto:antoi...@haltondc.com>> wrote:
> 
>> Hello,
>> 
>> Would anyone have clues on my on going SSVM issue below?
>> 
>> However, I can work around the issue by deleting my Storage Network
>> traffic definition and recreating the SSVM..
>> 
>> What would be the impact of deleting the Storage Network traffic
>> definition on other part of the system? My Primary Storage configuration
>> seems to all be done part of my hosts static configuration.
>> 
>> Regards,
>> Antoine
>> 
>> 
>>> On May 11, 2023, at 10:27 AM, Antoine Boucher 
>> wrote:
>>> 
>>> Good morning/afternoon/evening,
>>> 
>>> I am following up with my SSVM routing issue when a Storage Network is
>> defined.
>>> 
>>> I have a zone with Xen and KVM servers that have a Storage Network
>> defined as Cloudbr53 with a storage network-specific subnet (Cloudbr0 is
>> also defined for Management and Cloudbr1 for Guests)
>>> 
>>> The Cloudbr53 bridge is “hard coded” to VLAN 53 on all hosts within the
>> specific storage ip subnet range. The Storage traffic type for the Zone is
>> defined with Cloudbr53 and VLAN as blank.
>>> 
>>> You will see that the storage network route on the SSVM is pointed to
>> the wrong eth1 interface as it should be eth3
>>> 
>>> 10.101.6.0cloudrouter01.n 255.255.254.0   UG   00 0 eth1
>>> 
>>> root@s-394-VM:~# rou

Re: SSVM routing issue

[Antoine Boucher]

Hello,

Would anyone have clues on my on going SSVM issue below?  

However, I can work around the issue by deleting my Storage Network traffic 
definition and recreating the SSVM..

What would be the impact of deleting the Storage Network traffic definition on 
other part of the system? My Primary Storage configuration seems to all be done 
part of my hosts static configuration.

Regards,
Antoine


> On May 11, 2023, at 10:27 AM, Antoine Boucher  wrote:
> 
> Good morning/afternoon/evening,
> 
> I am following up with my SSVM routing issue when a Storage Network is 
> defined.
> 
> I have a zone with Xen and KVM servers that have a Storage Network defined as 
> Cloudbr53 with a storage network-specific subnet (Cloudbr0 is also defined 
> for Management and Cloudbr1 for Guests)
> 
> The Cloudbr53 bridge is “hard coded” to VLAN 53 on all hosts within the 
> specific storage ip subnet range. The Storage traffic type for the Zone is 
> defined with Cloudbr53 and VLAN as blank. 
> 
> You will see that the storage network route on the SSVM is pointed to the 
> wrong eth1 interface as it should be eth3
> 
> 10.101.6.0cloudrouter01.n 255.255.254.0   UG   00 0 eth1
> 
> root@s-394-VM:~# route
> Kernel IP routing table
> DestinationGateway  Genmask  Flags Metric Ref   Use Iface
> default  148.59.36.49   0.0.0.0  UG   00 0 eth2
> 10.0.0.0 cloudrouter01.n 255.0.0.0 UG   00 0 eth1
> 10.91.0.0 cloudrouter01.n 255.255.254.0   UG   00 0 eth1
> 10.91.6.0 cloudrouter01.n 255.255.255.0   UG   00 0 eth1
> 10.101.0.00.0.0.0  255.255.252.0   U00 0 eth1
> nimbus.haltondc 10.101.6.1255.255.255.255 UGH   00 0 eth3
> 10.101.6.0cloudrouter01.n 255.255.254.0   UG   00 0 eth1
> 148.59.36.48   0.0.0.0  255.255.255.240 U00 0 eth2
> link-local0.0.0.0  255.255.0.0U00 0 eth0
> 172.16.0.0cloudrouter01.n 255.240.0.0UG   00 0 eth1
> 192.168.0.0cloudrouter01.n 255.255.0.0UG   00 0 eth1
> 
> 
> I also tried to define the storage traffic type with VLAN 53; the VLAN/VNI 
> column shows blank, but It looks to be changing the routing to eth3; however, 
> I experienced the same overall communication issue. When communicating to the 
> management network is from the source IP on the storage network and dies 
> coming back since I have no routing between the two networks.
> 
> However, as a workaround, if I remove the storage traffic definition on the 
> Zone, all traffic will be routed through the management network. All is well 
> if I allow my secondary storage (NFS) on the management network.
> 
> 
> 
> I’m using the host-configured “storage network” for primary storage on all my 
> Zones without issues.
> 
> What would be the potential issues of deleting the Storage Network definition 
> traffic type in my zones, assuming I would keep all my secondary storage on 
> or accessible on the management network and recreating the SSVMs?
> 
> Is the storage definition only or mainly used for the SSVM?  
> 
> Regards,
> Antoine 
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.
> 
> 
>> On Feb 28, 2023, at 11:39 AM, Antoine Boucher  wrote:
>> 
>> # root@s-340-VM:~# cat /var/cache/cloud/cmdline
>> 
>> template=domP type=secstorage host=10.101.2.40 port=8250 name=s-340-VM 
>> zone=1 pod=1 guid=s-340-VM workers=5 authorized_key=
>> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource 
>> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 
>> eth2ip=148.59.36.60 eth2mask=255.255.255.240 gateway=148.59.36.49 
>> public.network.device=eth2 eth0ip=169.254.211.29 eth0mask=255.255.0.0 
>> eth1ip=10.101.3.231 eth1mask=255.255.252.0 mgmtcidr=10.101.0.0/22 
>> localgw=10.101.0.1 private.network.device=eth1 eth3ip=10.101.7.212 
>> eth3mask=255.255.254.0 storageip=10.101.7.212 storagenetmask=255.255.254.0 
>> storagegateway=10.101.6.1 internaldns1=10.101.0.1 dns1=1.1.1.1 dns2=8.8.8.8 
>> nfsVersion=null keystore_password=*
>> 
>> 
>> # cat /var/log/cloudstack/management/management-server.log.2023-02-*.gz | 
>

Re: SSVM routing issue

[Antoine Boucher]

Good morning/afternoon/evening,

I am following up with my SSVM routing issue when a Storage Network is defined.

I have a zone with Xen and KVM servers that have a Storage Network defined as 
Cloudbr53 with a storage network-specific subnet (Cloudbr0 is also defined for 
Management and Cloudbr1 for Guests)

The Cloudbr53 bridge is “hard coded” to VLAN 53 on all hosts within the 
specific storage ip subnet range. The Storage traffic type for the Zone is 
defined with Cloudbr53 and VLAN as blank. 

You will see that the storage network route on the SSVM is pointed to the wrong 
eth1 interface as it should be eth3

10.101.6.0cloudrouter01.n 255.255.254.0   UG   00 0 eth1

root@s-394-VM:~# route
Kernel IP routing table
DestinationGateway  Genmask  Flags Metric Ref   Use Iface
default  148.59.36.49   0.0.0.0  UG   00 0 eth2
10.0.0.0 cloudrouter01.n 255.0.0.0 UG   00 0 eth1
10.91.0.0 cloudrouter01.n 255.255.254.0   UG   00 0 eth1
10.91.6.0 cloudrouter01.n 255.255.255.0   UG   00 0 eth1
10.101.0.00.0.0.0  255.255.252.0   U00 0 eth1
nimbus.haltondc 10.101.6.1255.255.255.255 UGH   00 0 eth3
10.101.6.0cloudrouter01.n 255.255.254.0   UG   00 0 eth1
148.59.36.48   0.0.0.0  255.255.255.240 U00 0 eth2
link-local0.0.0.0  255.255.0.0U00 0 eth0
172.16.0.0cloudrouter01.n 255.240.0.0UG   00 0 eth1
192.168.0.0cloudrouter01.n 255.255.0.0UG   00 0 eth1


I also tried to define the storage traffic type with VLAN 53; the VLAN/VNI 
column shows blank, but It looks to be changing the routing to eth3; however, I 
experienced the same overall communication issue. When communicating to the 
management network is from the source IP on the storage network and dies coming 
back since I have no routing between the two networks.

However, as a workaround, if I remove the storage traffic definition on the 
Zone, all traffic will be routed through the management network. All is well if 
I allow my secondary storage (NFS) on the management network.



I’m using the host-configured “storage network” for primary storage on all my 
Zones without issues.

What would be the potential issues of deleting the Storage Network definition 
traffic type in my zones, assuming I would keep all my secondary storage on or 
accessible on the management network and recreating the SSVMs?

Is the storage definition only or mainly used for the SSVM?  

Regards,
Antoine 


Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Feb 28, 2023, at 11:39 AM, Antoine Boucher  wrote:
> 
> # root@s-340-VM:~# cat /var/cache/cloud/cmdline
> 
> template=domP type=secstorage host=10.101.2.40 port=8250 name=s-340-VM zone=1 
> pod=1 guid=s-340-VM workers=5 authorized_key=
> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource 
> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 
> eth2ip=148.59.36.60 eth2mask=255.255.255.240 gateway=148.59.36.49 
> public.network.device=eth2 eth0ip=169.254.211.29 eth0mask=255.255.0.0 
> eth1ip=10.101.3.231 eth1mask=255.255.252.0 mgmtcidr=10.101.0.0/22 
> localgw=10.101.0.1 private.network.device=eth1 eth3ip=10.101.7.212 
> eth3mask=255.255.254.0 storageip=10.101.7.212 storagenetmask=255.255.254.0 
> storagegateway=10.101.6.1 internaldns1=10.101.0.1 dns1=1.1.1.1 dns2=8.8.8.8 
> nfsVersion=null keystore_password=*
> 
> 
> # cat /var/log/cloudstack/management/management-server.log.2023-02-*.gz | 
> zgrep SecStorageSetupCommand
> 
> 2023-02-18 14:35:38,699 DEBUG [c.c.a.t.Request] 
> (AgentConnectTaskPool-290:ctx-cf94f90e) (logid:6dc1b961) Seq 
> 47-6546545008336437249: Sending  { Cmd , MgmtId: 130593671224, via: 
> 47(s-292-VM), Ver: v1, Flags: 100111, 
> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.91.6.5/volume1/ACS_Backup06","_role":"Image"}},"secUrl":"nfs://10.91.6.5/volume1/ACS_Backup06","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}]
>  }
> 2023-02-18 14:35:42,024 DEBUG [c.c.a.m.AgentManagerImpl] 
> (AgentConnectT

Failed Volume Migration

[Antoine Boucher]

Hello,

I few days ago we did 3 migrations that eventually failed because of secondary 
storage issues.  However, the volumes entry are duplicated on the Web UI, one 
showing ready and the other one showing migrating.

Could I edit the volume table to remove the migrating entry to remove it from 
the UI?

Regards,
Antoine

Re: SSVM routing issue on /23 storage network

[Antoine Boucher]

6:ctx-50d91205) (logid:4c7783a0) Seq 
52-8409064929230848002: Sending  { Cmd , MgmtId: 130593671224, via: 
52(s-339-VM), Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs://10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-26 14:46:45,435 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentConnectTaskPool-26:ctx-50d91205) (logid:4c7783a0) Details from executing 
class com.cloud.agent.api.SecStorageSetupCommand: success
2023-02-26 15:07:11,934 DEBUG [c.c.a.t.Request] 
(AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Seq 
52-7356067041356283905: Sending  { Cmd , MgmtId: 130593671224, via: 
52(s-339-VM), Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs://10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-26 15:07:14,985 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Details from executing 
class com.cloud.agent.api.SecStorageSetupCommand: success
2023-02-26 15:07:15,001 DEBUG [c.c.a.t.Request] 
(AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Seq 
52-7356067041356283906: Sending  { Cmd , MgmtId: 130593671224, via: 
52(s-339-VM), Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs://10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-26 15:07:17,516 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Details from executing 
class com.cloud.agent.api.SecStorageSetupCommand: success
2023-02-26 16:03:33,807 DEBUG [c.c.a.t.Request] 
(AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Seq 
53-603482350067646465: Sending  { Cmd , MgmtId: 130593671224, via: 
53(s-340-VM), Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs://10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-26 16:03:37,126 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Details from executing 
class com.cloud.agent.api.SecStorageSetupCommand: success
2023-02-26 16:03:37,142 DEBUG [c.c.a.t.Request] 
(AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Seq 
53-603482350067646466: Sending  { Cmd , MgmtId: 130593671224, via: 
53(s-340-VM), Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs://10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-26 16:03:39,890 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Details from executing 
class com.cloud.agent.api.SecStorageSetupCommand: success


Antoine Boucher


> On Feb 28, 2023, at 4:47 AM, Wei ZHOU  wrote:
> 
> The routes should use eth3 not eth1.
> 
> Can you share the `/var/cache/cloud/cmdline` file in SSVM, and filter
> management-server.log by keyword `SecStorageSetupCommand` ?
> 
> 
> -Wei
> 
> On Tue, 28 Feb 2023 at 10:42, Granwille Strauss
> mai

Re: SSVM unable to connect to SecStorage via Storage Network

[Antoine Boucher]

Yes,

Physical 1
Management is on cloudbr0
Storage on cloubr53

Physical 2
Guest and Public on cloudbr1





> On Feb 27, 2023, at 10:11 PM, Simon Weller  wrote:
> 
> Antoine,
> 
> When you defined the  zone, did you place storage on its own traffic label?
> 
> Could you share your zone network config, along with your traffic labels
> and the associated services on each label?
> 
> -Si
> 
> On Mon, Feb 27, 2023, 9:03 PM Antoine Boucher  wrote:
> 
>> Following to my previous message I have decided to simplify my
>> observations.
>> 
>> Looking at the SSVM network and routing.  My storage ip is defined on eth3
>> but the routing of the storage subnet is configured via eth1, the
>> Management Network.
>> 
>> Is this an oversight or a desired behaviour?  The only way to solve the
>> issue is for me to use the Management Network for SecStorage or change the
>> routing through eth3.
>> 
>> I’m assuming that this issue has been reported a number of times?
>> 
>> Regards,
>> Antoine
>> 
>> 

SSVM unable to connect to SecStorage via Storage Network

[Antoine Boucher]

Following to my previous message I have decided to simplify my observations.

Looking at the SSVM network and routing.  My storage ip is defined on eth3 but 
the routing of the storage subnet is configured via eth1, the Management 
Network.

Is this an oversight or a desired behaviour?  The only way to solve the issue 
is for me to use the Management Network for SecStorage or change the routing 
through eth3. 

I’m assuming that this issue has been reported a number of times?

Regards,
Antoine

SSVM routing issue on /23 storage network

[Antoine Boucher]

Hello,

I'm having a networking issue on SSVMs,  I have the following networks defined 
in “Zone 1”.  

Management: 10.101.0.0/22
Storage: 10.101.6.0/23

All worked well until we decided to configure new storage devices on 
10.101.7.x, the hosts and management server have no issue but the SSVM is not 
able to reach it.  Here are the defined interfaces of the SSVM and the routing 
table of the SSVM:

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
link/ether 0e:00:a9:fe:72:ec brd ff:ff:ff:ff:ff:ff
altname enp0s3
altname ens3
inet 169.254.114.236/16 brd 169.254.255.255 scope global eth0
   valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
link/ether 1e:00:a1:00:00:06 brd ff:ff:ff:ff:ff:ff
altname enp0s4
altname ens4
inet 10.101.3.205/22 brd 10.101.3.255 scope global eth1
   valid_lft forever preferred_lft forever
4: eth2:  mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
link/ether 1e:00:6b:00:00:3d brd ff:ff:ff:ff:ff:ff
altname enp0s5
altname ens5
inet 148.59.36.61/28 brd 148.59.36.63 scope global eth2
   valid_lft forever preferred_lft forever
5: eth3:  mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
link/ether 1e:00:09:00:00:6b brd ff:ff:ff:ff:ff:ff
altname enp0s6
altname ens6
inet 10.101.7.226/23 brd 10.101.7.255 scope global eth3
   valid_lft forever preferred_lft forever

default via 148.59.36.49 dev eth2 
10.0.0.0/8 via 10.101.0.1 dev eth1 
10.91.0.0/23 via 10.101.0.1 dev eth1 
10.91.6.0/24 via 10.101.0.1 dev eth1 
10.101.0.0/22 dev eth1 proto kernel scope link src 10.101.3.253 
10.101.6.0/23 via 10.101.0.1 dev eth1 
148.59.36.48/28 dev eth2 proto kernel scope link src 148.59.36.61 
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.232.208 
172.16.0.0/12 via 10.101.0.1 dev eth1 
192.168.0.0/16 via 10.101.0.1 dev eth1 

Why is the routing for 10.101.6.0/23 routing via eth1, shoudn’nt it be using 
eth3?  The router seems to be bypassing the routing rules for 10.101.6.x since 
I see no traffic going through the gateway but I see traffic going through the 
gateway when the destination is 10.101.7.x

If I modify the routing for 10.101.6.0/23 to eth3 all is well.

Is this by design?

Regards,
Antoine Boucher

More context information for alerts messages

[Antoine Boucher]

Hello,

This is the typical alert message we receive. It would be very useful, if not 
essential, to have more context information in messages, such as account name 
or project name in the below example, in order to take timely action. 

"Template 06cfc10e-a07d-4cee-8929-ca98946c60ab failed to upload. Error details: 
Maximum number of resources of type secondary_storage for account/project has 
exceeded”


Regards,
Antoine

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

Hi Wei,

Thank you for your information.  It turns out that the host was not properly 
updated to 4.17.2

All work now.

Thanks again!


Regards,
Antoine


> On Feb 22, 2023, at 7:51 AM, Wei ZHOU  wrote:
> 
> The scripts are transferred from kvm host via scp command. It would be good
> to check
> - kvm host has cloudstack-agent installed and upgraded
> - scp to ssvm/router vms via cloud0 is allowed on kvm host
> 
> -Wei
> 
> 
> 
> On Wednesday, 22 February 2023, Antoine Boucher  <mailto:antoi...@haltondc.com>>
> wrote:
> 
>> Is there any reason why the /usr/local/cloud/systemvm directory would be
>> missing from the SSVM one one zone but not on an other?
>> 
>> Is the directory added during the systemvm creation or is it part of the
>> base system VM template?
>> 
>> Both /etc/cloudstack-release show: Cloudstack Release 4.17.2 Fri 09 Dec
>> 2022 12:51:18 PM UTC
>> 
>> Regards,
>> Antoine
>> 
>> 
>> 
>>> On Feb 21, 2023, at 3:31 PM, Antoine Boucher 
>> wrote:
>>> 
>>> I compared the system template fro KVM of both zone and they seem
>> identical.
>>> 
>>> 
>>> The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist
>> but the /usr/local/cloud/ directory of the zone 2 SSVM does not exist?
>>> 
>>> 
>>> 
>>> Antoine Boucher
>>> antoi...@haltondc.com
>>> [o] +1-226-505-9734
>>> www.haltondc.com
>>> 
>>> “Data security made simple”
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy, copying,
>> circulation or other use of this message and any attachments is strictly
>> prohibited. If you are not the intended recipient, please notify the sender
>> immediately by return e-mail, and delete this message and any attachments
>> from your system.
>>> 
>>> 
>>>> On Feb 21, 2023, at 1:31 PM, Antoine Boucher 
>> wrote:
>>>> 
>>>> 4.17.2
>>>> 
>>>> Antoine Boucher
>>>> antoi...@haltondc.com
>>>> [o] +1-226-505-9734
>>>> www.haltondc.com
>>>> 
>>>> “Data security made simple and affordable”
>>>> 
>>>> 
>>>> On Feb 21, 2023, at 11:51, Wei ZHOU  wrote:
>>>> 
>>>> can you check the /etc/cloudstack-release in ssvm ?
>>>> 
>>>> -Wei
>>>> 
>>>> On Tuesday, 21 February 2023, Antoine Boucher > <mailto:antoi...@haltondc.com>> wrote:
>>>>> Here are the logs from daemon.log of the failing SSVM
>>>>> 
>>>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to
>> locate executable /usr/local/cloud/systemvm/_run.sh: No such file or
>> directory
>>>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step
>> EXEC spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
>>>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process
>> exited, code=exited, status=203/EXEC
>>>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result
>> 'exit-code'.
>>>>> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service:
>> Succeeded.
>>>>> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System
>> Runlevel Changes.
>>>>> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s
>> (kernel) + 2min 16.468s (userspace) = 2min 18.754s.
>>>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart
>> job, restart counter is at 1.
>>>>> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
>>>>> 
>>>>> This repeats 5 times and fails
>>>>> 
>>>>> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Antoine Boucher
>>>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com> 
>>>>> <mailto:antoi...@haltondc.com>
>>>>> [o] +1-226-505-9734
>>>>> www.haltondc.com <http://www.haltondc.com/> <http://www.haltondc.com/>
>>>>&

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

Is there any reason why the /usr/local/cloud/systemvm directory would be 
missing from the SSVM one one zone but not on an other?

Is the directory added during the systemvm creation or is it part of the base 
system VM template?

Both /etc/cloudstack-release show: Cloudstack Release 4.17.2 Fri 09 Dec 2022 
12:51:18 PM UTC

Regards,
Antoine



> On Feb 21, 2023, at 3:31 PM, Antoine Boucher  wrote:
> 
> I compared the system template fro KVM of both zone and they seem identical.  
> 
> 
> The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist but 
> the /usr/local/cloud/ directory of the zone 2 SSVM does not exist?
> 
> 
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
> 
> “Data security made simple”
> 
> 
> 
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.
> 
> 
>> On Feb 21, 2023, at 1:31 PM, Antoine Boucher  wrote:
>> 
>> 4.17.2
>> 
>> Antoine Boucher
>> antoi...@haltondc.com
>> [o] +1-226-505-9734
>> www.haltondc.com
>> 
>> “Data security made simple and affordable”
>> 
>> 
>> On Feb 21, 2023, at 11:51, Wei ZHOU  wrote:
>> 
>> can you check the /etc/cloudstack-release in ssvm ?
>> 
>> -Wei
>> 
>> On Tuesday, 21 February 2023, Antoine Boucher > <mailto:antoi...@haltondc.com>> wrote:
>>> Here are the logs from daemon.log of the failing SSVM
>>> 
>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate 
>>> executable /usr/local/cloud/systemvm/_run.sh: No such file or directory
>>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC 
>>> spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, 
>>> code=exited, status=203/EXEC
>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result 
>>> 'exit-code'.
>>> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: 
>>> Succeeded.
>>> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System 
>>> Runlevel Changes.
>>> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + 
>>> 2min 16.468s (userspace) = 2min 18.754s.
>>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, 
>>> restart counter is at 1.
>>> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
>>> 
>>> This repeats 5 times and fails
>>> 
>>> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.  
>>> 
>>> 
>>> 
>>> 
>>> Antoine Boucher
>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com>
>>> [o] +1-226-505-9734
>>> www.haltondc.com <http://www.haltondc.com/>
>>> 
>>> “Data security made simple”
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Confidentiality Warning: This message and any attachments are intended only 
>>> for the use of the intended recipient(s), are confidential, and may be 
>>> privileged. If you are not the intended recipient, you are hereby notified 
>>> that any review, retransmission, conversion to hard copy, copying, 
>>> circulation or other use of this message and any attachments is strictly 
>>> prohibited. If you are not the intended recipient, please notify the sender 
>>> immediately by return e-mail, and delete this message and any attachments 
>>> from your system.
>>> 
>>> 
>>>> On Feb 21, 2023, at 7:53 AM, Antoine Boucher >>> <mailto:antoi...@haltondc.com>> wrote:
>>>> 
>>>> They are not old they are the new ones.  I have deleted them a number of 
>>>> times after the upgrade but the issue prevails.  
>>>> 
>>>> Antoine Boucher
>>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com>
>>>> [o] +1-226-505-9734
>>>> www.haltondc.com <http://www.haltondc.com/>
>>>> 
>>

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

I compared the system template fro KVM of both zone and they seem identical.  


The /usr/local/cloud/systemvm/ directory exist on SSVM on zone 1 exist but the 
/usr/local/cloud/ directory of the zone 2 SSVM does not exist?



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Feb 21, 2023, at 1:31 PM, Antoine Boucher  wrote:
> 
> 4.17.2
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
> 
> “Data security made simple and affordable”
> 
> 
> On Feb 21, 2023, at 11:51, Wei ZHOU  wrote:
> 
> can you check the /etc/cloudstack-release in ssvm ?
> 
> -Wei
> 
> On Tuesday, 21 February 2023, Antoine Boucher  <mailto:antoi...@haltondc.com>> wrote:
>> Here are the logs from daemon.log of the failing SSVM
>> 
>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate 
>> executable /usr/local/cloud/systemvm/_run.sh: No such file or directory
>> Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC 
>> spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, 
>> code=exited, status=203/EXEC
>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result 
>> 'exit-code'.
>> Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: 
>> Succeeded.
>> Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System 
>> Runlevel Changes.
>> Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + 
>> 2min 16.468s (userspace) = 2min 18.754s.
>> Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, 
>> restart counter is at 1.
>> Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.
>> 
>> This repeats 5 times and fails
>> 
>> The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.  
>> 
>> 
>> 
>> 
>> Antoine Boucher
>> antoi...@haltondc.com <mailto:antoi...@haltondc.com>
>> [o] +1-226-505-9734
>> www.haltondc.com <http://www.haltondc.com/>
>> 
>> “Data security made simple”
>> 
>> 
>> 
>> 
>> 
>> Confidentiality Warning: This message and any attachments are intended only 
>> for the use of the intended recipient(s), are confidential, and may be 
>> privileged. If you are not the intended recipient, you are hereby notified 
>> that any review, retransmission, conversion to hard copy, copying, 
>> circulation or other use of this message and any attachments is strictly 
>> prohibited. If you are not the intended recipient, please notify the sender 
>> immediately by return e-mail, and delete this message and any attachments 
>> from your system.
>> 
>> 
>>> On Feb 21, 2023, at 7:53 AM, Antoine Boucher >> <mailto:antoi...@haltondc.com>> wrote:
>>> 
>>> They are not old they are the new ones.  I have deleted them a number of 
>>> times after the upgrade but the issue prevails.  
>>> 
>>> Antoine Boucher
>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com>
>>> [o] +1-226-505-9734
>>> www.haltondc.com <http://www.haltondc.com/>
>>> 
>>> “Data security made simple and affordable”
>>> 
>>> 
>>> On Feb 21, 2023, at 00:24, Rohit Yadav >> <mailto:rohit.ya...@shapeblue.com>> wrote:
>>> 
>>> 
>>> You can destroy these old systemvms or do a stop and start on them. 
>>> 
>>> Regards.
>>> 
>>>   
>>> 
>>>   
>>> From: Antoine Boucher mailto:antoi...@haltondc.com>>
>>> Sent: Tuesday, February 21, 2023 10:40:46 AM
>>> To: users mailto:users@cloudstack.apache.org>>
>>> Subject: Failed to setup certificates for system vm error on 2nd Zone after 
>>> upgrading to 4.17.2
>>>  
>>> After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and 
>>> VRs) of the first zone upgraded without 

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

Here are the logs from daemon.log of the failing SSVM

Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed to locate 
executable /usr/local/cloud/systemvm/_run.sh: No such file or directory
Feb 21 02:36:16 systemvm systemd[1963]: cloud.service: Failed at step EXEC 
spawning /usr/local/cloud/systemvm/_run.sh: No such file or directory
Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Main process exited, 
code=exited, status=203/EXEC
Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Failed with result 
'exit-code'.
Feb 21 02:36:16 systemvm systemd[1]: systemd-update-utmp-runlevel.service: 
Succeeded.
Feb 21 02:36:16 systemvm systemd[1]: Finished Update UTMP about System Runlevel 
Changes.
Feb 21 02:36:16 systemvm systemd[1]: Startup finished in 2.285s (kernel) + 2min 
16.468s (userspace) = 2min 18.754s.
Feb 21 02:36:16 systemvm systemd[1]: cloud.service: Scheduled restart job, 
restart counter is at 1.
Feb 21 02:36:16 systemvm systemd[1]: Stopped CloudStack Agent service.

This repeats 5 times and fails

The /usr/local/cloud/systemvm/ directory does not exist on the SSVM.  




Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Feb 21, 2023, at 7:53 AM, Antoine Boucher  wrote:
> 
> They are not old they are the new ones.  I have deleted them a number of 
> times after the upgrade but the issue prevails.  
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
> 
> “Data security made simple and affordable”
> 
> 
> On Feb 21, 2023, at 00:24, Rohit Yadav  wrote:
> 
> 
> You can destroy these old systemvms or do a stop and start on them. 
> 
> Regards.
> 
>   
> 
>   
> From: Antoine Boucher 
> Sent: Tuesday, February 21, 2023 10:40:46 AM
> To: users 
> Subject: Failed to setup certificates for system vm error on 2nd Zone after 
> upgrading to 4.17.2
>  
> After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and 
> VRs) of the first zone upgraded without issues but the system VMs (s-318-VM 
> and v-317-VM) of the second zone (Kitchener1) are no longer able to establish 
> connection with the management server.  I have rebooted the host they are on, 
> deleted the system vm to be recreated, I also logged into the two system VM 
> and found nothing so far except fo the following management server logs:
> 
> 2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] 
> (Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) 
> (logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM 
> 2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] 
> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
> (logid:352477d9) Failed to setup certificate for system vm: v-317-VM
> 
> The host of these 2 system VM was missed in the original update, such that is 
> was updated after the Management Server was running with 4.17.2 instead of 
> before.
> 
> Would anyone have suggestion to resolve the issue?
> 
> 
> 
> 
> 
> 2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request] 
> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
> (logid:352477d9) Seq 44-1267200345151373322: Received:  { Ans: , MgmtId: 
> 130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer, 
> CheckSshAnswer } }
> 2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl] 
> (Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
> (logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid: 
> "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state transited 
> from [Starting] to [Running] with event [OperationSucceeded]. VM's original 
> host: Host {"id": "44", "name": "kit1-kvm03", "uuid": 
> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host 
> {"id": "44", "name": "kit1-kvm03", "uuid": 
> "3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, host before state 
> transition: Host {"id": "44", "name": "kit1-kvm03", "uuid&qu

Re: Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

They are not old they are the new ones.  I have deleted them a number of times 
after the upgrade but the issue prevails.  

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Feb 21, 2023, at 00:24, Rohit Yadav  wrote:


You can destroy these old systemvms or do a stop and start on them. 

Regards.

  

  

From: Antoine Boucher 
Sent: Tuesday, February 21, 2023 10:40:46 AM
To: users 
Subject: Failed to setup certificates for system vm error on 2nd Zone after 
upgrading to 4.17.2
 
After upgrading my two zone ACS from 4.16.2 to 4.17.2, the system VM (and VRs) 
of the first zone upgraded without issues but the system VMs (s-318-VM and 
v-317-VM) of the second zone (Kitchener1) are no longer able to establish 
connection with the management server.  I have rebooted the host they are on, 
deleted the system vm to be recreated, I also logged into the two system VM and 
found nothing so far except fo the following management server logs:

2023-02-20 21:34:10,063 ERROR [c.c.v.VirtualMachineManagerImpl] 
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) 
(logid:5d1ede3b) Failed to setup certificate for system vm: s-318-VM 
2023-02-20 21:34:10,747 ERROR [c.c.v.VirtualMachineManagerImpl] 
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
(logid:352477d9) Failed to setup certificate for system vm: v-317-VM

The host of these 2 system VM was missed in the original update, such that is 
was updated after the Management Server was running with 4.17.2 instead of 
before.

Would anyone have suggestion to resolve the issue?





2023-02-20 21:34:09,599 DEBUG [c.c.a.t.Request] 
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
(logid:352477d9) Seq 44-1267200345151373322: Received:  { Ans: , MgmtId: 
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 10, { StartAnswer, 
CheckSshAnswer } }
2023-02-20 21:34:09,623 DEBUG [c.c.c.CapacityManagerImpl] 
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
(logid:352477d9) VM instance {id: "317", name: "v-317-VM", uuid: 
"0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"} state transited 
from [Starting] to [Running] with event [OperationSucceeded]. VM's original 
host: Host {"id": "44", "name": "kit1-kvm03", "uuid": 
"3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, new host: Host 
{"id": "44", "name": "kit1-kvm03", "uuid": 
"3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}, host before state 
transition: Host {"id": "44", "name": "kit1-kvm03", "uuid": 
"3bde1185-8db6-414a-9c59-f788a2e0e9cf", "type"="Routing"}
2023-02-20 21:34:09,625 DEBUG [c.c.a.t.Request] 
(Work-Job-Executor-15:ctx-34675425 job-20367/job-20401 ctx-d67c7cd1) 
(logid:5d1ede3b) Seq 44-1267200345151373326: Sending  { Cmd , MgmtId: 
130593671224, via: 44(kit1-kvm03), Ver: v1, Flags: 100011, 
[{"org.apache.cloudstack.ca.SetupCertificateCommand":{"handleByAgent":"false","accessDetails":{"router.name":"s-318-VM","Control":"169.254.254.63","Storage":"10.91.6.249","Public":"64.32.40.68","Management":"10.91.0.244","router.ip":"169.254.254.63"},"wait":"60","bypassHostMaintenance":"false"}}]
 }
2023-02-20 21:34:09,626 DEBUG [c.c.v.VirtualMachineManagerImpl] 
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400 ctx-d7f724f4) 
(logid:352477d9) Start completed for VM VM instance {id: "317", name: 
"v-317-VM", uuid: "0fbbd751-3bbe-46f6-bec0-e2887516fa6a", type="ConsoleProxy"}
2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) 
(logid:) Seq 44-1267200345151373326: Processing:  { Ans: , MgmtId: 
130593671224, via: 44, Ver: v1, Flags: 10, 
[{"org.apache.cloudstack.ca.SetupCertificateAnswer":{"result":"false","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-20 21:34:10,063 DEBUG [c.c.a.t.Request] 
(Work-Job-Exe

Re: Expired Libvirt certificate on CentOS 7 KVM host.

[Antoine Boucher]

My ca.framework.cert.validity.period is set to 365 days.

If assume that the Libvirt certificate expires in a year should I set 
ca.framework.cert.validity.period to be less than 365, say 360?

Regards,
Antoine



> On Feb 20, 2023, at 11:54 PM, Rohit Yadav  wrote:
> 
> You can configure them using the available global settings 
> ca.framework.cert.validity.period
> 
> By default the auto renewal is set to true. Read more here
> https://www.shapeblue.com/cloudstack-ca-framework/
> and
> http://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#security
> 
> Regards.
> 
> Regards.
> ________
> From: Antoine Boucher mailto:antoi...@haltondc.com>>
> Sent: Tuesday, February 21, 2023 7:45:55 AM
> To: users mailto:users@cloudstack.apache.org>>
> Subject: Re: Expired Libvirt certificate on CentOS 7 KVM host.
> 
> Excellent Wei,
> 
> I set listen_tls to 0, started Libirtd and cloudstack-agent.  The host 
> connected as unsecured, I did a “Provision Host Security Keys” and all is 
> well.
> 
> Thanks again,
> Antoine
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.
> 
> 
> 
> 
> 
>> On Feb 20, 2023, at 4:03 PM, Wei ZHOU  wrote:
>> 
>> You can refer to this code block
>> 
>> https://github.com/apache/cloudstack/blob/main/scripts/util/keystore-setup#L54-L61
>> 
>> 
>> if [ -f "$LIBVIRTD_FILE" ]; then
>>   echo "Reverting libvirtd to not listen on TLS"
>>   sed -i "s,^listen_tls=1,listen_tls=0,g" $LIBVIRTD_FILE
>>   systemctl restart libvirtd
>>   fi
>> 
>>   echo "Removing cloud.* files in /etc/cloudstack/agent"
>>   rm -f /etc/cloudstack/agent/cloud.*
>> 
>> 
>> -Wei
>> 
>> 
>> On Monday, 20 February 2023, Antoine Boucher  wrote:
>> 
>>> Thank you Wei,
>>> 
>>> My ca.plugin.root.auth.strictness was already set to false
>>> 
>>> The cloud-stack agent refused to run because Libvirt is not running
>>> because of the expired Libvirt certs.
>>> 
>>> Is there a way to turn off the secure connection requirement on libbvirt.
>>> Or at least to allow to connect and renew vie the WebUI and the turn it
>>> back on?
>>> 
>>> Regards,
>>> Antoine
>>> 
>>> 
>>> 
>>> *Antoine Boucher*
>>> antoi...@haltondc.com
>>> [o] +1-226-505-9734
>>> www.haltondc.com <http://www.haltondc.com/><http://www.haltondc.com 
>>> <http://www.haltondc.com/>>
>>> 
>>> “Data security made simple”
>>> 
>>> 
>>> [image: HDClogo7-small.png]
>>> 
>>> 
>>> Confidentiality Warning: This message and any attachments are intended
>>> only for the use of the intended recipient(s), are confidential, and may be
>>> privileged. If you are not the intended recipient, you are hereby notified
>>> that any review, retransmission, conversion to hard copy,
>>> copying, circulation or other use of this message and any attachments is
>>> strictly prohibited. If you are not the intended recipient, please notify
>>> the sender immediately by return e-mail, and delete this message and any
>>> attachments from your system.
>>> 
>>> 
>>> On Feb 20, 2023, at 2:24 PM, Wei ZHOU  wrote:
>>> 
>>> Agree.
>>> 
>>> For the cloudstack agent which can not be started, update global setting
>>> `ca.plugin.root.auth.strictness` to `false` and retry.
>>> 
>>> -Wei
>>> 
>>> On Mon, 20 Feb 2023 at 20:21, Aditya Sharma
>>>  wrote:
>>> 
>>> 
>>> Hello,
>>> 
>>> Yes it can be done simply by forcing “provision host security keys“ from
>>> the Web UI.
>>> 
>>> Regards,
>>> Aditya Sharma
>>> 
>>> On 21-Feb-2023, at 00:01, Antoine Boucher  wrote:
>>> 
>>> Hello,
>>> 
>>> I have just upgraded from 4.16.2 to 4.17.2 all went well.
>>> 
>>> However

Failed to setup certificates for system vm error on 2nd Zone after upgrading to 4.17.2

[Antoine Boucher]

v.VmWorkJobDispatcher] 
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) Done 
with run of VM work job: com.cloud.vm.VmWorkStart for VM 317, job origin: 20366
2023-02-20 21:34:10,764 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(Work-Job-Executor-14:ctx-502066d6 job-20366/job-20400) (logid:352477d9) Remove 
job-20400 from job monitoring
2023-02-20 21:34:10,773 INFO  [c.c.c.ConsoleProxyManagerImpl] 
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy v-317-VM is started
2023-02-20 21:34:10,773 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Processing Seq 
45-791665:  { Cmd , MgmtId: -1, via: 45, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"290","_loadInfo":"{
  "connections": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2023-02-20 21:34:10,773 DEBUG [c.c.a.ConsoleProxyAlertAdapter] 
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) received console proxy alert
2023-02-20 21:34:10,775 DEBUG [c.c.a.ConsoleProxyAlertAdapter] 
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) Console proxy is up, zone: 
Kitchener1, proxy: v-317-VM
2023-02-20 21:34:10,776 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-2:null) (logid:) SeqA 45-791665: Sending Seq 45-791665:  
{ Ans: , MgmtId: 130593671224, via: 45, Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
 }
2023-02-20 21:34:10,776 WARN  [c.c.a.AlertManagerImpl] 
(consoleproxy-1:ctx-2caa6d85) (logid:5554e671) alertType=[10] dataCenterId=[4] 
podId=[4] clusterId=[null] message=[Console proxy up in zone: Kitchener1, 
proxy: v-317-VM, public IP: 64.32.40.66, private IP: 10.91.0.237].



Regards,
Antoine Boucher

Re: Expired Libvirt certificate on CentOS 7 KVM host.

[Antoine Boucher]

Excellent Wei,

I set listen_tls to 0, started Libirtd and cloudstack-agent.  The host 
connected as unsecured, I did a “Provision Host Security Keys” and all is well.

Thanks again,
Antoine


Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Feb 20, 2023, at 4:03 PM, Wei ZHOU  wrote:
> 
> You can refer to this code block
> 
> https://github.com/apache/cloudstack/blob/main/scripts/util/keystore-setup#L54-L61
> 
> 
> if [ -f "$LIBVIRTD_FILE" ]; then
>echo "Reverting libvirtd to not listen on TLS"
>sed -i "s,^listen_tls=1,listen_tls=0,g" $LIBVIRTD_FILE
>systemctl restart libvirtd
>fi
> 
>echo "Removing cloud.* files in /etc/cloudstack/agent"
>rm -f /etc/cloudstack/agent/cloud.*
> 
> 
> -Wei
> 
> 
> On Monday, 20 February 2023, Antoine Boucher  wrote:
> 
>> Thank you Wei,
>> 
>> My ca.plugin.root.auth.strictness was already set to false
>> 
>> The cloud-stack agent refused to run because Libvirt is not running
>> because of the expired Libvirt certs.
>> 
>> Is there a way to turn off the secure connection requirement on libbvirt.
>> Or at least to allow to connect and renew vie the WebUI and the turn it
>> back on?
>> 
>> Regards,
>> Antoine
>> 
>> 
>> 
>> *Antoine Boucher*
>> antoi...@haltondc.com
>> [o] +1-226-505-9734
>> www.haltondc.com
>> 
>> “Data security made simple”
>> 
>> 
>> [image: HDClogo7-small.png]
>> 
>> 
>> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy,
>> copying, circulation or other use of this message and any attachments is
>> strictly prohibited. If you are not the intended recipient, please notify
>> the sender immediately by return e-mail, and delete this message and any
>> attachments from your system.
>> 
>> 
>> On Feb 20, 2023, at 2:24 PM, Wei ZHOU  wrote:
>> 
>> Agree.
>> 
>> For the cloudstack agent which can not be started, update global setting
>> `ca.plugin.root.auth.strictness` to `false` and retry.
>> 
>> -Wei
>> 
>> On Mon, 20 Feb 2023 at 20:21, Aditya Sharma
>>  wrote:
>> 
>> 
>> Hello,
>> 
>> Yes it can be done simply by forcing “provision host security keys“ from
>> the Web UI.
>> 
>> Regards,
>> Aditya Sharma
>> 
>> On 21-Feb-2023, at 00:01, Antoine Boucher  wrote:
>> 
>> Hello,
>> 
>> I have just upgraded from 4.16.2 to 4.17.2 all went well.
>> 
>> However, probably unrelated to the upgrade, I needed to do maintenance
>> 
>> on on of my Centos 7 kvm host. When I rebooted the host CloudStack agent
>> can not start, complaining about expired libvirt certificated.
>> 
>> 
>> I read that the certificate for libvirt of centos 7 is valid for one
>> 
>> year.  There is a fairly convoluted way to update them.  Is there a simpler
>> way to renew the cert?
>> 
>> 
>> I have not rebooted my other centos 7 kvm hosts, that are likely over
>> 
>> the one year mark.  Can these hosts libvirt certs be upgraded simply by
>> forcing “provision host security keys“ from the webui console in the
>> infrastructure/host section since I still have cloud-agent connection?
>> 
>> 
>> Regards,
>> Antoine Boucher
>> 
>> 
>> 
>> 
>> 
>> 
>> Confidentiality Warning: This message and any attachments are intended
>> 
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy, copying,
>> circulation or other use of this message and any attachments is strictly
>> prohibited. If you are not the intended recipient, please notify the sender
>> immediately by return e-mail, and delete this message an

Re: Expired Libvirt certificate on CentOS 7 KVM host.

[Antoine Boucher]

Thank you Wei,

My ca.plugin.root.auth.strictness was already set to false

The cloud-stack agent refused to run because Libvirt is not running because of 
the expired Libvirt certs.

Is there a way to turn off the secure connection requirement on libbvirt.  Or 
at least to allow to connect and renew vie the WebUI and the turn it back on?

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Feb 20, 2023, at 2:24 PM, Wei ZHOU  wrote:
> 
> Agree.
> 
> For the cloudstack agent which can not be started, update global setting
> `ca.plugin.root.auth.strictness` to `false` and retry.
> 
> -Wei
> 
> On Mon, 20 Feb 2023 at 20:21, Aditya Sharma
>  wrote:
> 
>> 
>> Hello,
>> 
>> Yes it can be done simply by forcing “provision host security keys“ from
>> the Web UI.
>> 
>> Regards,
>> Aditya Sharma
>> 
>>> On 21-Feb-2023, at 00:01, Antoine Boucher  wrote:
>>> 
>>> Hello,
>>> 
>>> I have just upgraded from 4.16.2 to 4.17.2 all went well.
>>> 
>>> However, probably unrelated to the upgrade, I needed to do maintenance
>> on on of my Centos 7 kvm host. When I rebooted the host CloudStack agent
>> can not start, complaining about expired libvirt certificated.
>>> 
>>> I read that the certificate for libvirt of centos 7 is valid for one
>> year.  There is a fairly convoluted way to update them.  Is there a simpler
>> way to renew the cert?
>>> 
>>> I have not rebooted my other centos 7 kvm hosts, that are likely over
>> the one year mark.  Can these hosts libvirt certs be upgraded simply by
>> forcing “provision host security keys“ from the webui console in the
>> infrastructure/host section since I still have cloud-agent connection?
>>> 
>>> Regards,
>>> Antoine Boucher
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy, copying,
>> circulation or other use of this message and any attachments is strictly
>> prohibited. If you are not the intended recipient, please notify the sender
>> immediately by return e-mail, and delete this message and any attachments
>> from your system.
>> 
>> --
>> This message is intended only for the use of the individual or entity to
>> which it is addressed and may contain confidential and/or privileged
>> information. If you are not the intended recipient, please delete the
>> original message and any copy of it from your computer system. You are
>> hereby notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited unless proper authorization has been
>> obtained for such action. If you have received this communication in
>> error,
>> please notify the sender immediately. Although IndiQus attempts to sweep
>> e-mail and attachments for viruses, it does not guarantee that both are
>> virus-free and accepts no liability for any damage sustained as a result
>> of
>> viruses.
>> 

Re: Expired Libvirt certificate on CentOS 7 KVM host.

[Antoine Boucher]

Ok, from the webui for connected KVM hosts, and manually for hosts that non 
longer are able to connect to the management server because of the expired 
certs?

Thank you,
Antoine

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Feb 20, 2023, at 14:21, Aditya Sharma  
wrote:


Hello,

Yes it can be done simply by forcing “provision host security keys“ from the 
Web UI.

Regards,
Aditya Sharma

> On 21-Feb-2023, at 00:01, Antoine Boucher  wrote:
> 
> Hello,
> 
> I have just upgraded from 4.16.2 to 4.17.2 all went well. 
> 
> However, probably unrelated to the upgrade, I needed to do maintenance on on 
> of my Centos 7 kvm host. When I rebooted the host CloudStack agent can not 
> start, complaining about expired libvirt certificated. 
> 
> I read that the certificate for libvirt of centos 7 is valid for one year.  
> There is a fairly convoluted way to update them.  Is there a simpler way to 
> renew the cert?
> 
> I have not rebooted my other centos 7 kvm hosts, that are likely over the one 
> year mark.  Can these hosts libvirt certs be upgraded simply by forcing 
> “provision host security keys“ from the webui console in the 
> infrastructure/host section since I still have cloud-agent connection?
> 
> Regards,
> Antoine Boucher
> 
> 
> 
> 
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.

-- 
This message is intended only for the use of the individual or entity to 
which it is addressed and may contain confidential and/or privileged 
information. If you are not the intended recipient, please delete the 
original message and any copy of it from your computer system. You are 
hereby notified that any dissemination, distribution or copying of this 
communication is strictly prohibited unless proper authorization has been 
obtained for such action. If you have received this communication in error, 
please notify the sender immediately. Although IndiQus attempts to sweep 
e-mail and attachments for viruses, it does not guarantee that both are 
virus-free and accepts no liability for any damage sustained as a result of 
viruses.

Expired Libvirt certificate on CentOS 7 KVM host.

[Antoine Boucher]

Hello,

I have just upgraded from 4.16.2 to 4.17.2 all went well. 

However, probably unrelated to the upgrade, I needed to do maintenance on on of 
my Centos 7 kvm host. When I rebooted the host CloudStack agent can not start, 
complaining about expired libvirt certificated. 

I read that the certificate for libvirt of centos 7 is valid for one year.  
There is a fairly convoluted way to update them.  Is there a simpler way to 
renew the cert?

I have not rebooted my other centos 7 kvm hosts, that are likely over the one 
year mark.  Can these hosts libvirt certs be upgraded simply by forcing 
“provision host security keys“ from the webui console in the 
infrastructure/host section since I still have cloud-agent connection?

Regards,
Antoine Boucher






Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

FIXED: Consoleproxy VM is no longer created

[Antoine Boucher]

We have one test cluster that we use to test all sort of failures.  This time 
all hosts on this specific test cluster were failed such that the SSVM and CPVM 
also failed.  This seem to prevent the management server to start any new CPVM 
on any other clusters, but the SSVM creation were fine.  A new CPVM was started 
on the production cluster as soon as I deleted the CPVM the failed test cluster.

Is this an expected blocking behaviour on the CPVM management logic?  

Regards,
Antoine


Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Dec 8, 2022, at 12:39 PM, vas...@gmx.de wrote:
> 
> Hmmm... well i guess turn ssl encryption off in global settings and 
> afterwards  restart SSVM and depending on your overall setup the mgmt server.
> Never have performed this before so i would suggest to dig a bit through the 
> documentation and wiki for this topic.
> 
> Regards
> 
> Am Do., 8. Dez. 2022 um 16:52 Uhr schrieb Antoine Boucher 
> mailto:antoi...@haltondc.com>>:
>> What is the simplest way to disable ssl? Global setting then WebUI?
>> 
>> 
>> 
>> Antoine Boucher
>> antoi...@haltondc.com <mailto:antoi...@haltondc.com>
>> [o] +1-226-505-9734
>> www.haltondc.com <http://www.haltondc.com/>
>> 
>> “Data security made simple”
>> 
>> 
>> 
>> 
>> 
>> Confidentiality Warning: This message and any attachments are intended only 
>> for the use of the intended recipient(s), are confidential, and may be 
>> privileged. If you are not the intended recipient, you are hereby notified 
>> that any review, retransmission, conversion to hard copy, copying, 
>> circulation or other use of this message and any attachments is strictly 
>> prohibited. If you are not the intended recipient, please notify the sender 
>> immediately by return e-mail, and delete this message and any attachments 
>> from your system.
>> 
>> 
>>> On Dec 8, 2022, at 4:28 AM, vas...@gmx.de <mailto:vas...@gmx.de> wrote:
>>> 
>>> Hi Antoine,
>>> 
>>> without any log files this is going to be quiet challangeing.
>>> The log in DEBUG level or 'worse' TRACE on the management server should at
>>> least give some information that
>>> a) a console proxy is missing
>>> b) the deployment should start
>>> c) that a deployment fails / timed out
>>> or what else is currently not working properly.
>>> 
>>> How ever: have you tried to disable SSL configuration for the whole setup=?
>>> Just to get a idea where to search onwards from this point on?
>>> 
>>> Regards
>>> 
>>> Am Do., 8. Dez. 2022 um 01:52 Uhr schrieb Antoine Boucher <
>>> antoi...@haltondc.com <mailto:antoi...@haltondc.com>>:
>>> 
>>>> I now see that the certificates should be in Keystore table and all seems
>>>> ok.
>>>> 
>>>> Does anyone have any suggestion?  It is becoming critical as we can not
>>>> get to any vm consoles.
>>>> 
>>>> Regards,
>>>> Antoine
>>>> 
>>>> 
>>>> 
>>>> Confidentiality Warning: This message and any attachments are intended
>>>> only for the use of the intended recipient(s), are confidential, and may be
>>>> privileged. If you are not the intended recipient, you are hereby notified
>>>> that any review, retransmission, conversion to hard copy, copying,
>>>> circulation or other use of this message and any attachments is strictly
>>>> prohibited. If you are not the intended recipient, please notify the sender
>>>> immediately by return e-mail, and delete this message and any attachments
>>>> from your system.
>>>> 
>>>> 
>>>>> On Dec 7, 2022, at 7:34 AM, Antoine Boucher >>>> <mailto:antoi...@haltondc.com>>
>>>> wrote:
>>>>> 
>>>>> We still have not found any resolution and still no console proxy.
>>>>> 
>>>>> We had to change the ssl certificate at the same time.  However, I was
>>>> looking at the database and I see that the sslcerts 

Re: Consoleproxy VM is no longer created

[Antoine Boucher]

I now see that the certificates should be in Keystore table and all seems ok.

Does anyone have any suggestion?  It is becoming critical as we can not get to 
any vm consoles.

Regards,
Antoine



Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Dec 7, 2022, at 7:34 AM, Antoine Boucher  wrote:
> 
> We still have not found any resolution and still no console proxy.
> 
> We had to change the ssl certificate at the same time.  However, I was 
> looking at the database and I see that the sslcerts table is empty so are any 
> other tables with name containing the word “cert” are also empty.  Is this 
> normal?
> 
> Regards,
> Antoine
> 
>> On Dec 6, 2022, at 7:39 AM, Nux  wrote:
>> 
>> There could be a few reasons.
>> Make sure your management and hypervisor logs are set to DEBUG when you are 
>> investigating.
>> If nothing shows up on the management side, keep an eye on the hypervisor, 
>> see if a VM creation is even attempted, check libvirt (or whatever) logs etc.
>> 
>> ---
>> Nux
>> www.nux.ro
>> 
>> On 2022-12-06 04:37, Antoine Boucher wrote:
>>> Hello,
>>> We had a failure with a secondary storage that created several issues.
>>> Nevertheless, after fixing the issues, we discovered that the system
>>> consoleproxy and secondarystorage vm were hung.
>>> We deleted both system VMs, the secondarystorage vm came back without
>>> issues, but the consoleproxy vm is not being created.
>>> We will investigate further, but we have found nothing out of the
>>> ordinary in the management log file so far.
>>> Has anyone had a similar issue before?
>>> Regards,
>>> Antoine
>>> Confidentiality Warning: This message and any attachments are intended
>>> only for the use of the intended recipient(s), are confidential, and
>>> may be privileged. If you are not the intended recipient, you are
>>> hereby notified that any review, retransmission, conversion to hard
>>> copy, copying, circulation or other use of this message and any
>>> attachments is strictly prohibited. If you are not the intended
>>> recipient, please notify the sender immediately by return e-mail, and
>>> delete this message and any attachments from your system.
> 

Re: Consoleproxy VM is no longer created

[Antoine Boucher]

We still have not found any resolution and still no console proxy.

We had to change the ssl certificate at the same time.  However, I was looking 
at the database and I see that the sslcerts table is empty so are any other 
tables with name containing the word “cert” are also empty.  Is this normal?

Regards,
Antoine

> On Dec 6, 2022, at 7:39 AM, Nux  wrote:
> 
> There could be a few reasons.
> Make sure your management and hypervisor logs are set to DEBUG when you are 
> investigating.
> If nothing shows up on the management side, keep an eye on the hypervisor, 
> see if a VM creation is even attempted, check libvirt (or whatever) logs etc.
> 
> ---
> Nux
> www.nux.ro
> 
> On 2022-12-06 04:37, Antoine Boucher wrote:
>> Hello,
>> We had a failure with a secondary storage that created several issues.
>> Nevertheless, after fixing the issues, we discovered that the system
>> consoleproxy and secondarystorage vm were hung.
>> We deleted both system VMs, the secondarystorage vm came back without
>> issues, but the consoleproxy vm is not being created.
>> We will investigate further, but we have found nothing out of the
>> ordinary in the management log file so far.
>> Has anyone had a similar issue before?
>> Regards,
>> Antoine
>> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and
>> may be privileged. If you are not the intended recipient, you are
>> hereby notified that any review, retransmission, conversion to hard
>> copy, copying, circulation or other use of this message and any
>> attachments is strictly prohibited. If you are not the intended
>> recipient, please notify the sender immediately by return e-mail, and
>> delete this message and any attachments from your system.

Consoleproxy VM is no longer created

[Antoine Boucher]

Hello,

We had a failure with a secondary storage that created several issues.

Nevertheless, after fixing the issues, we discovered that the system 
consoleproxy and secondarystorage vm were hung.

We deleted both system VMs, the secondarystorage vm came back without issues, 
but the consoleproxy vm is not being created. 

We will investigate further, but we have found nothing out of the ordinary in 
the management log file so far.

Has anyone had a similar issue before?

Regards,
Antoine
  



Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

Re: Web UI suddenly refuse to start

[Antoine Boucher]

The Web UI is finally up, 2 hours after I restarted the Management Server.  Is 
this an explainable behaviour?  I will look at the logs around the time.

Regards,
Antoine



> On Oct 18, 2022, at 4:24 PM, Wei ZHOU  wrote:
> 
> It would be good to upload the management server log since last start.
> 
> -Wei
> 
> On Tuesday, 18 October 2022, Antoine Boucher  wrote:
> 
>> Hello,
>> 
>> I have been working on saml configurarion for the past few days, while
>> refining our setup we have restarted our management server a dozen of times
>> but this time the UI is not up but logs are showing that the management
>> server is up.
>> 
>> Netstats shows no applications listening on port 8080,8250 or 9090, no UI
>> or cloud monkey access
>> 
>> Logs indicates that the Mangement server is up and running, here are the
>> only error that I see ( I skipped the INFO and WARN messages)
>> 
>> 
>> 
>> 2022-10-18 12:31:09,169 ERROR [c.c.c.ClusterManagerImpl] (main:null)
>> (logid:) Unable to ping management server at 10.101.2.40:9090 due to
>> ConnectException
>> 
>> 2022-10-18 12:31:09,317 WARN  [c.c.s.d.DownloadMonitorImpl] (main:null)
>> (logid:) Only realhostip.com ssl cert is supported, ignoring self-signed
>> and other certs
>> 
>> 2022-10-18 12:31:10,753 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:)
>> Unable to find properties file: commands.properties
>> 
>> Then a few of these related to snapshot schedule
>> 2022-10-18 12:31:17,106 WARN  [c.c.u.d.T.Transaction] (main:null) (logid:)
>> txn: Commit called when it is not a transaction: -SnapshotSchedulerImpl.
>> scheduleNextSnapshotJob:396-SnapshotSchedulerImpl.start:466-
>> CloudStackExtendedLifeCycle$1.with:74-CloudStackExtendedLifeCycle.
>> with:153-CloudStackExtendedLifeCycle.startBeans:71-
>> CloudStackExtendedLifeCycleStart.run:46-DefaultModuleDefinitionSet$1.
>> with:104-DefaultModuleDefinitionSet.withModule:244-
>> DefaultModuleDefinitionSet.withModule:249-DefaultModuleDefinitionSet.
>> withModule:249-DefaultModuleDefinitionSet.withModule:232-
>> DefaultModuleDefinitionSet.startContexts:96
>> 
>> 
>> 2022-10-18 12:31:20,039 WARN  [c.c.a.AlertManagerImpl]
>> (Cluster-Notification-1:ctx-b9523ef9) (logid:54f42903) alertType=[14]
>> dataCenterId=[0] podId=[0] clusterId=[null] message=[Management server node
>> 10.101.2.40 is up].
>> 
>> 2022-10-18 12:31:20,061 WARN  [c.c.c.ClusterManagerImpl]
>> (Cluster-Notification-1:ctx-b9523ef9) (logid:54f42903) Notifying
>> management server join event took 23 ms
>> 
>> 2022-10-18 12:31:20,354 WARN  [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-8:null) (logid:) Throwing away a request because it
>> came through as the first command on a connect: Seq 0-18565:  { Cmd ,
>> MgmtId: -1, via: 0, Ver: v1, Flags: 11, [{"com.cloud.agent.api.
>> PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_
>> hostVmStateReport":{},"_gatewayAccessible":"true","_
>> vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","
>> bypassHostMaintenance":"false"}}] }
>> 
>> 
>> 2022-10-18 12:31:20,371 WARN  [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-2:null) (logid:) Throwing away a request because it
>> came through as the first command on a connect: Seq 0-18570:  { Cmd ,
>> MgmtId: -1, via: 0, Ver: v1, Flags: 11, [{"com.cloud.agent.api.
>> PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_
>> hostVmStateReport":{},"_gatewayAccessible":"true","_
>> vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","
>> bypassHostMaintenance":"false"}}] }
>> 
>> 2022-10-18 12:31:20,381 WARN  [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-11:null) (logid:) Throwing away a request because it
>> came through as the first command on a connect: Seq 0-18579:  { Cmd ,
>> MgmtId: -1, via: 0, Ver: v1, Flags: 11, [{"com.cloud.agent.api.
>> PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_
>> hostVmStateReport":{},"_gatewayAccessible":"true","_
>> vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","
>> bypassHostMaintenance":"false"}}] }
>> 
>> 
>> 
>> 
>> Our management server in on Centos 7.9, the mysql is up a running,
>> sytemctl status is showing no issues including cloudstack-management up and
>> running without issues.
>> 
>> No firewalls or selinux, we rebooted and restarted a number of times but
>> still not access, stops is showing minimum activities.
>> 
>> Has anyone had this situation happened?  I will keep digging...
>> 
>> Regards,
>> Antoine
>> 
>> 
>> 

Re: Web UI suddenly refuse to start

[Antoine Boucher]

Port 9090 and 8250 are up but not 8443 or 8080



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Oct 18, 2022, at 1:36 PM, Antoine Boucher  wrote:
> 
> Hello,
> 
> I have been working on saml configurarion for the past few days, while 
> refining our setup we have restarted our management server a dozen of times 
> but this time the UI is not up but logs are showing that the management 
> server is up.
> 
> Netstats shows no applications listening on port 8080,8250 or 9090, no UI or 
> cloud monkey access
> 
> Logs indicates that the Mangement server is up and running, here are the only 
> error that I see ( I skipped the INFO and WARN messages)
> 
> 
> 
> 2022-10-18 12:31:09,169 ERROR [c.c.c.ClusterManagerImpl] (main:null) (logid:) 
> Unable to ping management server at 10.101.2.40:9090 due to ConnectException
> 
> 2022-10-18 12:31:09,317 WARN  [c.c.s.d.DownloadMonitorImpl] (main:null) 
> (logid:) Only realhostip.com <http://realhostip.com/> ssl cert is supported, 
> ignoring self-signed and other certs
> 
> 2022-10-18 12:31:10,753 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:) 
> Unable to find properties file: commands.properties
> 
> Then a few of these related to snapshot schedule
> 2022-10-18 12:31:17,106 WARN  [c.c.u.d.T.Transaction] (main:null) (logid:) 
> txn: Commit called when it is not a transaction: 
> -SnapshotSchedulerImpl.scheduleNextSnapshotJob:396-SnapshotSchedulerImpl.start:466-CloudStackExtendedLifeCycle$1.with:74-CloudStackExtendedLifeCycle.with:153-CloudStackExtendedLifeCycle.startBeans:71-CloudStackExtendedLifeCycleStart.run:46-DefaultModuleDefinitionSet$1.with:104-DefaultModuleDefinitionSet.withModule:244-DefaultModuleDefinitionSet.withModule:249-DefaultModuleDefinitionSet.withModule:249-DefaultModuleDefinitionSet.withModule:232-DefaultModuleDefinitionSet.startContexts:96
> 
> 
> 2022-10-18 12:31:20,039 WARN  [c.c.a.AlertManagerImpl] 
> (Cluster-Notification-1:ctx-b9523ef9) (logid:54f42903) alertType=[14] 
> dataCenterId=[0] podId=[0] clusterId=[null] message=[Management server node 
> 10.101.2.40 is up].
> 
> 2022-10-18 12:31:20,061 WARN  [c.c.c.ClusterManagerImpl] 
> (Cluster-Notification-1:ctx-b9523ef9) (logid:54f42903) Notifying management 
> server join event took 23 ms
> 
> 2022-10-18 12:31:20,354 WARN  [c.c.a.m.AgentManagerImpl] 
> (AgentManager-Handler-8:null) (logid:) Throwing away a request because it 
> came through as the first command on a connect: Seq 0-18565:  { Cmd , MgmtId: 
> -1, via: 0, Ver: v1, Flags: 11, 
> [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","bypassHostMaintenance":"false"}}]
>  }
> 
> 
> 2022-10-18 12:31:20,371 WARN  [c.c.a.m.AgentManagerImpl] 
> (AgentManager-Handler-2:null) (logid:) Throwing away a request because it 
> came through as the first command on a connect: Seq 0-18570:  { Cmd , MgmtId: 
> -1, via: 0, Ver: v1, Flags: 11, 
> [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","bypassHostMaintenance":"false"}}]
>  }
> 
> 2022-10-18 12:31:20,381 WARN  [c.c.a.m.AgentManagerImpl] 
> (AgentManager-Handler-11:null) (logid:) Throwing away a request because it 
> came through as the first command on a connect: Seq 0-18579:  { Cmd , MgmtId: 
> -1, via: 0, Ver: v1, Flags: 11, 
> [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","bypassHostMaintenance":"false"}}]
>  }
> 
> 
> 
> 
> Our management server in on Centos 7.9, the mysql is up a running, sytemctl 
> status is showing no issues including cloudstack-management up and running 
> without issues.
> 
> No firewalls or selinux, we rebooted and restarted a number of times but 
> still not access, stops is showing minimum activities.
> 
> Has anyone had this situation happened?  I will keep digging...
> 
> Regards,
> Antoine
> 
> 

Web UI suddenly refuse to start

[Antoine Boucher]

Hello,

I have been working on saml configurarion for the past few days, while refining 
our setup we have restarted our management server a dozen of times but this 
time the UI is not up but logs are showing that the management server is up.

Netstats shows no applications listening on port 8080,8250 or 9090, no UI or 
cloud monkey access

Logs indicates that the Mangement server is up and running, here are the only 
error that I see ( I skipped the INFO and WARN messages)



2022-10-18 12:31:09,169 ERROR [c.c.c.ClusterManagerImpl] (main:null) (logid:) 
Unable to ping management server at 10.101.2.40:9090 due to ConnectException

2022-10-18 12:31:09,317 WARN  [c.c.s.d.DownloadMonitorImpl] (main:null) 
(logid:) Only realhostip.com ssl cert is supported, ignoring self-signed and 
other certs

2022-10-18 12:31:10,753 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:) 
Unable to find properties file: commands.properties

Then a few of these related to snapshot schedule
2022-10-18 12:31:17,106 WARN  [c.c.u.d.T.Transaction] (main:null) (logid:) txn: 
Commit called when it is not a transaction: 
-SnapshotSchedulerImpl.scheduleNextSnapshotJob:396-SnapshotSchedulerImpl.start:466-CloudStackExtendedLifeCycle$1.with:74-CloudStackExtendedLifeCycle.with:153-CloudStackExtendedLifeCycle.startBeans:71-CloudStackExtendedLifeCycleStart.run:46-DefaultModuleDefinitionSet$1.with:104-DefaultModuleDefinitionSet.withModule:244-DefaultModuleDefinitionSet.withModule:249-DefaultModuleDefinitionSet.withModule:249-DefaultModuleDefinitionSet.withModule:232-DefaultModuleDefinitionSet.startContexts:96


2022-10-18 12:31:20,039 WARN  [c.c.a.AlertManagerImpl] 
(Cluster-Notification-1:ctx-b9523ef9) (logid:54f42903) alertType=[14] 
dataCenterId=[0] podId=[0] clusterId=[null] message=[Management server node 
10.101.2.40 is up].

2022-10-18 12:31:20,061 WARN  [c.c.c.ClusterManagerImpl] 
(Cluster-Notification-1:ctx-b9523ef9) (logid:54f42903) Notifying management 
server join event took 23 ms

2022-10-18 12:31:20,354 WARN  [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-8:null) (logid:) Throwing away a request because it came 
through as the first command on a connect: Seq 0-18565:  { Cmd , MgmtId: -1, 
via: 0, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","bypassHostMaintenance":"false"}}]
 }


2022-10-18 12:31:20,371 WARN  [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-2:null) (logid:) Throwing away a request because it came 
through as the first command on a connect: Seq 0-18570:  { Cmd , MgmtId: -1, 
via: 0, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","bypassHostMaintenance":"false"}}]
 }

2022-10-18 12:31:20,381 WARN  [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-11:null) (logid:) Throwing away a request because it came 
through as the first command on a connect: Seq 0-18579:  { Cmd , MgmtId: -1, 
via: 0, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"0","wait":"0","bypassHostMaintenance":"false"}}]
 }




Our management server in on Centos 7.9, the mysql is up a running, sytemctl 
status is showing no issues including cloudstack-management up and running 
without issues.

No firewalls or selinux, we rebooted and restarted a number of times but still 
not access, stops is showing minimum activities.

Has anyone had this situation happened?  I will keep digging...

Regards,
Antoine

Re: Attribute name mapping issue - Keycloak as idP

[Antoine Boucher]

Hi Vladimir,

Have you seen this : 
https://www.mail-archive.com/users@cloudstack.apache.org/msg29759.html 
<https://www.mail-archive.com/users@cloudstack.apache.org/msg29759.html> ?


Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Oct 17, 2022, at 11:50 AM, Vladimir Dombrovski 
>  wrote:
> 
> Hello Antoine (and others),
> 
> We've tried the same integration, and we are stuck on exactly the same
> error message. Not sure how to proceed from there, we are able to
> provide some elements from our setup.
> 
> Regards,
> 
> Vladimir
> 
> On Mon, 17 Oct 2022 at 15:58, Antoine Boucher  wrote:
>> 
>> Hello,
>> 
>> We need to integrate MFA for the CloudStack admin accounts, after trying 
>> Google and ADFS we have landed on using Keycloak.
>> 
>> However, after a week, we can not seem to be able to resolve the below 
>> error, Which we assume to be a mapping issue from Keycloak to CloudStack.
>> 
>> 
>> 531
>> Failed to find admin configured username attribute in the SAML 
>> Response. Please ask your administrator to check SAML user attribute 
>> name.
>> 
>> 
>> We will reward any help that helps us complete the integration.
>> 
>> Regards,
>> Antoine
>> 
>> 
>> Antoine Boucher
>> antoi...@haltondc.com
>> 
>> 
> 
> -- 
> *CONFIDENTIALITY AND DISCLAIMER NOTICE: *
> This email is intended only for 
> the person to whom it is addressed and/or otherwise authorized personnel. 
> The information contained herein and attached is confidential. If you are 
> not the intended recipient, please be advised that viewing this message and 
> any attachments, as well as copying, forwarding, printing, and 
> disseminating any information related to this email is prohibited, and that 
> you should not take any action based on the content of this email and/or 
> its attachments. If you received this message in error, please contact the 
> sender and destroy all copies of this email and any attachment. Please note 
> that the views and opinions expressed herein are solely those of the author 
> and do not necessarily reflect those of the company. While antivirus 
> protection tools have been employed, you should check this email and 
> attachments for the presence of viruses. No warranties or assurances are 
> made in relation to the safety and content of this email and attachments. 
> The Company accepts no liability for any damage caused by any virus 
> transmitted by or contained in this email and attachments. No liability is 
> accepted for any consequences arising from this email.
> 
> 
> *AVIS DE 
> CONFIDENTIALITÉ ET DE NON RESPONSABILITE* : 
> Ce courriel, ainsi que toute 
> pièce jointe, est confidentiel et peut être protégé par le secret 
> professionnel. Si vous n’en êtes pas le destinataire visé, veuillez en 
> aviser l’expéditeur immédiatement et le supprimer. Vous ne devez pas le 
> copier, ni l’utiliser à quelque fin que ce soit, ni divulguer son contenu à 
> qui que ce soit. BSO se réserve le droit de contrôler toute transmission 
> qui passe par son réseau. Veuillez noter que les opinions exprimées dans 
> cet e-mail sont uniquement celles de l'auteur et ne reflètent pas 
> nécessairement celles de la société. Bien que des outils de protection 
> antivirus aient été utilisés, vous devez vérifier cet e-mail et les pièces 
> jointes pour toute présence de virus. Aucune garantie ou assurance n'est 
> donnée concernant la sécurité et le contenu de cet e-mail et de ses pièces 
> jointes. La Société décline toute responsabilité pour tout dommage causé 
> par tout virus transmis par ou contenu dans cet e-mail et ses pièces 
> jointes. Aucune responsabilité n'est acceptée pour les conséquences 
> découlant de cet e-mail.

Attribute name mapping issue - Keycloak as idP

[Antoine Boucher]

Hello,

We need to integrate MFA for the CloudStack admin accounts, after trying Google 
and ADFS we have landed on using Keycloak.

However, after a week, we can not seem to be able to resolve the below error, 
Which we assume to be a mapping issue from Keycloak to CloudStack.


531
Failed to find admin configured username attribute in the SAML 
Response. Please ask your administrator to check SAML user attribute 
name.


We will reward any help that helps us complete the integration.

Regards,
Antoine


Antoine Boucher
antoi...@haltondc.com

Re: Proper procedure to delete orphan volumes from a dead Primary Storage

[Antoine Boucher]

Thank you Sina, I will look at the volume table. 

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Oct 14, 2022, at 05:31, Sina Kashipazha 
 wrote:

Hey Antoine,

We had the same issue. We found volumes that no longer exists on primary 
storage but they were visible in the UI. We sat the removed filed to now() in 
volumes table and the issue resolved.


Kind regards,
Sina



--- Original Message ---
On Thursday, September 29th, 2022 at 17:27, Antoine Boucher 
 wrote:


> 

> 

> I have a few VM less volumes in Destroy state in no longer existing (dead) 
> Primary storage.
> 

> Since I’m unable to remove volumes from the GUI. What would be the proper 
> procedure to remove the entries from the db?
> 

> Would I just delete the appropriate row of the “volumes” and “volume_view” 
> tables ? Or alternatively change the value of one of the fileds?
> 

> Regards,
> Antoine

Working Keycloak SAML configuration

[Antoine Boucher]

Hello,

Would anyone have a working Keycloak SAML configuration that they could share?

Thank you,
Antoine

Re: SSO via Saml to ADFS 2022

[Antoine Boucher]

Hi Rohit,

Thank you for your response.  It sounds like the path forward is Keycloak.  
Would you have a link to a working Keycloak configuration? 

Regards,
Antoine



> On Oct 13, 2022, at 4:21 AM, Rohit Yadav  wrote:
> 
> Hi Antoine,
> 
> I know a lot of folks have used in the past (maybe not 2022) and also used 
> keycloak. Can you check if ADFS allows the username attribute to be 
> configured?
> 
> See
> https://rajanikaruturi.blogspot.com/2018/10/configuring-adfs-sso-with-apache.html
> https://www.youtube.com/watch?v=kTvNbf-KFH0
> 
> 
> 
> Regards.
> 
> ____
> From: Antoine Boucher 
> Sent: Wednesday, October 12, 2022 08:55
> To: users 
> Subject: SSO via Saml to ADFS 2022
> 
> Hello,
> 
> Has anyone successfully setup SSO with ADFS on Windows 2022?
> 
> Somehow I’m stuck at the response back to ACS with this error:
> 
> 
> 531
> Failed to find admin configured username attribute in the SAML 
> Response. Please ask your administrator to check SAML user attribute 
> name.
> 
> 
> Regards,
> Antoine
> 
> 
> 
> 
> 

SSO via Saml to ADFS 2022

[Antoine Boucher]

Hello,

Has anyone successfully setup SSO with ADFS on Windows 2022?

Somehow I’m stuck at the response back to ACS with this error:


531
Failed to find admin configured username attribute in the SAML 
Response. Please ask your administrator to check SAML user attribute 
name.


Regards,
Antoine

Proper procedure to delete orphan volumes from a dead Primary Storage

[Antoine Boucher]

I have a few VM less volumes in Destroy state in no longer existing (dead) 
Primary storage.

Since I’m unable to remove volumes from the GUI. What would be the proper 
procedure to remove the entries from the db?

Would I just delete the appropriate row of the “volumes” and “volume_view” 
tables ? Or alternatively change the value of one of the fileds?

Regards,
Antoine

Re: Reconnecting KVM host after agent.properties became empty

[Antoine Boucher]

Thanks again Nux,

Indeed, time to backup the /etc.

I will try to recreate it.  I’m assuming that I could recreate forcing the 
recreation of the keys using the GUI?

Regards,
Antoine

> On Sep 28, 2022, at 11:39 AM, Nux  wrote:
> 
> Hi,
> 
> You could copy the file from another hypervisor and edit it.
> You can take the local storage uuid from "virsh pool-list" and the "guid" 
> from the Cloudstack DB.
> Not sure how you could recover the keystore.passphrase though, but there has 
> to be a way.
> 
> Time to start backing up /etc :)
> 
> ---
> Nux
> www.nux.ro
> 
> On 2022-09-28 14:58, Antoine Boucher wrote:
>> Hello,
>> I noticed that the agent.properties configuration file in one of my
>> KVM host is became empty and I do not have a saved copy.  This
>> specific host is the only host in a Zone.
>> What would be the best practice to reconnect the host to the Zone with
>> existing VMs on local and shared primary storage, templates and all.
>> Thank you,
>> Antoine

Fwd: Lost of a KVM host with VMs on Local storage

[Antoine Boucher]

Adding mailing list.

> 
> 
> From: Antoine Boucher 
> Subject: Re: Lost of a KVM host with VMs on Local storage
> Date: September 28, 2022 at 1:30:30 PM EDT
> To: Nux 
> 
> Hi Nux,
> 
> Thank you for your response.  The template path would be acceptable.  As for 
> messing with the db it should not be a problem once I understand the 
> structure, It would be great to have a UI option to force the clean-up 
> volumes in such conditions.  I have at least 3 in destroy state that I need 
> to manually clean-up.
> 
> Thanks for your help.
> 
> -Antoine
> 
> 
>> On Sep 28, 2022, at 11:32 AM, Nux mailto:n...@li.nux.ro>> 
>> wrote:
>> 
>> Hi,
>> 
>> Ideally you would have a full baremetal backup of the hypervisor if using 
>> local storage.
>> If that is not a possibility, then you could convert the volume snapshot of 
>> the VM into a template and deploy it on the desired hypervisor.
>> 
>> You will not be able to "revert to snapshot" if your hypervisor is 
>> completely gone, as you have noticed, in the DB there are references to both 
>> hypervisor and storage pool that would need messing with.
>> 
>> ---
>> Nux
>> www.nux.ro <http://www.nux.ro/>
>> 
>> On 2022-09-28 15:43, Antoine Boucher wrote:
>>> Hello,
>>> We have a few high iops VMs running on local storage on some of our
>>> KVM hosts.  We backup (snapshots) these VMs on a regular basis on
>>> secondary storage.
>>> In the event of a compete KVM host failure.
>>> What would be the best practice to clean-up and restart the VMs from
>>> backups to a new host.
>>> Would it just be a simple restore from the snapshot to the new host
>>> and the MS will fix the corresponding references to the dead VM and
>>> volume object?  I remember having to delete records in the MS db to
>>> remove the volume reference when a host was removed before removing
>>> its local volume from the GUI.
>>> Regards,
>>> Antoine
> 

Lost of a KVM host with VMs on Local storage

[Antoine Boucher]

Hello,

We have a few high iops VMs running on local storage on some of our KVM hosts.  
We backup (snapshots) these VMs on a regular basis on secondary storage.

In the event of a compete KVM host failure.  

What would be the best practice to clean-up and restart the VMs from backups to 
a new host.

Would it just be a simple restore from the snapshot to the new host and the MS 
will fix the corresponding references to the dead VM and volume object?  I 
remember having to delete records in the MS db to remove the volume reference 
when a host was removed before removing its local volume from the GUI.

Regards,
Antoine

Reconnecting KVM host after agent.properties became empty

[Antoine Boucher]

Hello,

I noticed that the agent.properties configuration file in one of my KVM host is 
became empty and I do not have a saved copy.  This specific host is the only 
host in a Zone.  
 
What would be the best practice to reconnect the host to the Zone with existing 
VMs on local and shared primary storage, templates and all.

Thank you,
Antoine

Re: Linstor Primary Storage on 4.16.1

[Antoine Boucher]

Thank you Suresh,

A zone-wide primary nfs is preexisting and seemed to be automatically added to 
this new cluster prior to configuring Linstor primary storage via the gui. We 
will file an issue.   It is a blocking issue for our new clusters. 

Thank you again. 
Antoine


On Aug 8, 2022, at 14:01, Suresh Kumar Anaparti 
 wrote:


Hi Antoine,

Have you tried to add cluster-wide or zone-wide Primary Storage? Can you check 
with both.

If you notice any issue with protocol / provider selection in the UI , please 
create an issue here: https://github.com/apache/cloudstack/issues

Regards,
Suresh

On Sun, Aug 7, 2022 at 11:42 PM Antoine Boucher  wrote:
> Hi Suresh,
> 
> I double checked and all 4 hosts and cluster is set to KVM.
> 
> This cluster also has 2 zone wide primary nfs storage that seems to have been 
> added automatically during the hosts creation.
> 
> Regards,
> Antoine
> 
> 
> 
> Antoine Boucher
> antoi...@haltondc.com
> [o] +1-226-505-9734
> www.haltondc.com
> 
> “Data security made simple and affordable”
> 
> 
> 
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.
> 
> 
>> On Aug 7, 2022, at 1:43 PM, Suresh Kumar Anaparti 
>>  wrote:
>> 
>> Hi Antoine,
>> 
>> It seems the Linstor protocol is listed for KVM hypervisor only. Can
>> you check it with any cluster having KVM hypervisor, or select KVM for
>> the zone. The fix here:
>> https://github.com/apache/cloudstack/pull/5672, was included in 4.16.1
>> related to Linstor protocol selection in UI.
>> 
>> Regards,
>> Suresh
>> 
>> On Sun, Aug 7, 2022 at 8:57 PM Antoine Boucher  wrote:
>>> 
>>> Hello,
>>> 
>>> Has anyone setup Linstor as primary storage?
>>> 
>>> From the documentation:
>>> 
>>> "After you are finished with the LINSTOR cluster setup, you can add a 
>>> Cloudstack primary storage as any other primary storage see Add Primary 
>>> Storage 
>>> <http://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#add-primary-storage>.
>>>  For protocol choose Linstor and as server specify the controller REST-API 
>>> URL e.g.: http://127.0.0.1:3370 and use the resource group name you added 
>>> in the LINSTOR cluster.”
>>> 
>>> I can not seem to be able to select “Linstor” as a Protocol selection.  The 
>>> only Linstor selection I can make is by selecting custom as Protocol and 
>>> Linstor as Provider but then I can not specify the server address.
>>> 
>>> Do I have to install or enable something for the plugin to work?
>>> 
>>> Regards,
>>> Antoine
>>> 
>>> 
>>> 
> 

Re: Linstor Primary Storage on 4.16.1

[Antoine Boucher]

Hi Suresh,

I double checked and all 4 hosts and cluster is set to KVM.

This cluster also has 2 zone wide primary nfs storage that seems to have been 
added automatically during the hosts creation.

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Aug 7, 2022, at 1:43 PM, Suresh Kumar Anaparti 
>  wrote:
> 
> Hi Antoine,
> 
> It seems the Linstor protocol is listed for KVM hypervisor only. Can
> you check it with any cluster having KVM hypervisor, or select KVM for
> the zone. The fix here:
> https://github.com/apache/cloudstack/pull/5672, was included in 4.16.1
> related to Linstor protocol selection in UI.
> 
> Regards,
> Suresh
> 
> On Sun, Aug 7, 2022 at 8:57 PM Antoine Boucher  wrote:
>> 
>> Hello,
>> 
>> Has anyone setup Linstor as primary storage?
>> 
>> From the documentation:
>> 
>> "After you are finished with the LINSTOR cluster setup, you can add a 
>> Cloudstack primary storage as any other primary storage see Add Primary 
>> Storage 
>> <http://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#add-primary-storage>.
>>  For protocol choose Linstor and as server specify the controller REST-API 
>> URL e.g.: http://127.0.0.1:3370 and use the resource group name you added in 
>> the LINSTOR cluster.”
>> 
>> I can not seem to be able to select “Linstor” as a Protocol selection.  The 
>> only Linstor selection I can make is by selecting custom as Protocol and 
>> Linstor as Provider but then I can not specify the server address.
>> 
>> Do I have to install or enable something for the plugin to work?
>> 
>> Regards,
>> Antoine
>> 
>> 
>> 

Linstor Primary Storage on 4.16.1

[Antoine Boucher]

Hello,

Has anyone setup Linstor as primary storage?

From the documentation:

"After you are finished with the LINSTOR cluster setup, you can add a 
Cloudstack primary storage as any other primary storage see Add Primary Storage 
.
 For protocol choose Linstor and as server specify the controller REST-API URL 
e.g.: http://127.0.0.1:3370 and use the resource group name you added in the 
LINSTOR cluster.”

I can not seem to be able to select “Linstor” as a Protocol selection.  The 
only Linstor selection I can make is by selecting custom as Protocol and 
Linstor as Provider but then I can not specify the server address.

Do I have to install or enable something for the plugin to work?

Regards,
Antoine

Re: Using S3 Storage for Secondary storage

[Antoine Boucher]

Hi Vladimir & Levin,

Thank you for your messages.  S3 compatible Secondary Storage on ACS does not 
seem realistic for the time being.

Unfortunate, ACS remains one of the only system that we have not been able to 
migrate to our S3 storage, minio has simplified our operations in many ways. 

Regards,
Antoine 

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Jun 28, 2022, at 3:05 AM, Vladimir Dombrovski  
> wrote:
> 
> Hello Antoine,
> 
> We've recently performed a thorough test of using S3 as a secondary
> storage on our QA platform using the 4.17.0 release. We've tried with
> 2 S3 Providers (Ceph, OpenIO), without any success in both cases. We
> will post one (or more) issues in the future concerning this subject,
> in the meantime here's what ve found:
> 
> - The current S3 implementation doesn't support any advanced
> parameters (such as region), which prevents it from working properly
> with S3 providers that require the V4 signature. The culprit being a
> deprecated use of the AWS S3 SDK for Java.
> - The following API call:
> https://cloudstack.apache.org/api/apidocs-4.17/apis/updateCloudToUseObjectStore.html,
> does work, however as there is no validation on any of the provided
> parameters, you are likely to get an NPE if you happen to miss one of
> them. Also, this call is not reversible, even in the case of an error.
> What it does under the hood is that it converts your current secondary
> storage to a NFS staging store, and adds an Image store of type S3,
> which will then initiate the download of all images present in your
> secondary (making them unavailable until downloaded).
> - Finally, the current implementation uses an asynchronous upload with
> a "ProgressListener" that discards any errors (aka exceptions) thrown
> to it. In practice you will see "Download error" on your templates,
> without any errors in the logs (empty error message). So basically it
> will either work fine if your S3 implementation works with it, or
> fails without any possibility for you to know why.
> 
> Please note, this is not a critique of the feature (which I'm sure was
> designed and tested against AWS S3 specifically, not S3 compatible
> APIs), this is simply what we've found about the state of the current
> implementation.
> 
> Regards,
> 
> Vladimir
> 
> 
> On Mon, 27 Jun 2022 at 16:56, Levin Ng  wrote:
>> 
>> Hi Antoine,
>> 
>> I’m looking for same question too. I’ve tested few s3fs implementation and 
>> only found rclone mount and juicejfs is working properly. However it require 
>> a huge set of buffering area to store intermediate images.
>> 
>> Regards,
>> Levin
>> 
>> From: Antoine Boucher 
>> Date: Monday, 27 June 2022 at 21:02
>> To: users 
>> Subject: Using S3 Storage for Secondary storage
>> Hello,
>> 
>> We are consolidating our backup storage to S3 using MinIO. It appears that 
>> migration from NFS-based secondary storage to S3 is none trivial task since 
>> both can not coexist for the transition period. Has anyone done the 
>> transition? We can’t lose the existing data from the current Secondary NFS 
>> storage data.
>> 
>> Alternatively, would an s3fs mounted on an NFS share work adequately for 
>> CloudStack‘s requirement?
>> 
>> Regards,
>> 
>> Antoine
> 
> -- 
> *CONFIDENTIALITY AND DISCLAIMER NOTICE: *
> This email is intended only for 
> the person to whom it is addressed and/or otherwise authorized personnel. 
> The information contained herein and attached is confidential. If you are 
> not the intended recipient, please be advised that viewing this message and 
> any attachments, as well as copying, forwarding, printing, and 
> disseminating any information related to this email is prohibited, and that 
> you should not take any action based on the content of this email and/or 
> its attachments. If you received this message in error, please contact the 
> sender and destroy all copies of this email and any attachment. Please note 
> that the views and opinions expressed herein are solely those of the author 
> and 

Using S3 Storage for Secondary storage

[Antoine Boucher]

Hello,

We are consolidating our backup storage to S3 using MinIO. It appears that 
migration from NFS-based secondary storage to S3 is none trivial task since 
both can not coexist for the transition period. Has anyone done the transition? 
We can’t lose the existing data from the current Secondary NFS storage data. 

Alternatively, would an s3fs mounted on an NFS share work adequately for 
CloudStack‘s requirement? 

Regards,

Antoine

Re: Cludstack vm backup

[Antoine Boucher]

Projects like this one may be interesting in the near future: 
https://www.youtube.com/watch?v=Jg40h1YjALk 



-Antoine

> On May 27, 2022, at 10:15 AM, Nux  wrote:
> 
> 
> 
> Yes indeed there are 2 types of snapshots, the VM (which also can snapshot 
> the RAM, thus preserving a full state) and the storage volume snapshot.
> 
> Both can be used for "backups", but they lend themselves to different 
> scenarios.
> 
> The VM snapshot is handy in those situations where you want a quick fallback 
> in case your "yum" or "windows" updates or other sensitive operations break 
> your VM or something. Of course, there are other scenarios where this could 
> be handy.
> 
> VM snapshots use the underlying file backing (say qcow2 file) for storing 
> what is required.
> 
> The volume snapshots are closer to the idea of proper backups because what 
> you get here is a full copy of the VM's disk, it'd be in a crash consistent 
> state by default, but could be good enough for most purposes. Cloudstack will 
> transfer this copy over on to the secondary storage, which is great because 
> it will be available even if the original hypervisor goes down etc. Also, 
> from this point on you can implement (outside of Cloudstack) further sync or 
> copy operations on these files, thus helping with disaster recorvery plans 
> and so on.
> 
> On KVM you need to make sure "kvm snapshot" is enabled in the Global Settings 
> btw.
> 
> ---
> Nux
> www.nux.ro  [1]
> 
> On 2022-05-27 14:55, vas...@gmx.de  wrote:
> 
>> Maybe it would be usefull to provide some more information regarding the 
>> different "snapshots" which are availeable.
>> As far as I am aware we have
>> - Volume Snapshots - which can be used with different VMs and can be planned 
>> ahead.
>> https://docs.cloudstack.apache.org/en/4.16.1.0/adminguide/storage.html#working-with-volume-snapshots
>> - VM Snapshots - which I guess is more "synonym" when talking about 
>> snapshots.
>> https://docs.cloudstack.apache.org/en/4.16.1.0/adminguide/virtual_machines.html#virtual-machine-snapshots
>> The documentation "only" mentions "Automatic Snapshot Creation" for volumes 
>> / storage. I am not quiet shure if this feature is also availeable for 
>> VM-Snapshots.
>> Am Fr., 27. Mai 2022 um 11:57 Uhr schrieb Nux :
>> You can have that, it's called "recurrent snapshots", in the UI you will 
>> find it under the "clock" icon when in the Volume page.
>> https://cloudstack.apache.org/api/apidocs-4.16/apis/createSnapshotPolicy.html
>> ---
>> Nux
>> www.nux.ro  [1]
>> On 2022-05-27 10:41, Mariusz Wojtarek wrote:
>> I look for solution with schedule option, as I know snaphost can be done 
>> only on demand
>> Mariusz Wojtarek
>> Administrator iT
>> P: 22 335 28 00
>> E: mariusz.wojta...@support-online.pl 
>> 
>> www.support-online.pl  [2]
>> Poleczki 23 | 02-822 Warszawa
>> [3][4]
>> Od: Nux mailto:n...@li.nux.ro>>
>> Data: piątek, 27 maja 2022 o 11:31
>> Do: users@cloudstack.apache.org  
>> mailto:users@cloudstack.apache.org>>
>> DW: Mariusz Wojtarek > >
>> Temat: Re: Odp: Cludstack vm backup
>> Thers is no backup driver for KVM at this time and there is none in 
>> development either, afaik.
>> You could always count on the volume snapshot feature, especially if you 
>> stick to single-disk/volume VMs.
>> HTH
>> ---
>> Nux
>> www.nux.ro  [1]
>> On 2022-05-27 09:34, Mariusz Wojtarek wrote:
>> So there is no backup solution for machines hosted on kvm ?
>> Błąd! Nie podano nazwy pliku.
>> Mariusz Wojtarek
>> Administrator iT
>> P: 22 335 28 00
>> E: mariusz.wojta...@support-online.pl 
>> 
>> www.support-online.pl  [2]
>> Poleczki 23 | 02-822 Warszawa
>> Błąd! Nie podano nazwy pliku. [3]   Błąd! Nie podano nazwy pliku. [4]
>> Od: Slavka Peleva > >
>> Data: piątek, 27 maja 2022 o 10:24
>> Do: users@cloudstack.apache.org  
>> mailto:users@cloudstack.apache.org>>
>> Temat: Re: Cludstack vm backup
>> Hi Mariusz,
>> The dummy backup plugin is only for tests. Only the Veeam backup provider
>> is supported for now by CS, which works with VMware HW.
>> Best regards,
>> Slavka
>> On Fri, May 27, 2022 at 11:09 AM Mariusz Wojtarek <
>> mariusz.wojta...@support-online.pl 
>> > wrote:
>>> Hi,
>>> I am using cloudstack with kvm should dummy backup works with vms host on
>>> kvm ? when I try to make backup using dummy it not working, every backup
>>> size is 1GB.
>>> 
>>> Support OnLine Sp. z o.o., ul. Poleczki 23, 02-822 Warszawa, NIP:
>>> 951-20-32-692, Regon: 017431965, KRS: 078497,
>>> XIII Wydzia? 

Re: NPE while listing Snapshots

[Antoine Boucher]

at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-16 18:18:49,748 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Remove job-2964 
from job monitoring
2022-05-16 18:18:50,692 DEBUG [c.c.a.ApiServlet] 
(qtp515715487-147221:ctx-4c1cd6c0) (logid:3652b6cf) ===START===  
0:0:0:0:0:0:0:1 -- GET  
command=queryAsyncJobResult=e6172481-968c-4ee4-993e-a9851af91c29=json=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:50,706 DEBUG [c.c.a.ApiServer] 
(qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) CIDRs from 
which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account 
{"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is 
allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:50,723 DEBUG [c.c.a.ApiServlet] 
(qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) ===END===  
0:0:0:0:0:0:0:1 -- GET  
command=queryAsyncJobResult=e6172481-968c-4ee4-993e-a9851af91c29=json=zLfGQdrCq6kxldkzrnAxYZ8HWQE

My snapshot.backup.to.secondary is configured to ‘true’

The snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5’ 
for id=117 no longer exist. All of the concerned snapshots reference 
'snapshots/16/111/...' for location but but directory 111 no longer exist.  110 
and 112 exists.

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com



> On May 13, 2022, at 9:18 AM, Suresh Anaparti  
> wrote:
> 
> Hi Antoine,
>  
> What is the error in the management server logs while deleting the ‘BackedUp’ 
> snapshots? Can you share the log if possible.
>  
> Are these snapshots backed up to secondary storage as well (and what’s the 
> config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes, 
> can you find the snapshot files at the install_path there? You can check the 
> install_path details in cloud.snapshot_store_ref  table with volume_id: 155 
> and store_role: Image.
>  
>  
> Regards,
> Suresh
>  
> 
>   
>   
> From: Antoine Boucher mailto:antoi...@haltondc.com>>
> Reply to: "users@cloudstack.apache.org <mailto:users@cloudstack.apache.org>" 
> mailto:users@cloudstack.apache.org>>
> Date: Wednesday, 11 May 2022 at 9:15 PM
> To: users 
> Subject: Re: NPE while listing Snapshots
>  
> Too fast id=146 also failed
>  
> {
>   "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
>   "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
>   "completed": "2022-05-11T11:43:01-0400",
>   "created": "2022-05-11T11:38:00-0400",
>   "jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
>   "jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
>   "jobinstancetype": "Snapshot",
>   "jobprocstatus": 0,
>   "jobresult": {
> "errorcode": 530,
> "errortext": "Failed to delete 
> snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove 
> snapshot "
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
> }
>  Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
> 
> 
> 
> 
> 
> Antoine Boucher
> antoi...@haltondc.com <mailto:antoi...@haltondc.com>
> [o] +1-226-505-9734
> www.haltondc.com <http://www.haltondc.com/>
>  
> “Data security made simple and affordable”
>  
> 
> 
> 
> 
> 
> 
> Confidentiality Warning: This message and any attachments are intended only 
> for the use of the intended recipient(s), are confidential, and may be 
> privileged. If you are not the intended recipient, you are hereby notified 
> that any review, retransmission, conversion to hard copy, copying, 
> circulation or other use of this message and any attachments is strictly 
> prohibited. If you are not the intended recipient, please notify the sender 
> immediately by return e-mail, and delete this message and any attachments 
> from your system.
> 
> 
> 
> 
>> On May 11, 2022, at 11:41 AM, Antoine Boucher > <mailto:antoi...@haltondc.com>> wrote:
>>  
>> Hello Suresh,
>>  
>> That primary storage was part of the local storage of a host that was 
>>

Re: NPE while listing Snapshots

[Antoine Boucher]

Too fast id=146 also failed

{
  "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
  "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
  "completed": "2022-05-11T11:43:01-0400",
  "created": "2022-05-11T11:38:00-0400",
  "jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
  "jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
  "jobinstancetype": "Snapshot",
  "jobprocstatus": 0,
  "jobresult": {
"errorcode": 530,
"errortext": "Failed to delete 
snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove 
snapshot "
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769




Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On May 11, 2022, at 11:41 AM, Antoine Boucher  wrote:
> 
> Hello Suresh,
> 
> That primary storage was part of the local storage of a host that was removed.
> 
> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool 
> WHERE id = 5
> ++--++---+-+-+
> | name   | uuid | pool_type  | scope | created | removed  
>|
> ++--++---+-+-+
> | cs-kvm03-local | NULL | Filesystem | HOST  | 2022-02-12 15:01:45 | 
> 2022-04-19 02:11:51 |
> ++--++---+-+——+
> 
> 
> SELECT volume_id, store_id, snapshot_id, store_role, install_path, size, 
> state FROM cloud.snapshot_store_ref WHERE store_id = '5';
> +---+--+-++---+-+---+
> | volume_id | store_id | snapshot_id | store_role | install_path  
> | 
> size| state |
> +---+--+-++---+-+---+
> |   155 |5 | 109 | Primary| 
> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5
>  | 85899345920 | Ready |
> |   155 |5 | 117 | Primary| 
> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46
>  | 85899345920 | Ready |
> |   155 |5 | 127 | Primary| 
> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014
>  | 85899345920 | Ready |
> |   155 |5 | 138 | Primary| 
> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529
>  | 85899345920 | Ready |
> +---+--+-++---+-+---+
> 
> 
> All 155 snapshots have the same account_id and domain_id.  
> 
> 
> SELECT id, uuid, name, status, path, created, removed, location_type FROM 
> cloud.snapshots WHERE volume_id = '155';
> +-+--+--+---+--+-+-+---+
> | id  | uuid | name | 
> status| path | created | removed | location_type |
> +-+--+--+---+--+-+-+---+
> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855 | 
> Destroyed | NULL | 2022-04-14 05:48:55 | NULL| NULL  |
> | 109 | 77d84bf6-f

NPE while listing Snapshots

[Antoine Boucher]

 | NULL| NULL  |
| 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934 | 
BackedUp  | NULL | 2022-04-27 05:49:34 | NULL| NULL  |
| 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934 | 
BackedUp  | NULL | 2022-04-28 05:49:34 | NULL| NULL  |
| 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935 | 
BackedUp  | NULL | 2022-04-29 05:49:35 | NULL| NULL  |
| 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935 | 
BackedUp  | NULL | 2022-04-30 05:49:35 | NULL| NULL  |
| 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935 | 
Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL  |
| 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742 | 
BackedUp  | NULL | 2022-05-09 01:57:42 | NULL| NULL  |
| 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242 | 
BackedUp  | NULL | 2022-05-09 17:02:42 | NULL| NULL  |
+-+--+--+---+--+-+-+---+

How would you suggest I resolve the issue moving forward?

I tried the following with error…

(nimbus)  > delete snapshot id=109
{
  "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
  "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
  "completed": "2022-05-11T11:32:08-0400",
  "created": "2022-05-11T11:32:08-0400",
  "jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
  "jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
  "jobinstancetype": "Snapshot",
  "jobprocstatus": 0,
  "jobresult": {
"errorcode": 530,
"errortext": "Command failed due to Internal Server Error"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
}
 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68

Deleting some table rows in the right sequence?

Same issue with id=117 to id=138, but I was able to delete id=146

Regards,
Antoine



> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti  
> wrote:
> 
> Hi Antoine,
> 
> The NPE error seems to be due to the primary storage with id '5', when 
> checking for snapshot revertible or not during list snapshots cmd. Is that 
> primary storage removed? Can you check with sql below.
> 
> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool 
> WHERE id = 5
> 
> 
> I think, the account/user might not have the permissions to list all 
> snapshots. You can check the account / domain details in snapshots table.
> 
> 
> Regards,
> Suresh
> 
> On 21/04/22, 6:50 PM, "Antoine Boucher"  wrote:
> 
>Thank you Suresh, you will find my answers below, 
> 
> 
> 
> 
> 
>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti  
>> wrote:
>> 
>> Hi Antoine,
>> 
>> What is the CloudStack version, where you see this issue.
> 
>Latest - 4.16.1
> 
> 
>> When you noticed this issue, is it while listing snapshots for a particular 
>> volume, or all volumes? Try list snapshots from API/cmk as well.
> 
>This is a customer VM, the account may have hit its snapshot limit during 
> a snapshot. 
> 
>cmk list snapshots > only lists 26 snapshots out of 62
> 
>cmk list snapshots volumeid= > Works for some volume ids, provides null 
> result while the WebUI lists the correct snapshots for others and provides 
> null results while show undefined error for one VM mentioned below
> 
> 
>> 
>> Also, Check the snapshots details for the volume (with issue) in the db, 
>> using the following sql queries.
>> 
>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes 
>> WHERE name LIKE ''
>> 
> 
>mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM 
> cloud.volumes WHERE name LIKE 'ROOT-110';
>
> +-+--+--+-+--+-+-+
>| id  | uuid | state| pool_id | path   
>   | size| removed |
>
> +-+--+--+-+--+-+-+
>| 111 | NULL

Re: No longer able to see any "Snapshots"

[Antoine Boucher]

Hello Suresh,

I deleted the snapshot policies using the UI and recreated them and it seems to 
fix the issue.  

Will the retention limit delete the snapshots from the previous policy?

Regards,
Antoine


> On May 4, 2022, at 3:25 AM, Suresh Anaparti  
> wrote:
> 
> Hi Antoine,
>  
> It seems, there are recurring snapshots scheduled on volume id: 
> 390198b4-45d4-425e-87c3-d0f11ef2ca11. Is that volume still exists and in 
> Ready state (Is it listed in UI? else, check with listVolumes API). Also, 
> check the snapshot policies for that volume (from UI if listed, else with API 
> listSnapshotPolicies using volumeid param) and remove them (from UI, or with 
> API deleteSnapshotPolicies using id from the listSnapshotPolicies API 
> response).
>  
>  
> Regards,
> Suresh
>  
> 
>   
>   
> From: Antoine Boucher mailto:antoi...@haltondc.com>>
> Reply to: "users@cloudstack.apache.org <mailto:users@cloudstack.apache.org>" 
> mailto:users@cloudstack.apache.org>>
> Date: Sunday, 1 May 2022 at 5:04 AM
> To: users 
> Subject: Re: No longer able to see any "Snapshots"
>  
> Hi Suresh,
>  
> Thank you for your reply, Yes this is the local storage of a Hiost that was 
> removed.  It is strange since all VM where migrated before. The host was then 
> put in maintenance and eventually removed.
>  
> What would be the fix?  Removing the snapshot record(s)?
>  
> Is this also related?  I’m seeing this events every 5 minutes,..
>  
>  
> 
> ERROR 
> <https://nimbus.haltondc.com:8443/client/#/event/d8ae2255-0f91-4041-9a19-ecd81bb38244>
>   
> SNAPSHOT.CREATE
> Created
> Error while creating entity for allocating snapshot
> system 
> <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=088d13e7-8bc4-11ec-9c43-001e67fd4838>
>   
> system 
> <https://nimbus.haltondc.com:8443/client/#/account?name=system=088d13e7-8bc4-11ec-9c43-001e67fd4838=true>
> 
> ROOT 
> <https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838>
>   
> 30 Apr 2022 19:32:41 
> INFO 
> <https://nimbus.haltondc.com:8443/client/#/event/50ea9c36-745d-4a44-935e-551f6294e439>
>
> SNAPSHOT.CREATE
> Scheduled
> creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11
> system 
> <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=80dc2363-7b2b-4614-a02f-b0e5a1782b11>
>   
> Networth-Abrams 
> <https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams=80dc2363-7b2b-4614-a02f-b0e5a1782b11=true>
>   
> Networth 
> <https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11>
>   
> 30 Apr 2022 19:32:41 
> ERROR 
> <https://nimbus.haltondc.com:8443/client/#/event/40bd9bc2-5899-49ff-9833-fb29dea7c0ba>
>   
> SNAPSHOT.CREATE
> Created
> Error while creating entity for allocating snapshot
> system 
> <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=088d13e7-8bc4-11ec-9c43-001e67fd4838>
>   
> system 
> <https://nimbus.haltondc.com:8443/client/#/account?name=system=088d13e7-8bc4-11ec-9c43-001e67fd4838=true>
> 
> ROOT 
> <https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838>
>   
> 30 Apr 2022 19:27:41 
> INFO 
> <https://nimbus.haltondc.com:8443/client/#/event/1dce5b57-4dbe-4201-b04a-6dfa63363014>
>
> SNAPSHOT.CREATE
> Scheduled
> creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11
> system 
> <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=80dc2363-7b2b-4614-a02f-b0e5a1782b11>
>   
> Networth-Abrams 
> <https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams=80dc2363-7b2b-4614-a02f-b0e5a1782b11=true>
>   
> Networth 
> <https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11>
>   
> 30 Apr 2022 19:27:41 
> ERROR 
> <https://nimbus.haltondc.com:8443/client/#/event/3c443f26-0e93-4f9e-9d69-192794c1efec>
>   
> SNAPSHOT.CREATE
> Created
> Error while creating entity for allocating snapshot
> system 
> <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=088d13e7-8bc4-11ec-9c43-001e67fd4838>
>   
> system 
> <https://nimbus.haltondc.com:8443/client/#/account?name=system=088d13e7-8bc4-11ec-9c43-001e67fd4838=true>
> 
> ROOT 
> <https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838>
>   
> 30 Apr 2022 19:22:41 
> INFO 
> <https://nimbus.haltondc.com:8443/client/#/event/ca2e5d1a-9ad1-4f9f-9d5a-45f3e9ec42ef>
>
> SNA

Re: Status of Virtual Router not updated

[Antoine Boucher]

Hi Vivek,

Thank you for your reply.  We recovered the VR no problem, and fortunately only 
test VMs where running of this VR.

I’m more concerned about the failure to notify the alarm.

-Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On May 2, 2022, at 4:37 AM, Vivek Kumar  
> wrote:
> 
> Try restart the VPC with cleanup options if you are using VPC, or simply 
> destroy the VR and re-create the VR.
> 
> 
> Regards,
> Vivek Kumar
> 
> 
>> On 01-May-2022, at 9:12 PM, Antoine Boucher  wrote:
>> 
>> Hello,
>> 
>> We use ACS 4.16.1 with multiple clusters and zones using Xen and KVM hosts.
>> 
>> We could not deploy a new VM on a specific cluster after one of our KVM 
>> hosts went down. Our investigation discovered that the VR of the network we 
>> were trying to use was on the failed host. We confirmed that the VR is dead; 
>> however, We see no alarm indication as the VR is showing as running and up. 
>> 
>> We have not uncovered why the system is not reporting the issue and 
>> re-spawning the VR on another host. 
>> 
>> Has anyone seen this issue? What could be the most likely reasons? 
>> Everything else seems to be working fine.
>> 
>> Regards,
>> Antoine
>> 
>> 
> 
> 
> -- 
> This message is intended only for the use of the individual or entity to 
> which it is addressed and may contain confidential and/or privileged 
> information. If you are not the intended recipient, please delete the 
> original message and any copy of it from your computer system. You are 
> hereby notified that any dissemination, distribution or copying of this 
> communication is strictly prohibited unless proper authorization has been 
> obtained for such action. If you have received this communication in error, 
> please notify the sender immediately. Although IndiQus attempts to sweep 
> e-mail and attachments for viruses, it does not guarantee that both are 
> virus-free and accepts no liability for any damage sustained as a result of 
> viruses.

Re: ACS 4.16 and xcp-ng - cant live storage migration

[Antoine Boucher]

Bonjour Benoit,

I had similar issues after I did a yum update and I was only able to fitx the 
issue by rebooting my hosts.

-Antoine

> On May 2, 2022, at 12:04 PM, benoit lair  wrote:
> 
> Hello all,
> 
> This is surely due to my yum update which updated to xcp 8.2.1
> 
> Do anybody know how to fix this ? xcp 8.2.1 is compatible ? would it be
> possible to add hypervisor capabilities without doing it in beta mode ?
> 
> Le lun. 2 mai 2022 à 16:15, benoit lair  a écrit :
> 
>> Hello folks,
>> 
>> I have a several issue
>> I try to live migrate my storage vm disks on a xcp-ng 8.2 cluster and i
>> cant live migrate
>> When clicking on the "Migrate volume" button, i have the following message
>> :
>> 
>> No primary storage pools available for migration
>> 
>> and  it generates this in logs : "the hypervisor doesn't support storage
>> motion."
>> 
>> 2022-05-02 15:52:33,120 DEBUG [c.c.a.ApiServlet]
>> (qtp1850777594-186961:ctx-2ee90dcf) (logid:1b094155) ===START===
>> 192.168.4.30 -- GET
>> id=b8d15b4c-93e9-4931-81ab-26a47ada32d5=findStoragePoolsForMigration=json
>> 2022-05-02 15:52:33,136 DEBUG [c.c.a.ApiServer]
>> (qtp1850777594-186961:ctx-2ee90dcf ctx-d6c062ae) (logid:1b094155) CIDRs
>> from which account 'Acct[a6441eae-68b8-11ec-acb6-96264736f9a1-admin] --
>> Account {"id": 2, "name": "admin", "uuid":
>> "a6441eae-68b8-11ec-acb6-96264736f9a1"}' is allowed to perform API calls:
>> 0.0.0.0/0,::/0
>> 2022-05-02 15:52:33,151 INFO [c.c.s.ManagementServerImpl]
>> (qtp1850777594-186961:ctx-2ee90dcf ctx-d6c062ae) (logid:1b094155) Volume
>> Vol[320|vm=191|DATADISK] is attached to any running vm. Looking for storage
>> pools in the cluster to which this volumes can be migrated.
>> 2022-05-02 15:52:33,157 ERROR [c.c.s.ManagementServerImpl]
>> (qtp1850777594-186961:ctx-2ee90dcf ctx-d6c062ae) (logid:1b094155)
>> Capabilities for host Host {"id": "2", "name": "xcp-cluster1-node2",
>> "uuid": "ae51578b-928c-4d25-9164-3bd7ca0afed4", "type"="Routing"} couldn't
>> be retrieved.
>> 2022-05-02 15:52:33,157 INFO [c.c.s.ManagementServerImpl]
>> (qtp1850777594-186961:ctx-2ee90dcf ctx-d6c062ae) (logid:1b094155) Volume
>> Vol[320|vm=191|DATADISK] is attached to a running vm and the hypervisor
>> doesn't support storage motion.
>> 2022-05-02 15:52:33,164 DEBUG [c.c.a.ApiServlet]
>> (qtp1850777594-186961:ctx-2ee90dcf ctx-d6c062ae) (logid:1b094155) ===END===
>> 192.168.4.30 -- GET
>> id=b8d15b4c-93e9-4931-81ab-26a47ada32d5=findStoragePoolsForMigration=json
>> 

Status of Virtual Router not updated

[Antoine Boucher]

Hello,

We use ACS 4.16.1 with multiple clusters and zones using Xen and KVM hosts.

We could not deploy a new VM on a specific cluster after one of our KVM hosts 
went down. Our investigation discovered that the VR of the network we were 
trying to use was on the failed host. We confirmed that the VR is dead; 
however, We see no alarm indication as the VR is showing as running and up. 

We have not uncovered why the system is not reporting the issue and re-spawning 
the VR on another host. 

Has anyone seen this issue? What could be the most likely reasons? Everything 
else seems to be working fine.

Regards,
Antoine

Re: No longer able to see any "Snapshots"

[Antoine Boucher]

Hi Suresh,

Thank you for your reply, Yes this is the local storage of a Hiost that was 
removed.  It is strange since all VM where migrated before. The host was then 
put in maintenance and eventually removed.

What would be the fix?  Removing the snapshot record(s)?

Is this also related?  I’m seeing this events every 5 minutes,..


 <>
ERROR 
<https://nimbus.haltondc.com:8443/client/#/event/d8ae2255-0f91-4041-9a19-ecd81bb38244>
SNAPSHOT.CREATE 
Created
Error while creating entity for allocating snapshot system 
<https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=088d13e7-8bc4-11ec-9c43-001e67fd4838>
system 
<https://nimbus.haltondc.com:8443/client/#/account?name=system=088d13e7-8bc4-11ec-9c43-001e67fd4838=true>
  ROOT 
<https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838>
30 Apr 2022 19:32:41 

 <>INFO 
<https://nimbus.haltondc.com:8443/client/#/event/50ea9c36-745d-4a44-935e-551f6294e439>
  SNAPSHOT.CREATE 
Scheduled
creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11system 
<https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=80dc2363-7b2b-4614-a02f-b0e5a1782b11>
Networth-Abrams 
<https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams=80dc2363-7b2b-4614-a02f-b0e5a1782b11=true>
Networth 
<https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11>
30 Apr 2022 19:32:41 

 <>ERROR 
<https://nimbus.haltondc.com:8443/client/#/event/40bd9bc2-5899-49ff-9833-fb29dea7c0ba>
 SNAPSHOT.CREATE 
Created
Error while creating entity for allocating snapshot system 
<https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=088d13e7-8bc4-11ec-9c43-001e67fd4838>
system 
<https://nimbus.haltondc.com:8443/client/#/account?name=system=088d13e7-8bc4-11ec-9c43-001e67fd4838=true>
  ROOT 
<https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838>
30 Apr 2022 19:27:41 

 <>INFO 
<https://nimbus.haltondc.com:8443/client/#/event/1dce5b57-4dbe-4201-b04a-6dfa63363014>
  SNAPSHOT.CREATE 
Scheduled
creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11system 
<https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=80dc2363-7b2b-4614-a02f-b0e5a1782b11>
Networth-Abrams 
<https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams=80dc2363-7b2b-4614-a02f-b0e5a1782b11=true>
Networth 
<https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11>
30 Apr 2022 19:27:41 

 <>ERROR 
<https://nimbus.haltondc.com:8443/client/#/event/3c443f26-0e93-4f9e-9d69-192794c1efec>
 SNAPSHOT.CREATE 
Created
Error while creating entity for allocating snapshot system 
<https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=088d13e7-8bc4-11ec-9c43-001e67fd4838>
system 
<https://nimbus.haltondc.com:8443/client/#/account?name=system=088d13e7-8bc4-11ec-9c43-001e67fd4838=true>
  ROOT 
<https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838>
30 Apr 2022 19:22:41 

 <>INFO 
<https://nimbus.haltondc.com:8443/client/#/event/ca2e5d1a-9ad1-4f9f-9d5a-45f3e9ec42ef>
  SNAPSHOT.CREATE 
Scheduled
creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11system 
<https://nimbus.haltondc.com:8443/client/#/accountuser?username=system=80dc2363-7b2b-4614-a02f-b0e5a1782b11>
Networth-Abrams 
<https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams=80dc2363-7b2b-4614-a02f-b0e5a1782b11=true>
Networth 
<https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11>
30 Apr 2022 19:22:41




Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti  
> wrote:
> 
> Hi Antoine,
> 
> The NPE error seems to be due to the primary storage with id '5', when 
> checking for snapshot revertible or not during list snapshots cmd. Is that 
> primary storage removed? Can you check with sql below.
> 
> SELECT name, uuid, pool_type, scope, created, removed FROM cloud.storage_pool 
> WHERE 

Re: No longer able to see any "Snapshots"

[Antoine Boucher]

--+
1 row in set (0.00 sec)

mysql> SELECT volume_id, store_id, store_role, install_path, size, state FROM 
cloud.snapshot_store_ref WHERE snapshot_id = '109';
+---+--++---+-+---+
| volume_id | store_id | store_role | install_path  
| size| state |
+---+--++---+-+---+
|   138 |5 | Primary| 
/var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5
 | 85899345920 | Ready |
|   138 |1 | Image  | 
snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5   
  | 85899345920 | Ready |
+---+--++---+-+---+
2 rows in set (0.00 sec)

Regards,
Antoine

> Regards,
> Suresh
> 
> On 21/04/22, 8:45 AM, "Antoine Boucher"  wrote:
> 
>Unfortunately I spoke too fast. The list is back but listing the snapshots 
> of the volume with issue still show a blank list with error “undefined”. 
> 
> 
> 
> 
> On Apr 20, 2022, at 23:11, Antoine Boucher  wrote:
> 
>I was able to recover from the situation by forcing a snapshot on the 
> volume with issue.
> 
>On Apr 20, 2022, at 22:00, Antoine Boucher  wrote:
> 
>I pin-pointed the volume with the issue by going through all volumes and 
> clicking the “view snapshot"
> 
> 
>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher  wrote:
>> 
>> I’m no longer able to see my list of snapshots (storage > snapshots), 
>> instead I see a small popup window with the word “undefined" on an empty 
>> list snapshot list.
>> 
>> Has anyone seen this issue?  I restart and rebooted the Management server 
>> but no resolution. 
>> 
>> Here are my logs:
>> 
>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] 
>> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START===  10.101.254.1 
>> -- GET  listall=true=1=20=listSnapshots=json
>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] 
>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from 
>> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account 
>> {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' 
>> is allowed to perform API calls: 0.0.0.0/0,::/0
>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] 
>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled 
>> exception executing api command: [Ljava.lang.String;@42fa8f7d
>> java.lang.NullPointerException
>>  at 
>> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>  at 
>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>  at 
>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>  at 
>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>  at 
>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>  at 
>> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>  at 
>> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>  at 
>> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>  at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>  at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>  at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>  at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>  at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>  at 
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>  at 
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>  at 
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>&

Re: No longer able to see any "Snapshots"

[Antoine Boucher]

Unfortunately I spoke too fast. The list is back but listing the snapshots of 
the volume with issue still show a blank list with error “undefined”. 

On Apr 20, 2022, at 23:11, Antoine Boucher  wrote:

I was able to recover from the situation by forcing a snapshot on the volume 
with issue.

On Apr 20, 2022, at 22:00, Antoine Boucher  wrote:

I pin-pointed the volume with the issue by going through all volumes and 
clicking the “view snapshot"


> On Apr 20, 2022, at 9:35 PM, Antoine Boucher  wrote:
> 
> I’m no longer able to see my list of snapshots (storage > snapshots), instead 
> I see a small popup window with the word “undefined" on an empty list 
> snapshot list.
> 
> Has anyone seen this issue?  I restart and rebooted the Management server but 
> no resolution. 
> 
> Here are my logs:
> 
> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] 
> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START===  10.101.254.1 -- 
> GET  listall=true=1=20=listSnapshots=json
> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] 
> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from 
> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account 
> {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' 
> is allowed to perform API calls: 0.0.0.0/0,::/0
> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] 
> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled 
> exception executing api command: [Ljava.lang.String;@42fa8f7d
> java.lang.NullPointerException
>   at 
> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>   at 
> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>   at 
> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>   at 
> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>   at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>   at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>   at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>   at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>   at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>   at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>   at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>   at 
> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(Sess

Re: No longer able to see any "Snapshots"

[Antoine Boucher]

I was able to recover from the situation by forcing a snapshot on the volume 
with issue.

On Apr 20, 2022, at 22:00, Antoine Boucher  wrote:

I pin-pointed the volume with the issue by going through all volumes and 
clicking the “view snapshot"


> On Apr 20, 2022, at 9:35 PM, Antoine Boucher  wrote:
> 
> I’m no longer able to see my list of snapshots (storage > snapshots), instead 
> I see a small popup window with the word “undefined" on an empty list 
> snapshot list.
> 
> Has anyone seen this issue?  I restart and rebooted the Management server but 
> no resolution. 
> 
> Here are my logs:
> 
> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] 
> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START===  10.101.254.1 -- 
> GET  listall=true=1=20=listSnapshots=json
> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] 
> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from 
> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account 
> {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' 
> is allowed to perform API calls: 0.0.0.0/0,::/0
> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] 
> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled 
> exception executing api command: [Ljava.lang.String;@42fa8f7d
> java.lang.NullPointerException
>   at 
> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>   at 
> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>   at 
> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>   at 
> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>   at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>   at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>   at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>   at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>   at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>   at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>   at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>   at 
> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.do

Re: No longer able to see any "Snapshots"

[Antoine Boucher]

I pin-pointed the volume with the issue by going through all volumes and 
clicking the “view snapshot"


> On Apr 20, 2022, at 9:35 PM, Antoine Boucher  wrote:
> 
> I’m no longer able to see my list of snapshots (storage > snapshots), instead 
> I see a small popup window with the word “undefined" on an empty list 
> snapshot list.
> 
> Has anyone seen this issue?  I restart and rebooted the Management server but 
> no resolution. 
> 
> Here are my logs:
> 
> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] 
> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START===  10.101.254.1 -- 
> GET  listall=true=1=20=listSnapshots=json
> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] 
> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from 
> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account 
> {"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' 
> is allowed to perform API calls: 0.0.0.0/0,::/0
> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] 
> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled 
> exception executing api command: [Ljava.lang.String;@42fa8f7d
> java.lang.NullPointerException
>   at 
> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>   at 
> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>   at 
> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>   at 
> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>   at 
> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>   at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>   at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>   at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>   at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>   at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>   at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>   at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>   at 
> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.ecl

No longer able to see any "Snapshots"

[Antoine Boucher]

lipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at 
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at 
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] (qtp515715487-292:ctx-a220f915 
ctx-c0fe9f91) (logid:624389ae) ===END===  10.101.254.1 -- GET  
listall=true=1=20=listSnapshots=json

Antoine Boucher


Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

Re: Unauthorized access to VR VM

[Antoine Boucher]

Thank you gents,

I just discovered that the customer was experimenting with the vr and left ip 
forwarding on port 22/22 to a vm created with the template with 
password=password!

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Apr 5, 2022, at 18:33, ahmed jabbar  wrote:

Dear Antoine,
You can simply block inbound connections on your virtual router public ip's
by any external firewall,And accept just outbond connections.
BR
Ahmed


On Tue, Apr 5, 2022 at 10:46 PM Antoine Boucher 
wrote:

> Someone has externally gained access to one of our VR vm and installed an
> application that tried to ssh to other ips on the web.
> 
> The VR started to miss health checks about a day ago, looking at the VR
> running process we discovered that the process ksoftirqd was 95% busy.  We
> killed the VR and discovered during our investigation from other systems
> that the vm was blasting the web trying to connect on port 22.
> Unfortunately, the vr has been deleted.
> 
> What could have happened? Any known security issues on the 4.16.1.0 vr
> template?
> 
> Regards,
> Antoine

Re: Adding management network and storage network to created root managed VM

[Antoine Boucher]

My management and Storage or on Physical Network 1 and Guest and Public on 
Physical Network 2. 

Other than adding routing rule in my core hardware router that would allow a 
specific network to be able to route to all  of the “Management” network or a 
potion of it.

I think I figured it out.

I added a new Guest traffic tag on Physical Network 1, both defined Physical 
Networks have Guest traffic type.

I ran into complexity about adding the appropriate tags to both Physical 
Network and Network Offering.  

But in my process of understanding the requirement of using tags properly, I 
added a tag to the DefaultL2NetworkOffering.  Fixed a typo the tag in 
Zone/Physical Network.  Continued on to tag the other Network offering using 
the correct tag spelling.  Now when update the tag of the 
DefaultL2NetworkOffering it complains  that it is unable to find physical 
network which match the old tag.

I can’t delete the default Network Offering and re-create it.

How would I resolve this situation?

Regards,
Antoine

 



> On Mar 13, 2022, at 8:04 AM, Nux  wrote:
> 
> Can't quite wrap my head around that question, can you please rephrase, add 
> more details?
> 
> ---
> Nux!
> www.nux.ro
> 
> On 2022-03-11 23:51, Antoine Boucher wrote:
>> Hello,
>> Is there any clever way to add the management or the storage network
>> to a created root managed VM?
>> Regards,
>> Antoine
>> Antoine Boucher
>> antoi...@haltondc.com

Adding management network and storage network to created root managed VM

[Antoine Boucher]

Hello,

Is there any clever way to add the management or the storage network to a 
created root managed VM?

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com

Re: KVM Host Disconnects from Management server

[Antoine Boucher]

Hello,

I have a suspicion that the situation may be related to a mtu issue on the 
tunnel. 

Standby. 

Antoine Boucher
antoi...@haltondc.com



Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Mar 5, 2022, at 13:22, Antoine Boucher  wrote:

Hi Wei,

Here are my corresponding log for a disconnection event.

Management server is 10.101.2.40 and the kit1-kvm01, host 15 is 10.91.1.11







Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”






Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Mar 4, 2022, at 2:29 PM, Wei ZHOU  wrote:
> 
> Hi,
> 
> It would be good to upload management server log as well.
> 
> -Wei
> 
> On Fri, 4 Mar 2022 at 19:56, Antoine Boucher  wrote:
> 
>> Any one?
>> 
>> Killing the Java processes does not recover from the situation.  The Java
>> processes end-up restarting and using 100% of all CPUs.
>> 
>> Regards,
>> Antoine
>> 
>> 
>> 
>> Confidentiality Warning: This message and any attachments are intended
>> only for the use of the intended recipient(s), are confidential, and may be
>> privileged. If you are not the intended recipient, you are hereby notified
>> that any review, retransmission, conversion to hard copy, copying,
>> circulation or other use of this message and any attachments is strictly
>> prohibited. If you are not the intended recipient, please notify the sender
>> immediately by return e-mail, and delete this message and any attachments
>> from your system.
>> 
>> 
>>> On Mar 3, 2022, at 9:17 AM, Antoine Boucher 
>> wrote:
>>> 
>>> I have 2 zones hosted from the same management server,  one larger local
>> to management server zone and one smaller “remote” zone using an IPSec
>> tunnel.
>>> 
>>> All is well except that recently one of the “remote” KVM host keeps
>> loosing connectivity with the management server.   It used to happened
>> every few days 24 but now it happens every few hours.
>>> 
>>> Other than rebooting, I’m only able to recover by killing the Java
>> process and restarting the CloudStack-Agent.
>>> 
>>> The tunnel is monitored and used by many other processes without any
>> interruptions.
>>> 
>>> Has anyone experienced anything similar before?
>>> 
>>> 
>>> Here are my host logs, the management server is 10.101.2.40 and the kvm
>> host is 10.91.1.11:
>>> 
>>> 2022-03-03 08:21:09,477 INFO  [kvm.storage.LibvirtStorageAdaptor]
>> (agentRequest-Handler-3:null) (logid:7ff2c4c1) Trying to fetch storage pool
>> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
>>> 2022-03-03 08:21:09,482 INFO  [kvm.storage.LibvirtStorageAdaptor]
>> (agentRequest-Handler-3:null) (logid:7ff2c4c1) Asking libvirt to refresh
>> storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
>>> 2022-03-03 08:22:03,802 WARN
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:null) (logid:4fc5e215) Expected 1 answers while
>> executing SetMonitorServiceCommand but received 3
>>> 2022-03-03 08:22:07,740 INFO
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-5:null) (logid:ba7d1810) Fetching health check result
>> for 169.254.4.222 and executing fresh checks: false
>>> 2022-03-03 08:22:13,881 INFO  [kvm.storage.LibvirtStorageAdaptor]
>> (agentRequest-Handler-3:null) (logid:f0ed50d5) Trying to fetch storage pool
>> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
>>> 2022-03-03 08:22:13,886 INFO  [kvm.storage.LibvirtStorageAdaptor]
>> (agentRequest-Handler-3:null) (logid:f0ed50d5) Asking libvirt to refresh
>> storage pool ce

Re: KVM Host Disconnects from Management server

[Antoine Boucher]

Any one?

Killing the Java processes does not recover from the situation.  The Java 
processes end-up restarting and using 100% of all CPUs.

Regards,
Antoine



Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Mar 3, 2022, at 9:17 AM, Antoine Boucher  wrote:
> 
> I have 2 zones hosted from the same management server,  one larger local to 
> management server zone and one smaller “remote” zone using an IPSec tunnel.  
> 
> All is well except that recently one of the “remote” KVM host keeps loosing 
> connectivity with the management server.   It used to happened every few days 
> 24 but now it happens every few hours.  
> 
> Other than rebooting, I’m only able to recover by killing the Java process 
> and restarting the CloudStack-Agent. 
> 
> The tunnel is monitored and used by many other processes without any 
> interruptions.
> 
> Has anyone experienced anything similar before?
> 
> 
> Here are my host logs, the management server is 10.101.2.40 and the kvm host 
> is 10.91.1.11:
> 
> 2022-03-03 08:21:09,477 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-3:null) (logid:7ff2c4c1) Trying to fetch storage pool 
> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
> 2022-03-03 08:21:09,482 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-3:null) (logid:7ff2c4c1) Asking libvirt to refresh 
> storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
> 2022-03-03 08:22:03,802 WARN  
> [resource.virtualnetwork.VirtualRoutingResource] 
> (agentRequest-Handler-1:null) (logid:4fc5e215) Expected 1 answers while 
> executing SetMonitorServiceCommand but received 3
> 2022-03-03 08:22:07,740 INFO  
> [resource.virtualnetwork.VirtualRoutingResource] 
> (agentRequest-Handler-5:null) (logid:ba7d1810) Fetching health check result 
> for 169.254.4.222 and executing fresh checks: false
> 2022-03-03 08:22:13,881 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-3:null) (logid:f0ed50d5) Trying to fetch storage pool 
> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
> 2022-03-03 08:22:13,886 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-3:null) (logid:f0ed50d5) Asking libvirt to refresh 
> storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
> 2022-03-03 08:22:17,641 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
> 2022-03-03 08:22:17,651 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
> 2022-03-03 08:22:17,659 INFO  [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
> 2022-03-03 08:26:45,714 INFO  [cloud.agent.Agent] (Agent-Handler-4:null) 
> (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting 
> reconnection while we still have 0 commands in progress.
> 2022-03-03 08:26:45,715 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
> (logid:3959c447) NioClient connection closed
> 2022-03-03 08:26:45,716 INFO  [cloud.agent.Agent] (Agent-Handler-4:null) 
> (logid:3959c447) Reconnecting to host:10.101.2.40
> 2022-03-03 08:26:45,716 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
> (logid:3959c447) Connecting to 10.101.2.40:8250
> 2022-03-03 08:26:45,733 INFO  [utils.nio.Link] (Agent-Handler-4:null) 
> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties
> 2022-03-03 08:27:15,762 WARN  [utils.nio.Link] (Agent-Handler-4:null) 
> (logid:3959c447) This SSL engine was forced to close inbound due to end of 
> stream.
> javax.net.ssl.SSLException: closing inbound before receiving peer's 
> close_notify
>   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
>   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
>   at 
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339)
>   at 
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295)
>   at 
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286)
>   at 
> java.base/sun.security.ssl.

Re: microvm support

[Antoine Boucher]

Mostly used for virtual network appliances and web services, we used 
Firecracker on OpenNebula.  That is a feature we used and liked, however, we 
could not appreciate their storage paradigm but then again the networking was 
too basic for our needs.  

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Mar 3, 2022, at 10:41 AM, Nux  wrote:
> 
> Antoine,
> 
> Out of curiosity, what would be your use/business case for such microVM 
> feature?
> 
> Perhaps I'm misinformed, but had a quick look at it and it's just a "slim" 
> KVM. It seems like it would make sense to use something like this if you have 
> significant "serverless" type of workloads, or indeed sell such thing 
> publicly.
> I can see why Amazon wants Firecracker, it goes with their narrative (and 
> "hype" may I add) for "serverless".
> 
> PS: Opennebula is pretty nice, love the "contextualisation", very powerful; 
> but yeah, quirky, like everything else.
> 
> ---
> Nux!
> www.nux.ro
> 
> On 2022-03-03 15:30, Antoine Boucher wrote:
>> microVM support is definitely a feature I’m missing that we were
>> accustom to from OpenNebula, fortunately CloudStack makes up for many
>> OpenNebula weirdness…
>> -Antoine
>>> On Mar 3, 2022, at 10:04 AM, Wei ZHOU  wrote:
>>> Hi Ricardo,
>>> CloudStack does not support customized qemu machine type for vms or
>>> templates (All VMs use `pc` or `q35`) .
>>> The feature is currently on our idea board.
>>> -Wei
>>> On Thu, 3 Mar 2022 at 13:00, Ricardo Pertuz 
>>> wrote:
>>>> Hi,
>>>> Is there a way to deploy microvm (firecracker approach) using qemu on
>>>> Cloudstack?
>>>> As explained in this link
>>>> https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
>>>> Regards,

Expected 1 answers while executing SetMonitorServiceCommand but received 3

[Antoine Boucher]

Hello,

Is the following issue significant?

From one of my kvm host logs:
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) 
(logid:69ceff92) Expected 1 answers while executing SetMonitorServiceCommand 
but received 3
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) 
(logid:5b95c453) Fetching health check result for 169.254.4.222 and executing 
fresh checks: false

I was looking at issue: vr: fix python exception when configure VRs #4489 


I'm using ACS 4.16 with KVM qemu-ev on CentOS 7.9

Regards,
Antoine

Re: microvm support

[Antoine Boucher]

microVM support is definitely a feature I’m missing that we were accustom to 
from OpenNebula, fortunately CloudStack makes up for many OpenNebula weirdness…

-Antoine


> On Mar 3, 2022, at 10:04 AM, Wei ZHOU  wrote:
> 
> Hi Ricardo,
> 
> CloudStack does not support customized qemu machine type for vms or
> templates (All VMs use `pc` or `q35`) .
> The feature is currently on our idea board.
> 
> -Wei
> 
> On Thu, 3 Mar 2022 at 13:00, Ricardo Pertuz 
> wrote:
> 
>> Hi,
>> 
>> Is there a way to deploy microvm (firecracker approach) using qemu on
>> Cloudstack?
>> As explained in this link
>> https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
>> 
>> Regards,
>> 

Re: microvm support

[Antoine Boucher]

Hi Ricardo,

I’m also interested in this topic as opposed to lxc

Regards,
Antoine



> On Mar 3, 2022, at 6:59 AM, Ricardo Pertuz  wrote:
> 
> Hi,
> 
> Is there a way to deploy microvm (firecracker approach) using qemu on 
> Cloudstack?
> As explained in this link
> https://qemu.readthedocs.io/en/latest/system/i386/microvm.html
> 
> Regards,

KVM Host Disconnects from Management server

[Antoine Boucher]

I have 2 zones hosted from the same management server,  one larger local to 
management server zone and one smaller “remote” zone using an IPSec tunnel.  

All is well except that recently one of the “remote” KVM host keeps loosing 
connectivity with the management server.   It used to happened every few days 
24 but now it happens every few hours.  

Other than rebooting, I’m only able to recover by killing the Java process and 
restarting the CloudStack-Agent. 

The tunnel is monitored and used by many other processes without any 
interruptions.

Has anyone experienced anything similar before?


Here are my host logs, the management server is 10.101.2.40 and the kvm host is 
10.91.1.11:

2022-03-03 08:21:09,477 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:7ff2c4c1) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:21:09,482 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:7ff2c4c1) Asking libvirt to refresh 
storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
2022-03-03 08:22:03,802 WARN  [resource.virtualnetwork.VirtualRoutingResource] 
(agentRequest-Handler-1:null) (logid:4fc5e215) Expected 1 answers while 
executing SetMonitorServiceCommand but received 3
2022-03-03 08:22:07,740 INFO  [resource.virtualnetwork.VirtualRoutingResource] 
(agentRequest-Handler-5:null) (logid:ba7d1810) Fetching health check result for 
169.254.4.222 and executing fresh checks: false
2022-03-03 08:22:13,881 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:f0ed50d5) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:22:13,886 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:f0ed50d5) Asking libvirt to refresh 
storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c
2022-03-03 08:22:17,641 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:22:17,651 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:22:17,659 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool 
cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt
2022-03-03 08:26:45,714 INFO  [cloud.agent.Agent] (Agent-Handler-4:null) 
(logid:3959c447) Lost connection to host: 10.101.2.40. Attempting reconnection 
while we still have 0 commands in progress.
2022-03-03 08:26:45,715 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) NioClient connection closed
2022-03-03 08:26:45,716 INFO  [cloud.agent.Agent] (Agent-Handler-4:null) 
(logid:3959c447) Reconnecting to host:10.101.2.40
2022-03-03 08:26:45,716 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) Connecting to 10.101.2.40:8250
2022-03-03 08:26:45,733 INFO  [utils.nio.Link] (Agent-Handler-4:null) 
(logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties
2022-03-03 08:27:15,762 WARN  [utils.nio.Link] (Agent-Handler-4:null) 
(logid:3959c447) This SSL engine was forced to close inbound due to end of 
stream.
javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339)
at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295)
at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286)
at 
java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.reconnect(Agent.java:536)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-03-03 08:27:15,768 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) SSL: Handshake done
2022-03-03 08:27:15,768 INFO  [utils.nio.NioClient] (Agent-Handler-4:null) 
(logid:3959c447) Connected to 

Re: Multiple primary storage issue

[Antoine Boucher]

Use tags on storage and on templates, compute offering or instance creation. 

Antoine Boucher
antoi...@haltondc.com


Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Mar 1, 2022, at 19:02, jcapagcuan .  wrote:

Hi,

I have created 2 primary storage for my Cloudstack (4.16.0). Whenever I launch 
an instance from a template, it randomizes from the 2 primary storage on where 
it will put the root disk. Is there a way to restrict the root disk to a 
specific storage?

Thanks,
Irvin

Re: Virtual router issue on multiple hosts

[Antoine Boucher]

Hi Irvin,

We are also running xcp-ng clusters. I find xen much easier to diagnose with 
XenCenter or xen orchestra. 

If all the network creation seem right on the hosts with vlans, I would check 
the host firewall setting and the switch setting. 

Regards,
Antoine

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

On Mar 1, 2022, at 19:01, jcapagcuan .  wrote:

Hi,

I have 2 hosts (XCP-NG hosts) in my cloudstack. Whenever I launch an instance 
that is in a different host as the virtual routers, the instance cannot 
retrieve proper IP (internal, private) configuration and is not accessible even 
from the virtual router it is attached. But if the virtual router and the 
instance are on the same host, everything works fine. Why is that? I’m using 
4.16.0 btw.

Thanks,
Irvin

Re: Cannot mount NFS share as a primary storage

[Antoine Boucher]

Hi Irvin,

Have you solved your problem?  We are having same problem with TrueNAS Core for 
primary NFS storage.  We have tried about everything.  

However, TureNAS as a secondary storage works well. 

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Jan 27, 2022, at 10:43 PM, jcapagcuan .  wrote:
> 
> Configuration is all good. I tried mounting it on the same server manually 
> and it’s fine. I just can’t mount it using Cloudstack. Btw, I’m using XCP-NG 
> 8.2 as a host.
> 
> Thanks,
> Irvin
> 
>> On Jan 28, 2022, at 10:58 AM, pradeep pal  wrote:
>> 
>> Hi,
>> 
>> Please check the shared folder permissions over the NFS server and 
>> configuration file.
>> 
>> Thanks
>> Pradeep
>> 
>> Get Outlook for iOS<https://aka.ms/o0ukef>
>> 
>> From: SVI 
>> Sent: Friday, January 28, 2022 8:02:08 AM
>> To: users@cloudstack.apache.org 
>> Subject: Cannot mount NFS share as a primary storage
>> 
>> Hi,
>> 
>> I’m using 4.16.0.0 installed on Ubuntu 20.04 LTS. I was adding a NFS storage 
>> but is receiving errors. The NFS server is on a Debian server. I tried 
>> manually mounting the nfs path in the management server and it’s working 
>> fine, but when I try via cloudstack, I’m receiving errors. Below are the 
>> errors I’m receiving:
>> 
>> 2022-01-27 18:29:06,919 DEBUG [c.c.a.t.Request] 
>> (StatsCollector-5:ctx-ba68a5a0) (logid:11777161) Seq 1-2134143273420193900: 
>> Received:  { Ans: , MgmtId: 52229674989, via: 1(gpu01.nwhtc.com), Ver: v1, 
>> Flags: 10, { GetGPUStatsAnswer } }
>> 2022-01-27 18:29:08,034 DEBUG [c.c.a.ApiServlet] 
>> (qtp515715487-297:ctx-d758f8d6) (logid:1cb64509) ===START===  10.10.1.4 -- 
>> POST  command=createStoragePool=json
>> 2022-01-27 18:29:08,054 DEBUG [c.c.a.ApiServer] 
>> (qtp515715487-297:ctx-d758f8d6 ctx-10b23d47) (logid:1cb64509) CIDRs from 
>> which account 'Acct[9d7347a9-62f1-11ec-98f8-000c29b2b4c8-admin] -- Account 
>> {"id": 2, "name": "admin", "uuid": "9d7347a9-62f1-11ec-98f8-000c29b2b4c8"}' 
>> is allowed to perform API calls: 0.0.0.0/0,::/0
>> 2022-01-27 18:29:08,066 DEBUG 
>> [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] 
>> (qtp515715487-297:ctx-d758f8d6 ctx-10b23d47) (logid:1cb64509) createPool 
>> Params @ scheme - nfs storageHost - 10.1.1.188 hostPath - 
>> /cloudstack/primary port - -1
>> 2022-01-27 18:29:08,076 DEBUG 
>> [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] 
>> (qtp515715487-297:ctx-d758f8d6 ctx-10b23d47) (logid:1cb64509) creating pool 
>> primaary-hdd on  host 1
>> 2022-01-27 18:29:08,080 DEBUG [c.c.a.t.Request] 
>> (qtp515715487-297:ctx-d758f8d6 ctx-10b23d47) (logid:1cb64509) Seq 
>> 1-2134143273420193901: Sending  { Cmd , MgmtId: 52229674989, via: 
>> 1(gpu01.nwhtc.com), Ver: v1, Flags: 100011, 
>> [{"com.cloud.agent.api.CreateStoragePoolCommand":{"_createDatastore":"false","add":"true","pool":{"id":"6","uuid":"bd47fcce-63a4-3e5a-8153-ca212681c34b","host":"10.1.1.188","path":"/cloudstack/primary","port":"2049","type":"NetworkFilesystem"},"localPath":"/mnt//bd47fcce-63a4-3e5a-8153-ca212681c34b","wait":"0","bypassHostMaintenance":"false"}}]
>>  }
>> 2022-01-27 18:29:08,081 DEBUG [c.c.a.t.Request] 
>> (qtp515715487-297:ctx-d758f8d6 ctx-10b23d47) (logid:1cb64509) Seq 
>> 1-2134143273420193901: Executing:  { Cmd , MgmtId: 52229674989, via: 
>> 1(gpu01.nwhtc.com), Ver: v1, Flags: 100011, 
>> [{"com.cloud.agent.api.CreateStoragePoolCommand":{"_createDatastore":"false","add":"true","pool":{"id":"6","uuid":"bd47fcce-63a4-3e5a-8153-ca212681c34b","host":"10.1.1.188","path&q

Re: Implicit Dedicated Hosts

[Antoine Boucher]

Hi Daan et al.,

Following up on my previous message.

My top request for improvement is the handling ImplicitDedication during VM 
creation.

Business motivation: dedicated hosts are a cost-effective way for cloud 
customers to manage predictable cloud costs and manage over-provisioning 
profiles.

The implicitdedicationplanner should be implicit and not require the additional 
step of having to specify the Affinity group during VM creation.

Regards,
Antoine


Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Jan 17, 2022, at 9:57 AM, Antoine Boucher  wrote:
> 
> Hello Daan,
> 
> Thank you for your reply. Seamless dedication is a critical use case for us.  
>  Members of the dedicated resource should have implicit access based on the 
> set planner, ImplicitDedicationPlanner-Strict or 
> ImplicitDedicationPlanner-Preferred. 
> 
> From the 4.16.0 documentation:
> 
> "For implicit dedication: The administrator creates a compute service 
> offering and in the Deployment Planner field, chooses 
> ImplicitDedicationPlanner. Then in Planner Mode, the administrator specifies 
> either Strict or Preferred, depending on whether it is permissible to allow 
> some use of shared resources when dedicated resources are not available. 
> Whenever a user creates a VM based on this service offering, it is allocated 
> on one of the dedicated hosts."
> 
> Without selecting the Affinity group during instance creation, I would expect 
> the ImplicitDedicationPlanner-Strict only to choose dedicated resources and 
> fail if unavailable, or spillover to other undedicated available resources 
> for the ImplicitDedicationPlanner-Prefered.
> 
> I looked at issue 5803, and it seems to be the same issue. 
> 
> From my logs it seems that the dedicated host never has a chance to be 
> selected:
> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-111:ctx-494924a3 
> job-2176 ctx-735080ac) (logid:3f02ba82) DeploymentPlanner allocation 
> algorithm: null
> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-111:ctx-494924a3 
> job-2176 ctx-735080ac) (logid:3f02ba82) Trying to allocate a host and storage 
> pools from dc:1, pod:null,cluster:null, requested cpu: 1000, requested ram: 
> (768.00 MB) 805306368
> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-111:ctx-494924a3 
> job-2176 ctx-735080ac) (logid:3f02ba82) Is ROOT volume READY (pool already 
> allocated)?: No
> 2022-01-17 09:02:29,996 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
> (API-Job-Executor-111:ctx-494924a3 job-2176 ctx-735080ac) (logid:3f02ba82) 
> Deploy avoids pods: [], clusters: [], hosts: [11]
> 
> Host 11 is domain-dedicated to the user from the same domain trying to create 
> the instance.
> 
> [c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
> ctx-735080ac) (logid:3f02ba82) Host 12 found to be unsuitable for implicit 
> dedication c.c.d.ImplicitDedicationPlanner] 
> (API-Job-Executor-111:ctx-494924a3 job-2176 ctx-735080ac) (logid:3f02ba82) 
> Host 12 found to be running a vm created by a planner other than implicit.
> [c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
> ctx-735080ac) (logid:3f02ba82) Host 10 found to be unsuitable for implicit 
> dedication as it is running instances of another account
> [c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
> ctx-735080ac) (logid:3f02ba82) Host 10 found to be running a vm created by a 
> planner other than implicit.
> [c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
> ctx-735080ac) (logid:3f02ba82) Host 9 found to be unsuitable for implicit 
> dedication as it is running instances of another account
> c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
> ctx-735080ac) (logid:3f02ba82) Host 9 found to be running a vm created by a 
> planner other than implicit.
> ...
> 
> Host 11 never gets enumerated.
> 
> Regards,
> Antoine
> 
> 
>> On Jan 17, 2022, at 4:33 AM, Daan Hoogland > <mailto:daan.hoogl...@gmail.com>> wrote:
>> 
>> Antoine,
>> As far as I understand, what you want is a missing feature. You

Re: ACS setup with 200+ accounts per domain

[Antoine Boucher]

Hi Daan,

Thank you again for your response.

We decided to implement the following for our school customers from our limited 
knowledge of CloudStack and our time constraints.  

Here is our v1:

Initial Setup
A) We provisioned an external router for each school capable of supporting 
hundreds of VLAN and client-VPNs. Fortunately, a Mikrotik hEX does an excellent 
job for less than $100 CAD.

B) We provisioned a VPN and a /24 subnet with gateway on a dedicated VLAN for 
each student. 

C) For every school, we created a Domain, domain administrators, and enough 
accounts for every student; xxx-account01, xxx-account02, etc., with the user 
to accounts removed.

D) We created a new Shared Network Offering with no ACS services; no DHCP,etc. 
such that no virtual routers are created during instantiation. 

E) For each subnet VLAN of B) we created a shared network using D) and 
associated the network to a corresponding account; -network01 to 
-account01, -network02 to -account02, etc.

Ongoing Management
Domain admins (the schools) can now add and remove users to the created 
accounts and manage the client-VPNs throughout the school year.

All is automated on Mikrotik, and from what I read, it should also be simple to 
automate on ACS.

My top improvement request would be on the Implicit Dedication. I will follow 
up on a separate message.

Regards,
Antoine
  

Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”





Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.


> On Jan 26, 2022, at 8:53 AM, Daan Hoogland  wrote:
> 
> Antoine, I hop you get your design right. I'm sure I would not help a lot
> except for confusing things, but let me try:
> 
> You could install a dedicated set of resources (zone/pods/clusters/hosts)
> per department/institution and assign resource admin role to a local guru
> to instantiate networks.
> You could create VPCs for lower level organisational units and let people
> organise themselves in tiers/guestnetworks.
> installing an IPv6 environment will be your way forward, but this is not
> yet supported by all parts of cloudstack.
> 
> please let us know if you succeed in designing something acceptable and let
> us know if there are any features you need/miss.
> 
> On Wed, Jan 19, 2022 at 7:09 PM Antoine Boucher 
> wrote:
> 
>> Hello,
>> 
>> We have been slowly migrating our various customer VMs to ACS configured
>> with Advanced Networking (without Security Group enabled) configured with
>> multiple KVM and XCP-NG clusters with great success.  After experimenting
>> with Open Nebula and Open Stack for most of last year we are impressed with
>> ACS.
>> 
>> In addition to our traditional enterprise customers, we also have
>> education institutions using our infrastructure for classes and training.
>> What would be the best way to support a Domains with 200+ accounts with
>> their respective isolated network and some shared networks in ACS?
>> 
>> We can assign new hosts, external gateways, vlan, vxlan, etc., but one
>> public ipv4 per account would be undesirable.
>> 
>> We our current knowledge, the out-of-the-box networking scalability seems
>> to be a limiting factor for us. We have been experimenting with different
>> permutations for a few weeks.
>> 
>> We've also tried using hardware routers for gateway and VPN termination.
>> As such, we dedicated a router for VPNs with 200 predefined VLANs and
>> subnets. 200 L2 networks are then defined with each VLAN-id and assigned to
>> an account as their "isolated" network (with Source NAT). A domain shared
>> network is also defined for intra-account communication. However, the root
>> admin can only do the network definition and association to the account.
>> Ideally, the use case would be for the domain admin to define and assign or
>> the account to create the "isolated" network.
>> 
>> We could always deploy a new zone with different networking configuration
>> if it would help.
>> 
>> Any suggestion would be appreciated.
>> 
>> Regards,
>> Antoine
>> 
>> 
> 
> -- 
> Daan

ACS setup with 200+ accounts per domain

[Antoine Boucher]

Hello,

We have been slowly migrating our various customer VMs to ACS configured with 
Advanced Networking (without Security Group enabled) configured with multiple 
KVM and XCP-NG clusters with great success.  After experimenting with Open 
Nebula and Open Stack for most of last year we are impressed with ACS.

In addition to our traditional enterprise customers, we also have education 
institutions using our infrastructure for classes and training. What would be 
the best way to support a Domains with 200+ accounts with their respective 
isolated network and some shared networks in ACS?

We can assign new hosts, external gateways, vlan, vxlan, etc., but one public 
ipv4 per account would be undesirable.

We our current knowledge, the out-of-the-box networking scalability seems to be 
a limiting factor for us. We have been experimenting with different 
permutations for a few weeks.

We've also tried using hardware routers for gateway and VPN termination. As 
such, we dedicated a router for VPNs with 200 predefined VLANs and subnets. 200 
L2 networks are then defined with each VLAN-id and assigned to an account as 
their "isolated" network (with Source NAT). A domain shared network is also 
defined for intra-account communication. However, the root admin can only do 
the network definition and association to the account. Ideally, the use case 
would be for the domain admin to define and assign or the account to create the 
"isolated" network.

We could always deploy a new zone with different networking configuration if it 
would help.
 
Any suggestion would be appreciated.

Regards,
Antoine

Re: Implicit Dedicated Hosts

[Antoine Boucher]

Hello Daan,

Thank you for your reply. Seamless dedication is a critical use case for us.   
Members of the dedicated resource should have implicit access based on the set 
planner, ImplicitDedicationPlanner-Strict or 
ImplicitDedicationPlanner-Preferred. 

From the 4.16.0 documentation:

"For implicit dedication: The administrator creates a compute service offering 
and in the Deployment Planner field, chooses ImplicitDedicationPlanner. Then in 
Planner Mode, the administrator specifies either Strict or Preferred, depending 
on whether it is permissible to allow some use of shared resources when 
dedicated resources are not available. Whenever a user creates a VM based on 
this service offering, it is allocated on one of the dedicated hosts."

Without selecting the Affinity group during instance creation, I would expect 
the ImplicitDedicationPlanner-Strict only to choose dedicated resources and 
fail if unavailable, or spillover to other undedicated available resources for 
the ImplicitDedicationPlanner-Prefered.

I looked at issue 5803, and it seems to be the same issue. 

From my logs it seems that the dedicated host never has a chance to be selected:
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-111:ctx-494924a3 
job-2176 ctx-735080ac) (logid:3f02ba82) DeploymentPlanner allocation algorithm: 
null
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-111:ctx-494924a3 
job-2176 ctx-735080ac) (logid:3f02ba82) Trying to allocate a host and storage 
pools from dc:1, pod:null,cluster:null, requested cpu: 1000, requested ram: 
(768.00 MB) 805306368
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-111:ctx-494924a3 
job-2176 ctx-735080ac) (logid:3f02ba82) Is ROOT volume READY (pool already 
allocated)?: No
2022-01-17 09:02:29,996 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-111:ctx-494924a3 job-2176 ctx-735080ac) (logid:3f02ba82) 
Deploy avoids pods: [], clusters: [], hosts: [11]

Host 11 is domain-dedicated to the user from the same domain trying to create 
the instance.

[c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
ctx-735080ac) (logid:3f02ba82) Host 12 found to be unsuitable for implicit 
dedication c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 
job-2176 ctx-735080ac) (logid:3f02ba82) Host 12 found to be running a vm 
created by a planner other than implicit.
[c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
ctx-735080ac) (logid:3f02ba82) Host 10 found to be unsuitable for implicit 
dedication as it is running instances of another account
[c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
ctx-735080ac) (logid:3f02ba82) Host 10 found to be running a vm created by a 
planner other than implicit.
[c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
ctx-735080ac) (logid:3f02ba82) Host 9 found to be unsuitable for implicit 
dedication as it is running instances of another account
c.c.d.ImplicitDedicationPlanner] (API-Job-Executor-111:ctx-494924a3 job-2176 
ctx-735080ac) (logid:3f02ba82) Host 9 found to be running a vm created by a 
planner other than implicit.
...

Host 11 never gets enumerated.

Regards,
Antoine


> On Jan 17, 2022, at 4:33 AM, Daan Hoogland  wrote:
> 
> Antoine,
> As far as I understand, what you want is a missing feature. You can
> dedicate a zone/pod/cluster/host to a domain but for the users in that
> domain to be forced on that domain they have to only have access to compute
> offerings with the implicitDedicationPlanner *and* when deploying, they
> still have to select the affinity group for the dedicated resourcegroup.
> 
> Your observations are correct and the only thing that is a bit less
> intuitive (afaics) is that the dedicated resources will only work with this
> specific setup. There is an issue (5803) out that requires about the same.
> (maybe a colleague?)
> 
> On Sat, Jan 15, 2022 at 3:37 AM Antoine Boucher 
> wrote:
> 
>> Hello,
>> 
>> I would like to implicitly force users of a Domain to use a specific sets
>> of hosts and block all other domain users to use the said hosts.
>> 
>> If I dedicate the hosts to the domain. It will prevent the other domains
>> to use them and allow the domain owner users to use them explicitly using
>> the affinity profile.
>> 
>> However, I have not found a way to implicitly make it work during instance
>> creation without having to select the affinity option.  Including using  a
>> specially created compute offering with the implicitDedicationPlanner
>> strict or preferred.
>> 
>> From the logs, I see that the dedicated hosts are removed right away from
>> the potential hosts candidate regardless of the user creating the instance
>> being a member on the domain owner.
>> 
>> Perhaps I’m not under

Implicit Dedicated Hosts

[Antoine Boucher]

Hello,

I would like to implicitly force users of a Domain to use a specific sets of 
hosts and block all other domain users to use the said hosts. 

If I dedicate the hosts to the domain. It will prevent the other domains to use 
them and allow the domain owner users to use them explicitly using the affinity 
profile. 

However, I have not found a way to implicitly make it work during instance 
creation without having to select the affinity option.  Including using  a 
specially created compute offering with the implicitDedicationPlanner strict or 
preferred.   

From the logs, I see that the dedicated hosts are removed right away from the 
potential hosts candidate regardless of the user creating the instance being a 
member on the domain owner.  

Perhaps I’m not understanding the feature properly, what am I missing?

Regards,
Antoine



Antoine Boucher
antoi...@haltondc.com
[o] +1-226-505-9734
www.haltondc.com

“Data security made simple and affordable”



Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.