Yes, cloudstack will copy the template from secondary storage to primary
storage again
On Wednesday, July 10, 2024, Marty Godsey <mar...@rudio.net> wrote:
> Does anyone know if this database entry can be deleted? See below.
> ________________________________
> From: Marty Godsey <mar...@rudio.net>
> Sent: Tuesday, July 9, 2024 8:32:17 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
>
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> I think I may have found the issue. The database table of
> template_spool_ref seems to have the wrong local_path and install_path for
> the systemvm template which I have confirmed is there.
>
> I tried changing it to the path I know is correct, but the it did not
> help. Can this reference be removed?
>
> From: Marty Godsey <mar...@rudio.net>
> Date: Tuesday, July 9, 2024 at 4:00 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> In the logs it keeps saying a host is in the avoid set..
>
> Where is this set, where can I modify it, etc.
>
> FirstFitAllocator has 3 hosts to check for allocation: [Host
> {"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"},
> Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:54,985 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-4:ctx-c152aba2 job-8587/job-8599 ctx-fb1d2cff
> FirstFitRoutingAllocator) (logid:d159f77b) Found 3 hosts for allocation
> after prioritization: [Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:54,985 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-4:ctx-c152aba2 job-8587/job-8599 ctx-fb1d2cff
> FirstFitRoutingAllocator) (logid:d159f77b) Host name: csc-n1, hostId: 5 is
> in avoid set, skipping this and trying other available hosts
> 2024-07-09 19:47:55,058 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-3:ctx-52fbf7ea job-8584/job-8596 ctx-3c8e35da
> FirstFitRoutingAllocator) (logid:ab7cbd9f) FirstFitAllocator has 3 hosts to
> check for allocation: [Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:55,072 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-3:ctx-52fbf7ea job-8584/job-8596 ctx-3c8e35da
> FirstFitRoutingAllocator) (logid:ab7cbd9f) Found 3 hosts for allocation
> after prioritization: [Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:55,072 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-3:ctx-52fbf7ea job-8584/job-8596 ctx-3c8e35da
> FirstFitRoutingAllocator) (logid:ab7cbd9f) Host name: csc-n1, hostId: 5 is
> in avoid set, skipping this and trying other available hosts
> 2024-07-09 19:47:55,103 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-4:ctx-c152aba2 job-8587/job-8599 ctx-fb1d2cff)
> (logid:d159f77b) VM instance {"id":712,"instanceName":"s-712-VM","type":"
> SecondaryStorageVm","uuid":"6254087a-bae5-4bfa-9e50-0379317b5599"} state
> transited from [Starting] to [Starting] with event [OperationRetry]. VM's
> original host: null, new host: Host {"id":7,"name":"csc-n3","type"
> :"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"}, host before
> state transition: Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}
>
> From: Marty Godsey <mar...@rudio.net>
> Date: Tuesday, July 9, 2024 at 3:10 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> The storage is not having issues. All the currently running VMs are
> running, I can create new VMs via XOA on that cluster using the primary SR.
>
> So, if the template is missing, and since I do not have a secondary server
> to download a new template, I acted like this was a new install and mounted
> the secondary storage on the Cloudstack server and ran the script for the
> template. This connected and ran without issue. I changed the system VM
> setting back to the default, since I thought this would this template and
> still no go.
>
> Whats the best way to go about replacing them template or determine if
> that is the real reason?
>
> From: Marty Godsey <mar...@rudio.net>
> Date: Tuesday, July 9, 2024 at 9:55 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> So the template shows as downloaded and ready in the console. What would
> be the best way to redownload this to the same ID if it is missing?
>
> From: Slavka Peleva <slav...@storpool.com.INVALID>
> Date: Tuesday, July 9, 2024 at 4:47 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> Hi Marty,
>
> My guess here is that the template `13f86c22-a422-459b-90d2-450a145724b3`
> from which you are creating a VM is missing on your primary storage, but
> CloudStack has the information that it is there. To confirm this you can
> check:
> - if `13f86c22-a422-459b-90d2-450a145724b3` exist on your primary storage
> - from the CloudStack DB check if the template was downloaded on your
> primary storage - `select * from template_spool_ref where
> local_path="13f86c22-a422-459b-90d2-450a145724b3";`
> Best regards,
> Slavka
>
> On Tue, Jul 9, 2024 at 10:39 AM Wei ZHOU <ustcweiz...@gmail.com> wrote:
>
> > Hi Marty,
> >
> > It looks like the issue is not caused by server capacity, but storage.
> >
> > 2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) ClusterScopeStoragePoolAllocator is returning [0]
> > suitable storage pools [[]].
> > 2024-07-09 06:57:20,306 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) Could not find any zone wide storage pool that
> > matched with any of the following tags [[]].
> > 2024-07-09 06:57:20,308 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) ZoneWideStoragePoolAllocator is returning [0]
> > suitable storage pools [[]].
> > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) No suitable pools found for volume:
> > Vol[1000|name=ROOT-598|vm=598|ROOT] under cluster: 3
> > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) No suitable pools found
> > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) No suitable storagePools found under this Cluster: 3
> >
> >
> >
> > On Tue, Jul 9, 2024 at 9:27 AM Marty Godsey <mar...@rudio.net> wrote:
> > >
> > > I am also getting this which is strange: Whey does it say there is not
> > enough capacity.
> > >
> > > 2024-07-09 01:08:22,201 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) VM instance
> > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy",
> "uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"}
> > state transited from [Starting] to [Starting] with event
> [OperationRetry].
> > VM's original host: null, new host: Host
> > {"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-
> 9931-410e-a482-7df361b14a50"},
> > host before state transition: null
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying
> > overprovisioning: 387200
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) We are allocating VM, increasing the used capacity of
> this
> > host:5
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Current Used CPU: 41400 , Free CPU:345800 ,Requested
> CPU:
> > 500
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Current Used RAM: (49.00 GB) 52613349376 , Free
> > RAM:(687.04 GB) 737701920768 ,Requested RAM: (1.00 GB) 1073741824
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) CPU STATS after allocation: for host: 5, old used:
> 41400,
> > old reserved: 0, actual total: 193600, total with overprovisioning:
> 387200;
> > new used:41900, reserved:0; requested cpu:500,alloc_from_last:false
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) RAM STATS after allocation: for host: 5, old used:
> (49.00
> > GB) 52613349376, old reserved: (0 bytes) 0, total: (736.04 GB)
> > 790315270144; new used: (50.00 GB) 53687091200, reserved: (0 bytes) 0;
> > requested mem: (1.00 GB) 1073741824,alloc_from_last:false
> > > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Host: 5 has cpu capability (cpu:88, speed:2200) to
> support
> > requested CPU: 1 and requested speed: 500
> > > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Checking if host: 5 has enough capacity for requested
> CPU:
> > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor:
> 2.0
> > > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying
> > overprovisioning: 387200
> > > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) We need to allocate to the last host again, so checking
> if
> > there is enough reserved capacity
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Reserved CPU: 0 , Requested CPU: 500
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB)
> > 1073741824
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) STATS: Failed to alloc resource from host: 5
> reservedCpu:
> > 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB)
> > 1073741824
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Host does not have enough reserved CPU available, cannot
> > allocate to this host.
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Checking if host: 5 has enough capacity for requested
> CPU:
> > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor:
> 2.0
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying
> > overprovisioning: 387200
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Free CPU: 345800 , Requested CPU: 500
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Free RAM: (687.04 GB) 737701920768 , Requested RAM:
> (1.00
> > GB) 1073741824
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Host has enough CPU and RAM available
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) STATS: Can alloc CPU from host: 5, used: 41400,
> reserved:
> > 0, actual total: 193600, total with overprovisioning: 387200; requested
> > cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) STATS: Can alloc MEM from host: 5, used: (49.00 GB)
> > 52613349376, reserved: (0 bytes) 0, total: (736.04 GB) 790315270144;
> > requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false ,
> > considerReservedCapacity?: true
> > > 2024-07-09 01:08:22,234 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Network id=208 is already implemented
> > > 2024-07-09 01:08:22,281 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Network id=210 is already implemented
> > > 2024-07-09 01:08:22,451 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-3dafbb92) (logid:72e1d259) HA health
> check
> > task is running...
> > > 2024-07-09 01:08:22,527 DEBUG [c.c.n.g.ControlNetworkGuru]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Reserved NIC for v-466-VM [ipv4:169.254.17.78
> > netmask:255.255.0.0 gateway:169.254.0.1]
> > > 2024-07-09 01:08:22,544 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Network id=209 is already implemented
> > > 2024-07-09 01:08:22,564 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Releasing ip address for instance=1462
> > > 2024-07-09 01:08:22,574 DEBUG [c.c.n.g.PodBasedNetworkGuru]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Allocated a nic NicProfile
> > {"broadcastUri":null,"iPv4Address":"10.61.0.91","id"
> :1462,"reservationId":"cc4156ed-873a-4485-a9a8-ff30725ea984","vmId":466}
> > for VM instance
> > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy",
> "uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"}
> > > 2024-07-09 01:08:22,608 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) template 316 is already in store:6, type:Image
> > >
> > > From: Rohit Yadav <rohit.ya...@shapeblue.com>
> > > Date: Tuesday, July 9, 2024 at 12:44 AM
> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> > > Subject: Re: Unable to create new VMs or Console Proxy
> > > WARNING: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and know
> > the content is safe.
> > >
> > >
> > > Hi Marty,
> > >
> > > Can you check and ensure all your CloudStack hosts (management server,
> > kvm hosts if applicable and ssvm, cpvm) are on the same version
> (4.19.0.2)
> > & restarted post update/upgrade.
> > >
> > > Once you've done this, can you share your management server logs where
> > you see the exceptions/errors around VM deployment. You may also want to
> > check if you've any capacity issues, or using any offering that's tagged
> > (host or storage tags) for which you don't have available capacity?
> > >
> > >
> > >
> > > Regards.
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Marty Godsey <mar...@rudio.net>
> > > Sent: Tuesday, July 9, 2024 07:29
> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> > > Subject: Re: Unable to create new VMs or Console Proxy
> > >
> > > Ok, figured, out the service offerings issues, I rebooted the database
> > and it came back. I still can not create VMs.
> > >
> > > It is stating that the storage is suitable, but it is in avoid set. I
> > have no idea why it is in avoid set since all hosts and the Cloudstack
> > management server can get to it.
> > >
> > > Where would I check this?
> > >
> > > From: Marty Godsey <mar...@rudio.net>
> > > Date: Monday, July 8, 2024 at 9:19 PM
> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> > > Subject: Unable to create new VMs or Console Proxy
> > > WARNING: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and know
> > the content is safe.
> > >
> > >
> > > Question. All of a sudden, I cannot create new VMs, and the console
> > proxy also can’t be created. As I spent the past hour looking, I have
> found
> > that in the console, when I go to offerings for compute, disk, or
> system, I
> > get a 404, and it says it can’t find anything for domain -1. But when I
> > look in the database, I see the offerings. I took this opportunity to
> > update to 4.19.0.2 since I was already on 4.19.0.1. At first I thought it
> > was storage but it is not. Running VMS and secStorage and VRs are all
> > running fine.
> > >
> > > Where should I look from here? Nothing was changed for the root domain.
> >
>
