Wei.. You are a lifesaver. Thank you for letting me know. This resolved the
issue. Now, to find out why it happened in the first place.
From: Wei ZHOU <ustcweiz...@gmail.com>
Date: Wednesday, July 10, 2024 at 10:40 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Unable to create new VMs or Console Proxy
WARNING: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
Yes, cloudstack will copy the template from secondary storage to primary
storage again
On Wednesday, July 10, 2024, Marty Godsey <mar...@rudio.net> wrote:
> Does anyone know if this database entry can be deleted? See below.
> ________________________________
> From: Marty Godsey <mar...@rudio.net>
> Sent: Tuesday, July 9, 2024 8:32:17 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
>
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> I think I may have found the issue. The database table of
> template_spool_ref seems to have the wrong local_path and install_path for
> the systemvm template which I have confirmed is there.
>
> I tried changing it to the path I know is correct, but the it did not
> help. Can this reference be removed?
>
> From: Marty Godsey <mar...@rudio.net>
> Date: Tuesday, July 9, 2024 at 4:00 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> In the logs it keeps saying a host is in the avoid set..
>
> Where is this set, where can I modify it, etc.
>
> FirstFitAllocator has 3 hosts to check for allocation: [Host
> {"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"},
> Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:54,985 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-4:ctx-c152aba2 job-8587/job-8599 ctx-fb1d2cff
> FirstFitRoutingAllocator) (logid:d159f77b) Found 3 hosts for allocation
> after prioritization: [Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:54,985 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-4:ctx-c152aba2 job-8587/job-8599 ctx-fb1d2cff
> FirstFitRoutingAllocator) (logid:d159f77b) Host name: csc-n1, hostId: 5 is
> in avoid set, skipping this and trying other available hosts
> 2024-07-09 19:47:55,058 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-3:ctx-52fbf7ea job-8584/job-8596 ctx-3c8e35da
> FirstFitRoutingAllocator) (logid:ab7cbd9f) FirstFitAllocator has 3 hosts to
> check for allocation: [Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:55,072 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-3:ctx-52fbf7ea job-8584/job-8596 ctx-3c8e35da
> FirstFitRoutingAllocator) (logid:ab7cbd9f) Found 3 hosts for allocation
> after prioritization: [Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}, Host
> {"id":7,"name":"csc-n3","type":"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"},
> Host {"id":6,"name":"csc-n2","type":"Routing","uuid":"137e43d2-
> 0862-4c78-b1e4-d21126f8408c"}]
> 2024-07-09 19:47:55,072 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-3:ctx-52fbf7ea job-8584/job-8596 ctx-3c8e35da
> FirstFitRoutingAllocator) (logid:ab7cbd9f) Host name: csc-n1, hostId: 5 is
> in avoid set, skipping this and trying other available hosts
> 2024-07-09 19:47:55,103 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-4:ctx-c152aba2 job-8587/job-8599 ctx-fb1d2cff)
> (logid:d159f77b) VM instance {"id":712,"instanceName":"s-712-VM","type":"
> SecondaryStorageVm","uuid":"6254087a-bae5-4bfa-9e50-0379317b5599"} state
> transited from [Starting] to [Starting] with event [OperationRetry]. VM's
> original host: null, new host: Host {"id":7,"name":"csc-n3","type"
> :"Routing","uuid":"d007845a-be22-401c-9d7a-c14ff4223c15"}, host before
> state transition: Host {"id":5,"name":"csc-n1","type"
> :"Routing","uuid":"52950eed-9931-410e-a482-7df361b14a50"}
>
> From: Marty Godsey <mar...@rudio.net>
> Date: Tuesday, July 9, 2024 at 3:10 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> The storage is not having issues. All the currently running VMs are
> running, I can create new VMs via XOA on that cluster using the primary SR.
>
> So, if the template is missing, and since I do not have a secondary server
> to download a new template, I acted like this was a new install and mounted
> the secondary storage on the Cloudstack server and ran the script for the
> template. This connected and ran without issue. I changed the system VM
> setting back to the default, since I thought this would this template and
> still no go.
>
> Whats the best way to go about replacing them template or determine if
> that is the real reason?
>
> From: Marty Godsey <mar...@rudio.net>
> Date: Tuesday, July 9, 2024 at 9:55 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> So the template shows as downloaded and ready in the console. What would
> be the best way to redownload this to the same ID if it is missing?
>
> From: Slavka Peleva <slav...@storpool.com.INVALID>
> Date: Tuesday, July 9, 2024 at 4:47 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to create new VMs or Console Proxy
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> Hi Marty,
>
> My guess here is that the template `13f86c22-a422-459b-90d2-450a145724b3`
> from which you are creating a VM is missing on your primary storage, but
> CloudStack has the information that it is there. To confirm this you can
> check:
> - if `13f86c22-a422-459b-90d2-450a145724b3` exist on your primary storage
> - from the CloudStack DB check if the template was downloaded on your
> primary storage - `select * from template_spool_ref where
> local_path="13f86c22-a422-459b-90d2-450a145724b3";`
> Best regards,
> Slavka
>
> On Tue, Jul 9, 2024 at 10:39 AM Wei ZHOU <ustcweiz...@gmail.com> wrote:
>
> > Hi Marty,
> >
> > It looks like the issue is not caused by server capacity, but storage.
> >
> > 2024-07-09 06:57:20,304 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) ClusterScopeStoragePoolAllocator is returning [0]
> > suitable storage pools [[]].
> > 2024-07-09 06:57:20,306 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) Could not find any zone wide storage pool that
> > matched with any of the following tags [[]].
> > 2024-07-09 06:57:20,308 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) ZoneWideStoragePoolAllocator is returning [0]
> > suitable storage pools [[]].
> > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) No suitable pools found for volume:
> > Vol[1000|name=ROOT-598|vm=598|ROOT] under cluster: 3
> > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) No suitable pools found
> > 2024-07-09 06:57:20,308 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-4:ctx-61e43dea job-7483/job-7492 ctx-440bf369)
> > (logid:c56ddd5b) No suitable storagePools found under this Cluster: 3
> >
> >
> >
> > On Tue, Jul 9, 2024 at 9:27 AM Marty Godsey <mar...@rudio.net> wrote:
> > >
> > > I am also getting this which is strange: Whey does it say there is not
> > enough capacity.
> > >
> > > 2024-07-09 01:08:22,201 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) VM instance
> > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy",
> "uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"}
> > state transited from [Starting] to [Starting] with event
> [OperationRetry].
> > VM's original host: null, new host: Host
> > {"id":5,"name":"csc-n1","type":"Routing","uuid":"52950eed-
> 9931-410e-a482-7df361b14a50"},
> > host before state transition: null
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying
> > overprovisioning: 387200
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) We are allocating VM, increasing the used capacity of
> this
> > host:5
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Current Used CPU: 41400 , Free CPU:345800 ,Requested
> CPU:
> > 500
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Current Used RAM: (49.00 GB) 52613349376 , Free
> > RAM:(687.04 GB) 737701920768 ,Requested RAM: (1.00 GB) 1073741824
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) CPU STATS after allocation: for host: 5, old used:
> 41400,
> > old reserved: 0, actual total: 193600, total with overprovisioning:
> 387200;
> > new used:41900, reserved:0; requested cpu:500,alloc_from_last:false
> > > 2024-07-09 01:08:22,214 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) RAM STATS after allocation: for host: 5, old used:
> (49.00
> > GB) 52613349376, old reserved: (0 bytes) 0, total: (736.04 GB)
> > 790315270144; new used: (50.00 GB) 53687091200, reserved: (0 bytes) 0;
> > requested mem: (1.00 GB) 1073741824,alloc_from_last:false
> > > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Host: 5 has cpu capability (cpu:88, speed:2200) to
> support
> > requested CPU: 1 and requested speed: 500
> > > 2024-07-09 01:08:22,217 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Checking if host: 5 has enough capacity for requested
> CPU:
> > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor:
> 2.0
> > > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying
> > overprovisioning: 387200
> > > 2024-07-09 01:08:22,218 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) We need to allocate to the last host again, so checking
> if
> > there is enough reserved capacity
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Reserved CPU: 0 , Requested CPU: 500
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB)
> > 1073741824
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) STATS: Failed to alloc resource from host: 5
> reservedCpu:
> > 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB)
> > 1073741824
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Host does not have enough reserved CPU available, cannot
> > allocate to this host.
> > > 2024-07-09 01:08:22,219 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Checking if host: 5 has enough capacity for requested
> CPU:
> > 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor:
> 2.0
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Hosts's actual total CPU: 193600 and CPU after applying
> > overprovisioning: 387200
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Free CPU: 345800 , Requested CPU: 500
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Free RAM: (687.04 GB) 737701920768 , Requested RAM:
> (1.00
> > GB) 1073741824
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Host has enough CPU and RAM available
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) STATS: Can alloc CPU from host: 5, used: 41400,
> reserved:
> > 0, actual total: 193600, total with overprovisioning: 387200; requested
> > cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
> > > 2024-07-09 01:08:22,220 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) STATS: Can alloc MEM from host: 5, used: (49.00 GB)
> > 52613349376, reserved: (0 bytes) 0, total: (736.04 GB) 790315270144;
> > requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false ,
> > considerReservedCapacity?: true
> > > 2024-07-09 01:08:22,234 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Network id=208 is already implemented
> > > 2024-07-09 01:08:22,281 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Network id=210 is already implemented
> > > 2024-07-09 01:08:22,451 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-3dafbb92) (logid:72e1d259) HA health
> check
> > task is running...
> > > 2024-07-09 01:08:22,527 DEBUG [c.c.n.g.ControlNetworkGuru]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Reserved NIC for v-466-VM [ipv4:169.254.17.78
> > netmask:255.255.0.0 gateway:169.254.0.1]
> > > 2024-07-09 01:08:22,544 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Network id=209 is already implemented
> > > 2024-07-09 01:08:22,564 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Releasing ip address for instance=1462
> > > 2024-07-09 01:08:22,574 DEBUG [c.c.n.g.PodBasedNetworkGuru]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) Allocated a nic NicProfile
> > {"broadcastUri":null,"iPv4Address":"10.61.0.91","id"
> :1462,"reservationId":"cc4156ed-873a-4485-a9a8-ff30725ea984","vmId":466}
> > for VM instance
> > {"id":466,"instanceName":"v-466-VM","type":"ConsoleProxy",
> "uuid":"94f2f2cc-15c9-41a7-a678-a9d1cd65d14a"}
> > > 2024-07-09 01:08:22,608 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
> > (Work-Job-Executor-81:ctx-db736df7 job-6013/job-6205 ctx-fa9daf4b)
> > (logid:7fa21822) template 316 is already in store:6, type:Image
> > >
> > > From: Rohit Yadav <rohit.ya...@shapeblue.com>
> > > Date: Tuesday, July 9, 2024 at 12:44 AM
> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> > > Subject: Re: Unable to create new VMs or Console Proxy
> > > WARNING: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and know
> > the content is safe.
> > >
> > >
> > > Hi Marty,
> > >
> > > Can you check and ensure all your CloudStack hosts (management server,
> > kvm hosts if applicable and ssvm, cpvm) are on the same version
> (4.19.0.2)
> > & restarted post update/upgrade.
> > >
> > > Once you've done this, can you share your management server logs where
> > you see the exceptions/errors around VM deployment. You may also want to
> > check if you've any capacity issues, or using any offering that's tagged
> > (host or storage tags) for which you don't have available capacity?
> > >
> > >
> > >
> > > Regards.
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Marty Godsey <mar...@rudio.net>
> > > Sent: Tuesday, July 9, 2024 07:29
> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> > > Subject: Re: Unable to create new VMs or Console Proxy
> > >
> > > Ok, figured, out the service offerings issues, I rebooted the database
> > and it came back. I still can not create VMs.
> > >
> > > It is stating that the storage is suitable, but it is in avoid set. I
> > have no idea why it is in avoid set since all hosts and the Cloudstack
> > management server can get to it.
> > >
> > > Where would I check this?
> > >
> > > From: Marty Godsey <mar...@rudio.net>
> > > Date: Monday, July 8, 2024 at 9:19 PM
> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> > > Subject: Unable to create new VMs or Console Proxy
> > > WARNING: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and know
> > the content is safe.
> > >
> > >
> > > Question. All of a sudden, I cannot create new VMs, and the console
> > proxy also can’t be created. As I spent the past hour looking, I have
> found
> > that in the console, when I go to offerings for compute, disk, or
> system, I
> > get a 404, and it says it can’t find anything for domain -1. But when I
> > look in the database, I see the offerings. I took this opportunity to
> > update to 4.19.0.2 since I was already on 4.19.0.1. At first I thought it
> > was storage but it is not. Running VMS and secStorage and VRs are all
> > running fine.
> > >
> > > Where should I look from here? Nothing was changed for the root domain.
> >
>
