Hi Ricardo, Thanks for sharing, in addition to what Simon has asked; can you also share if this behaviour is consistent across different types of account (root admin, domain admin and/or user accounts) and domains, was there any change in the infrastructure after which this was observed?
We've introduced a small optimisation with https://github.com/apache/cloudstack/pull/7911 that you've tested, this is coming in 4.18.1.0 and later which calls the listVirtualMachinesMetrics API with a limited details view, this is used to render the instance list page in the UI. I think most users could see 5-10x improvements in the instance page load times. Regards. ________________________________ From: Simon Weller <siwelle...@gmail.com> Sent: Friday, August 25, 2023 19:18 To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Query to Cloudstack API is degraded Kuasar, How big is your installation, and what do you have your MySQL innodb_buffer_pool_size set to? Are you running a single, or multiple management servers? -Si On Fri, Aug 25, 2023 at 8:43 AM Kuasar <ricardo.per...@kuasar.co.invalid> wrote: > Hi Rohit, > > Thanks for you prompt reply. > > Could you say try with a MySQL 8.x? > > We are having this problem in production which is bigger and changing the > db engine right now will take us weeks > > you could try Ubuntu 22.04/20.04 > > Yes, we are using Ubuntu 20.04 > > Can you measure (say using dev tools of your browser) > > List-vms always takes around 5.6s checking on the timing tab of the > browser and > > http://<cloudstack mgmt host domain or IP>:8080/client/#/vm?details=min > http://<cloudstack mgmt host domain or > IP>:8080/client/#/vm?details=servoff,tmpl,nics > > Only 1 second, much better > > > BR, > > Ricardo Pertuz > > > On 25 Aug 2023, 6:41 AM -0500, Rohit Yadav <rohit.ya...@shapeblue.com>, > wrote: > > Hi Ricardo, > > > > Thanks for sharing. I can't say without more details on how to reproduce > the env/condition, or look at the DB server. It might be statistics or > something else, you could try to log mysql queries and see if they're > slow(ing down). > > > > Could you say try with a MySQL 8.x server and use a different host > server (for example, there is an issue reportedly affecting EL9 - > https://github.com/apache/cloudstack/issues/7910; you could try Ubuntu > 22.04/20.04 or EL8 to rule out the DB server). > > > > Can you measure (say using dev tools of your browser) the time it takes > to render the list VM API call (specifically the API would be called > listVirtualMachinesMetrics API), share the outcomes with us. > > > > Can you try the following URLs for the list instances page in the UI, > see if any makes difference in loading times for you: > > > > > > http://<cloudstack mgmt host domain or IP>:8080/client/#/vm?details=min > > http://<cloudstack mgmt host domain or > IP>:8080/client/#/vm?details=servoff,tmpl,nics > > > > Compared to the following that you've by default: > > > > > > http://<cloudstack mgmt host domain or IP>8080/client/#/vm > > > > I've raised a PR to explore an idea further: > > https://github.com/apache/cloudstack/pull/7911 > > > > > > Regards. > > > > ________________________________ > > From: Kuasar <ricardo.per...@kuasar.co.INVALID> > > Sent: Thursday, August 24, 2023 19:21 > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>; > users@cloudstack.apache.org <users@cloudstack.apache.org> > > Subject: Re: Query to Cloudstack API is degraded > > > > Hi Rohit, > > > > Thanks for asking, my reply below > > > > - which APIs you think have degraded -> > > listing instances > > > > - how have you concluded that they are degrading? -> > > we have an upper app that has a configured timeout and it wasn’t firing, > we had to change the value, and it is visible as well, it takes 5.6s > bringing 5 vms. > > > > - are those list API or other async/action oriented APIs? -> > > yes, it it async > > > > - which version of CloudStack and MySQL server is this, other details > about your env (hypervisor, scale/size of resources etc). > > ACS 4.18.0. —> 4GB 2Cores > > Mariadb 10.5.x -> 4GB 2Cores > > Hyp KVM > > > > - does restarting mgmt server help? any other information on how to > reproduce this? > > no change > > > > We are guessing it is related to database performance/index as it not > happening in QA > > > > > > Atte, > > > > Ricardo Pertuz > > > > > > On 24 Aug 2023, 6:33 AM -0500, Rohit Yadav <rohit.ya...@shapeblue.com>, > wrote: > > > Hi Ricardo, > > > > > > Can you share more details, such as; > > > > > > - which APIs you think have degraded > > > - how have you concluded that they are degrading? > > > - are those list API or other async/action oriented APIs? > > > - which version of CloudStack and MySQL server is this, other details > about your env (hypervisor, scale/size of resources etc) > > > - does restarting mgmt server help? any other information on how to > reproduce this? > > > > > > > > > Regards. > > > > > > ________________________________ > > > From: Kuasar <ricardo.per...@kuasar.co.INVALID> > > > Sent: Wednesday, August 23, 2023 22:00 > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > > Subject: Query to Cloudstack API is degraded > > > > > > During the last weeks we have been experiencing a degrading > performance in API queries to Cloudstack, is there a procedure you > recommend to check the possible bottleneck?, we are checking database > indices. > > > > > > > > > BR, > > > > > > Ricardo Pertuz > > > > > > > > > > > > > > > > > > > > > >
