Hi All, I'm sharing our latest API performance benchmarking results from ShapeBlue's performance tests. We conduct these tests to ensure CloudStack upgrades maintain or improve API response times at enterprise scale. As CloudStack deployments grow, API performance becomes critical for management operations and user experience. We systematically benchmark each release to:
* Detect performance regressions * Verify database query optimization at scale * Provide the community with real-world performance data Our testing focuses on the most common list operations that query large datasets. Test Environment Infrastructure: * Test Controller VM: Oracle Linux 8 server (running the apache/cloudstack-csbench<https://github.com/apache/cloudstack-csbench/tree/main> tool) * Management Server Under Test: CloudStack instance deployed as VM with dedicated vCenter resource pool: * 6 vCPUs (6.0 GHz) * 32 GB RAM * 20 GB disk storage * OS: Oracle Linux 8 * Hypervisor: VMware vSphere * Database: MySQL * KVM Host: Runs CloudStack system infrastructure (1 SSVM + 1 Console Proxy) * Storage: NFS (2 pools) Database Scale (Mock Objects): * 1 Zone, 1,000 Clusters * 2,453 Mock Routing Hosts (database entries simulating large deployment) * 2,370 VMs, 4,740 Volumes, 2,370 Networks This configuration tests database query performance under enterprise-scale loads without requiring thousands of physical resources. Testing Methodology We developed an automation script that orchestrates the complete performance testing workflow. The script handles benchmark execution, result validation, CSV archival, and automated diff report generation. Workflow: 1. Restore to baseline snapshot with mock data populated in database 2. Run baseline benchmark: * Measures API response times against current CloudStack version (e.g., 4.21.0.0) * Runs 100 iterations per API (~40 minutes; configurable) * Validates results and archives CSV reports 3. Upgrade CloudStack to target version (e.g., release candidate 4.22.0.0) 4. Run comparison benchmark: * Performs identical API performance tests against upgraded version * Compares response times against baseline 5. Automated diff report generated showing performance deltas (absolute time + percentage changes) APIs Benchmarked: * listAccounts, listDomains, listHosts, listNetworks, listVirtualMachines, listVolumes Performance Threshold: API degradation >20% requires investigation Test Results Release 4.20.1 → 4.20.2 (Maintenance Release) API Base (s) Patch (s) Diff Change listAccounts 6.0 5.99 -0.01 -0.17% listDomains 0.77 0.76 -0.01 -1.3% listHosts 1.44 1.49 +0.05 +3.47% listNetworks 7.68 7.7 +0.02 +0.26% listVirtualMachines 3.11 3.2 +0.09 +2.89% listVolumes 1.04 1.04 0.0 0.0% Observations: * Minor improvements in listAccounts and listDomains * Slight regression in listVirtualMachines (+0.09s, +2.89%) which falls within typical benchmark variance and represents no meaningful performance impact * All changes well below 5% threshold Conclusion: Very stable maintenance release Release 4.21.0.0 → 4.22.0.0 (LTS Release) API Base (s) Patch (s) Diff Change listAccounts 7.2 7.2 0.0 0.0% listDomains 1.62 1.68 +0.06 +3.7% listHosts 1.32 1.32 0.0 0.0% listNetworks 6.77 7.0 +0.23 +3.4% listVirtualMachines 2.78 2.85 +0.07 +2.52% listVolumes 0.9 0.94 +0.04 +4.44% Observations: * Largest change: listNetworks (+0.23s, +3.4%) * All APIs show <5% variation * No performance improvements, but stable behavior Conclusion: Minor variations within acceptable range Overall Assessment Both releases demonstrate stable API performance at enterprise scale: * 4.20.1 → 4.20.2: Excellent stability with minor improvements * 4.21.0.0 → 4.22.0.0: Acceptable variations for major LTS release All tested APIs remain well below our 20% investigation threshold. The largest absolute slowdown across both releases is listNetworks (+0.23s in 4.22.0.0), representing only 3.4% increase. Best regards, Rosi Rositsa Kyuchukova Senior QA Engineer s: | d: +44 203 603 0540 e: [email protected] | w: www.shapeblue.com | t: @shapeblue a: 3 London Bridge Street, 3rd floor, News Building, London United Kingdom Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design Build | CloudStack Consulting | CloudStack Software Engineering CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses
