Hello CloudStack community, I would like to share an overview of the **CloudStack-based KVM HCI** we developed at TRTEK. This is not only a lab experiment: our goal was a **practical, widely usable solution in Turkey**, built on **Apache CloudStack**, that helps organizations move away from rising VMware licensing costs and from hesitation around less familiar hypervisors.
First, **thank you to everyone on the Apache CloudStack team** who has contributed. This product would not exist without CloudStack’s solid orchestration layer. Our work has been to complete it for Turkish public and private institutions—especially around **storage, installation, migration, replication, and console** behavior. ## Why we built this TRTEK works with roughly **100 public and private institutions in Turkey**. What we keep seeing in the field: - **VMware license and renewal costs** put real pressure on budgets; sustainability of licensing is often under discussion. - Alternatives such as **Proxmox** come up, but in government and compliance-sensitive environments concerns about support, roadmap, institutional acceptance, and operational familiarity (what we hear as **Proxmox hesitation**) slow or block adoption. - What was needed: a familiar cloud orchestrator (**CloudStack**), an open hypervisor (**KVM**), **single-image install**, thin storage, migration, and cross-site continuity—an **HCI shaped for our customers and ourselves**. So we built a vertical appliance / HCI on Rocky Linux + **Apache CloudStack 4.23 (KVM)**, aimed at **increasing CloudStack adoption in Turkey**. Current install image: **trtek-cs-setup59**. **The ISO is fully public.** Any organization or individual can download it free of charge: https://cloudstack.trtek.tr This is not a closed “only-for-us” package. We chose **open distribution** so the CloudStack ecosystem can be more visible in Turkey. ## Product intent - Small–medium **HCI**: no Ceph/OVS requirement; a simple KVM node model. - **Thin** storage for space efficiency; reclaim via discard/UNMAP. - Migrate existing **VMware / Hyper-V** workloads into CloudStack. - Continuous **CS→CS replication + promote** across two sites (DR / warm standby). - Console, live migrate, shared disk (CLVM), and HA that hold up in production. - Install and updates managed from **one channel** (public ISO + update panel). In short: a **CloudStack HCI usable in Turkey**—vertical integration from disk to console, migration to replication. ## Architecture (short) | Layer | What | |-------|------| | Install | Rocky kickstart + GUI wizard + appliance installer | | Hypervisor | KVM / libvirt, cloudbr0 | | Orchestration | CloudStack 4.23 management + agent | | Storage | LVM thin VG **trtekthin** (+ optional shared **CLVM / trtekvg**) | | Ops panel | :8091 — disks, migration, repl, updates (TR/EN) | | Distribution | Public ISO (trtek-cs-setupNN) + update CDs | Roles: **management (host/cloud)** and **extra KVM (client)**; clients join via a join code. ## How we use storage - Limited system area (root + swap); the large remainder is for VMs. - VM disks live under **trtekthin** thin pool (`data`) instead of fat LVs / ballooning qcow. - Thin volume **export** → XFS `/export` (templates, ISOs, helpers); VG activation on boot. - `issue_discards=1` — guest UNMAP reclaims thin space. - Thin orphan scan (visibility; does not auto-delete). - Shared disk via CLVM + sanlock; VG lock / LVM wrappers after hard power-off. - Surviving node can take an **exclusive** activation on the fallen host’s volume (with our CLVM patch). Model: **thin-first HCI**—overcommit, discard, shared disk, and single-node local in one language. ## What we changed on QEMU / libvirt Not an upstream fork—targeted production fixes: 1. **Console VNC** — console proxy speaks plain VNC; host VNC TLS off (otherwise the console sticks on “Connecting”). 2. **iothread poll hook** — I/O polling after guest start. 3. **VirtIO CD hook** — Windows migration / driver ISO paths. 4. **Sparse copy** — migration/repl/delta; BLKDISCARD when punch is unavailable; sparse path for live migrate. 5. **Optional trtek-qemu RPM** — special cases (e.g. LSI SAS); not required by the install ISO; BIOS/IDE migration uses stock Rocky qemu-kvm. 6. Simple KVM node sysctl/modprobe (IOMMU, aio limits, etc.). ## CloudStack patches (4.23.0.0) and why | Patch | Why | |-------|-----| | **Preallocation → falloc** | “Full” zero-fills the disk in qemu and hurts thin; value becomes `falloc`, XFS allocates without zero-fill. | | **Discard / UNMAP (virtio bus)** | UNMAP was keyed to virtio-blk; root bus stays `virtio` so discard never applied → thin reclaim. | | **ClvmStorageAdaptor** | After host loss, exclusive `lvchange -aey` without useless pool refresh. | | **MigrateKVMAsync** | Sparse migrate on live/panel move. | | **VirtualMachineMO** | vCenter HTTP 500 on not-yet-attached VMDK broke disk attach during migration; flow continues. | | **Console proxy multiviewer** | Second console must not drop the first. | Appliance complements: VMware/Hyper-V migration, CS→CS repl + promote, keep/import, two-node HA, 8080→8443 redirect, panel TLS upload proxy, public update channel. ## Migration and replication - **Migration:** VMware / Hyper-V → thin LV → KVM → CloudStack import (VirtIO / IDE as needed). - **CS→CS repl:** seed + dirty-bitmap delta; peer status Flowing / Lag / In sync. - **Promote:** planned or emergency; `start --paused` before import (OS never boots), then CloudStack start → correct VNC/XML and the real first boot (so console works). ## Outcome For institutions in Turkey facing VMware cost and alternative-hypervisor hesitation, we shipped a **CloudStack-centered HCI appliance** with thin storage, migration, and cross-site replication. The ISO is **public** at **https://cloudstack.trtek.tr**. Again, thank you to the CloudStack community and to every developer who contributed. This is **not a CloudStack fork**; it is a vertical product on **4.23**—storage and QEMU behavior, targeted patches, and open distribution for the Turkish field. We welcome feedback and collaboration. Best regards, Ümit Eyigün TRTEK — Systems [email protected] https://cloudstack.trtek.tr https://www.trtekyazilim.com
