bhouse-nexthop opened a new pull request, #14743:
URL: https://github.com/apache/cloudstack/pull/14743

   ### Description
   
   When a host is put into maintenance, or HA moves VMs off a host, each VM 
went back to its last host whenever that host was Up and had capacity, skipping 
the planner and `vm.allocation.algorithm`. Every migration records its source 
as the VM's last host, so in a rolling maintenance the VMs piled up on the 
hosts evacuated earlier. With memory overcommit, the capacity check let those 
hosts take far more VM memory than they have. In our case one host ended up 
with about 2.3 TiB of VM memory on 1.1 TiB of RAM, and VMs were OOM-killed 
there.
   
   `deployInVmLastHost()` no longer tries the last host for a migration plan, 
so the planner and allocators choose the destination.
   
   Fixes #14742
   
   #### Changes
   
   - New `DeploymentPlanningManagerImpl.shouldConsiderLastHost()`. It returns 
`false` for a plan marked `isMigrationPlan()`, then applies the existing 
checks: no last host, `ConsiderLastHost=false`, an HA tag.
   - Only the two host-evacuation paths mark a plan as a migration plan, and 
nothing read the flag until now:
     - `VirtualMachineManagerImpl.orchestrateMigrateAway()`, used by host 
maintenance and HA, including its HA planner fallback;
     - `ResourceManagerImpl.migrateAwayVmWithVolumes()`, used by host 
maintenance for VMs on local storage.
   
     The flag was added for KVM rolling maintenance (#3610).
   
   A user-initiated migration without a destination host already ignores the 
last host: `UserVmManagerImpl.chooseVmMigrationDestination()` clears it before 
planning.
   
   #### Not changed here
   
   - The last host is still recorded on migration and still preferred when a 
stopped VM is started.
   - The source host is still excluded.
   - With `migrate.vm.across.clusters=false`, 
`avoidOtherClustersForDeploymentIfMigrationDisabled()` still limits the plan to 
the last host's cluster. That is separate from choosing the host.
   - `findHostAndMigrate()`, which moves a VM to scale it, doesn't mark its 
plan as a migration plan, so it keeps its current behavior. Scaling doesn't 
evacuate a host.
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [x] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [x] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### How Has This Been Tested?
   
   - New unit tests in `DeploymentPlanningManagerImplTest`:
     - a start with a last host considers it;
     - a migration plan never does, even with `ConsiderLastHost=true`;
     - no last host, `ConsiderLastHost=false` or an HA tag still turn it off.
   - The cause was found on a 4.22 cluster of 12 KVM hosts during a rolling 
maintenance. In the management server's log for that day, 376 of 447 migrations 
off hosts in maintenance were placed by `deployInVmLastHost()` (`Last host 
[...] of VM [...] is UP and has enough capacity`), not by the allocator.
   - I'll add a test report from a live rolling maintenance.
   
   #### How did you try to break this feature and the system with this change?
   
   - When no other host fits, planning still fails as before, and 
`migrateAway()` retries with the HA planner. That retry is still a migration 
plan, so it doesn't fall back to the last host.
   - Starting a stopped VM, and HA restarting a VM after its host fails, don't 
use a migration plan, so they keep preferring the last host.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to