After more investigation, I found out that changing template's "OS Type" assignment affects how a VM is started as PV or HVM guest on either XenServer 6.5 or 7.0 clusters:
On XenServer 6.5 clusters, 64 bit RHEL guests with versions up to 6.7 (last 6.x version listed by ACS 4.9.3.0) are always started as PV guest, and they all run properly. On XenServer 7.0 clusters, 64 bit RHEL guests with versions up to 6.4 are started as PV guests and run properly, guests with version 6.5 onwards are started as HVM guest, and these won't start. So, the simple work around is to assign "RHEL 6.4 (64bit)" to all my rhel 6.x templates so that guests using these templates are always started as PV guest regardless the version of XenServer cluster they are running on. IMHO, I think this is a bug in CloudStack, that is all rhel 6.x guests should be started as PV guests regardless Xen hypervisor versions (I only verified on XenServer 6.5 and 7.0). Yiping On 4/11/18, 2:31 PM, "Yiping Zhang" <yzh...@marketo.com> wrote: Hi, list: I am in the process of upgrading my hypervisors clusters from XenServer 6.5 to 7.0 for all my ACS instances. My XS 7.0 clusters are patched up to XS70E050. During cluster rolling upgrade, VM instances are live migrated several times, eventually all of them running on XS 7.0 hosts. However, out of a hundred or so instances, a few of rhle 6.x VM got stuck at “Plex86/Bochs VGABios …” screen with at least one vCPU at 100%. Comparing VM parameters between running and stuck instances, I can see that running instances are PV guests while stuck instances are HVM guests. RHEL 7.x guest are always in HVM mode and always runs on both XS 6.5 and 7.0 hosts. Running RHEL6.x guests: HVM-boot-policy ( RW): HVM-boot-params (MRW): HVM-shadow-multiplier ( RW): 1.000 PV-args ( RW): graphical utf8 PV-bootloader ( RW): pygrub Stuck RHEL 6.x guests: HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: dc HVM-shadow-multiplier ( RW): 1.000 PV-args ( RW): PV-bootloader ( RW): Why does CloudStack convert just a few rhel 6.x instances into HVM mode, while leaving most in PV mode? How would I force them back to PV guests? Thanks for all the help Yiping