Thanks all for this useful thread. I can confirm that a CoreOS template runs fine on an ESXi hypervisor in CloudStack 4.3.1. The type is set to "Other (64-bit)". I didn't set up the template so I don't know about relative performance with other type settings.
Regarding CoreOS, does anyone know what port(s) are required for CoreOS to do auto-updating? Obviously I can set an egress rule to allow all ports but I might want to be more restrictive. -Phillip