Re: Document hostctl commands for XenServer

[Mike Belopuhov]

On Fri, Jul 21, 2017 at 17:49 +0200, Erik van Westen wrote:
> Hi Ingo,
> 
> Op 21-7-2017 om 16:20 schreef Ingo Schwarze:
> > Hi Mike,
> >
> > Mike Belopuhov wrote on Fri, Jul 21, 2017 at 03:14:08PM +0200:
> >
> >> Together with Maxim Khitrov we have figured out what needs to
> >> be set for XenServer
> > If XenServer were free software, i would say that the OpenBSD
> > operating system should detect whether it is running under
> > XenServer and then do all this automatically, by default, but ...
> >
> >> (that's a Citrix product based on Xen) to "recognize"
> >> the OpenBSD VM and let it do things like reboot and so on.
> >  ... that sounds like XenServer is a commercial product, so maybe
> > we don't want to bloat OpenBSD with specific code targeting it,
> > certainly not with large amounts of code.  If it can automatically
> > be done in just a handful of lines of code, i don't feel strongly
> > either way, even if XenServer is commercial.  I mean, other
> > virtual machine hosts are completely commercial in the first
> > place (unlike Xen) and we have large amounts of code in the kernel
> > specifically targeting them (unless i misunderstand).
> 
> A quick check makes sure that XenServer is open source and not commercial.
> http://xenserver.org.
>

Right, thanks for correction. I apologize for being too vague
and misleading Ingo, it wasn't intentional.

> [quote]
> 
> 
>   ABOUT XENSERVER
> 
> XenServer is the leading open source virtualization platform, powered by
> the Xen Project hypervisor
>  and the XAPI
> toolstack . It is used
> in the world's largest clouds and enterprises.
>  
> Commercial support for XenServer
>  is available from Citrix.
> [/quote]
> 
> [snip]
> 
> Maybe that helps.
> 
> Regards,
> 
> Erik

Re: Document hostctl commands for XenServer

[Maxim Khitrov]

On Fri, Jul 21, 2017 at 2:22 PM, Mike Belopuhov  wrote:
> On Fri, Jul 21, 2017 at 19:58 +0200, Ingo Schwarze wrote:
>> If these numbers are completely fake and irrelevant,
>> then saying so in one short sentence - or even in the
>> comment line above - may be sufficient.
>>
>> But this quote from Maxim fuels doubts:
>>
>> :: I don't know whether XenServer actually cares about what
>> :: version is reported, but if it does, this would be tied
>> :: to features supported by xen, xbf, and xnf drivers.
>> :: You typically update the tools with each new XenServer
>> :: release, which gives you most recent disk and
>> :: network drivers, at least on Windows.
>>
>> If that is true, than you have just built a time bomb:
>> YOU need to remember to regularly update the manual.
>> USERS need to remember to regularly update their rc.local.
>
> Yes and yes, however, these interfaces don't change often so
> there's no need to constantly update them.  We can leave this
> as a hint and an exercise for the reader.  I've changed a
> comment there saying that this is a version of "XenServer Tools".
> XenServer users must be able to comprehend what this means.
>
>> If the latter is true, this may need to be explained
>> in this manual page.  Of course, that can be figured
>> out and improved later, if needed.  Including a change
>> to explain where to get the right magic numbers from
>> rather than simply advertising some numbers that may be
>> good today, but not tomorrow.

See the following link for more info:

https://wiki.xenproject.org/wiki/XAPI_fake_presence_of_PV_drivers

It's XAPI that cares about this stuff, but the actual version number
doesn't seem to matter. The script they provide is intended to run on
the host, so it can get the current product version from
/etc/xensource-inventory. I don't think this information is exposed in
any way to the VM itself, at least not through XenStore.

One thing you can do, however, is get (but not set) viridian state via
`hostctl platform/viridian` (true/false). You can use that to decide
whether pvbus1 is needed or not. You can also report VM IP by setting
attr/eth0/{ip,ipv6}.

Here are the values used by other OSs:

CentOS 7:

attr/PVAddons/MajorVersion 7
attr/PVAddons/MinorVersion 0
attr/PVAddons/MicroVersion 0
attr/PVAddons/BuildVersion 130092
attr/PVAddons/Installed 1

data/os_uname = "3.10.0-514.21.2.el7.x86_64"
data/os_name = "CentOS Linux release 7.3.1611 (Core)"
data/os_distro = "centos"
data/os_majorver = "7"
data/os_minorver = "3"

Windows Server 2016:

attr/PVAddons/MajorVersion 7
attr/PVAddons/MinorVersion 1
attr/PVAddons/MicroVersion 0
attr/PVAddons/BuildVersion 825
attr/PVAddons/Installed 1

data/os_distro = "windows"
data/os_majorver = "6"
data/os_minorver = "2"
data/os_name = "Microsoft Windows Server 2016
Standard|C:\\Windows|\\Device\\Harddisk0\\Partition2"

-Max

Re: Document hostctl commands for XenServer

[Mike Belopuhov]

On Fri, Jul 21, 2017 at 19:58 +0200, Ingo Schwarze wrote:
> Hi Mike,
> 
> Mike Belopuhov wrote on Fri, Jul 21, 2017 at 07:08:06PM +0200:
> 
> > Thanks for the detailed respose, I share your outlook and in
> > this case it is better to keep this stuff in the userland since
> > we actually can do it just fine.
> 
> Fair enough!
> 
> > I will however add some text to the
> > hostctl(8) man page to highlight the fact that multiple pvbus
> > devices may be present on a single VM.
> 
> Makes sense to me.
> 
> > OK?
> 
> Reads well to me.
> 
> Optionally, consider the two following nits:
> 
>  1. It might make sense to mention in half a sentence why it is
> useful to set these properties, probably up front like in
> 
> When running under XenServer, to let the host know that the guest
> has finished initializing and to allow graceful restarts (or
> whatever), set the following XenStore properties with
> .Xr hostctl 8
> in
> .Xr rc.local 8 :
>

This sounds good, thanks.

> or (less convincing to me) after your existing text like so:
> 
> Without these settings, graceful restarts (or whatever)
> may not work.
> 
> In any case, you want "in rc.local", not "in the rc.local".
> 
>  2. Magic number alarm:
> 
> # XenServer Tools version
> hostctl attr/PVAddons/MajorVersion 6
> hostctl attr/PVAddons/MinorVersion 2
> hostctl attr/PVAddons/MicroVersion 0
> hostctl attr/PVAddons/BuildVersion 76888
> hostctl attr/PVAddons/Installed 1
> 
> With the changed content, this reads a bit like a HOWTO:
> Type these commands, but i won't tell you what they do.
>
> If these numbers are completely fake and irrelevant,
> then saying so in one short sentence - or even in the
> comment line above - may be sufficient.
>
> But this quote from Maxim fuels doubts:
> 
> :: I don't know whether XenServer actually cares about what
> :: version is reported, but if it does, this would be tied
> :: to features supported by xen, xbf, and xnf drivers.
> :: You typically update the tools with each new XenServer
> :: release, which gives you most recent disk and
> :: network drivers, at least on Windows.
> 
> If that is true, than you have just built a time bomb:
> YOU need to remember to regularly update the manual.
> USERS need to remember to regularly update their rc.local.

Yes and yes, however, these interfaces don't change often so
there's no need to constantly update them.  We can leave this
as a hint and an exercise for the reader.  I've changed a
comment there saying that this is a version of "XenServer Tools".
XenServer users must be able to comprehend what this means.

> If the latter is true, this may need to be explained
> in this manual page.  Of course, that can be figured
> out and improved later, if needed.  Including a change
> to explain where to get the right magic numbers from
> rather than simply advertising some numbers that may be
> good today, but not tomorrow.
> 
> Yours,
>   Ingo

Re: Document hostctl commands for XenServer

[Ingo Schwarze]

Hi Mike,

Mike Belopuhov wrote on Fri, Jul 21, 2017 at 07:08:06PM +0200:

> Thanks for the detailed respose, I share your outlook and in
> this case it is better to keep this stuff in the userland since
> we actually can do it just fine.

Fair enough!

> I will however add some text to the
> hostctl(8) man page to highlight the fact that multiple pvbus
> devices may be present on a single VM.

Makes sense to me.

> OK?

Reads well to me.

Optionally, consider the two following nits:

 1. It might make sense to mention in half a sentence why it is
useful to set these properties, probably up front like in

When running under XenServer, to let the host know that the guest
has finished initializing and to allow graceful restarts (or
whatever), set the following XenStore properties with
.Xr hostctl 8
in
.Xr rc.local 8 :

or (less convincing to me) after your existing text like so:

Without these settings, graceful restarts (or whatever)
may not work.

In any case, you want "in rc.local", not "in the rc.local".

 2. Magic number alarm:

# XenServer Tools version
hostctl attr/PVAddons/MajorVersion 6
hostctl attr/PVAddons/MinorVersion 2
hostctl attr/PVAddons/MicroVersion 0
hostctl attr/PVAddons/BuildVersion 76888
hostctl attr/PVAddons/Installed 1

With the changed content, this reads a bit like a HOWTO:
Type these commands, but i won't tell you what they do.

If these numbers are completely fake and irrelevant,
then saying so in one short sentence - or even in the
comment line above - may be sufficient.

But this quote from Maxim fuels doubts:

:: I don't know whether XenServer actually cares about what
:: version is reported, but if it does, this would be tied
:: to features supported by xen, xbf, and xnf drivers.
:: You typically update the tools with each new XenServer
:: release, which gives you most recent disk and
:: network drivers, at least on Windows.

If that is true, than you have just built a time bomb:
YOU need to remember to regularly update the manual.
USERS need to remember to regularly update their rc.local.
If the latter is true, this may need to be explained
in this manual page.  Of course, that can be figured
out and improved later, if needed.  Including a change
to explain where to get the right magic numbers from
rather than simply advertising some numbers that may be
good today, but not tomorrow.

Yours,
  Ingo

Re: Document hostctl commands for XenServer

[Mike Belopuhov]

On Fri, Jul 21, 2017 at 10:28 -0400, Maxim Khitrov wrote:
> On Fri, Jul 21, 2017 at 9:14 AM, Mike Belopuhov  wrote:
> > Hi,
> >
> > Together with Maxim Khitrov we have figured out what needs to
> > be set for XenServer (that's a Citrix product based on Xen) to
> > "recognize" the OpenBSD VM and let it do things like reboot and
> > so on.
> >
> > I'd like to get this documented in the xen(4) man page instead
> > of referring users to mailing list archives.
> >
> > There are two things that we can mention:
> >
> > 1) viridian capability, that XenServer comes with enabled by
> >default, interferes with hostctl: you need to either disable
> >it for your VM (if you have access) or MAKEDEV /dev/pvbus1
> >and use that with hostctl(8).
> >
> > 2) to let XenServer management software know that OpenBSD is
> >there in full glory we need to set a few XenStore properties
> >with hostctl(8).  User needs to do this on every boot, so
> >putting them somewhere around /etc/rc.local is necessary.
> >
> > I've come up with the diff below.  Please let me know if this
> > makes sense and if we can improve it.
> >
> > Maxim, can you please double check the script itself.  Are all
> > these values necessary?  I've changed a few things including
> > the BuildVersion value.
> >
> > Thanks.
> 
> Hi Mike,
> 
> attr/PVAddons/* contains version information about XenServer Tools
> installed in the VM, not the OS. Dinar copied the original values from
> tools v6.2. I don't know whether XenServer actually cares about what
> version is reported, but if it does, this would be tied to features
> supported by xen, xbf, and xnf drivers. You typically update the tools
> with each new XenServer release, which gives you most recent disk and
> network drivers, at least on Windows.
> 
> Only attr/PVAddons/{MajorVersion,MinorVersion} and data/updated are
> required to get graceful reboot and shutdown support in XenCenter, but
> I would leave the rest in there to avoid confusing any tools that
> might expect to find the other keys as well.
> data/{os_name,os_uname,os_distro} provide OS information, which is
> shown in VM properties (not required, but useful to have). Setting
> data/updated to 1 triggers a refresh of this information in
> XenStore/XenCenter.
> 
> -Max

Ok, thanks for clarifying this, I've updated the patch.

Re: Document hostctl commands for XenServer

[Mike Belopuhov]

On Fri, Jul 21, 2017 at 16:20 +0200, Ingo Schwarze wrote:
> Hi Mike,
> 
> Mike Belopuhov wrote on Fri, Jul 21, 2017 at 03:14:08PM +0200:
> 
> > Together with Maxim Khitrov we have figured out what needs to
> > be set for XenServer
> 
> If XenServer were free software, i would say that the OpenBSD
> operating system should detect whether it is running under
> XenServer and then do all this automatically, by default, but ...
> 
> > (that's a Citrix product based on Xen) to "recognize"
> > the OpenBSD VM and let it do things like reboot and so on.
> 
>  ... that sounds like XenServer is a commercial product, so maybe
> we don't want to bloat OpenBSD with specific code targeting it,
> certainly not with large amounts of code.  If it can automatically
> be done in just a handful of lines of code, i don't feel strongly
> either way, even if XenServer is commercial.  I mean, other
> virtual machine hosts are completely commercial in the first
> place (unlike Xen) and we have large amounts of code in the kernel
> specifically targeting them (unless i misunderstand).
> 
> > I'd like to get this documented in the xen(4) man page instead
> > of referring users to mailing list archives.
> 
> Sure, if you decide to not do this automatically by default,
> then documenting it in a manual page is a good idea, and xen(4)
> seems like the logical place to me - FWTW, i know nothing about Xen,
> pvbus(4), or hostctl(8).
>

Thanks for the detailed respose, I share your outlook and in
this case it is better to keep this stuff in the userland since
we actually can do it just fine. As Maxim has pointed out we have
to fake a version of XenServer Tools in addition to telling our
own and I certainly don't want to do that in the kernel. Reyk
has mentioned that this configuration be performed automatically
in the provisioning tool like cloud-agent.

> 
> > Index: xen.4
> > ===
> > RCS file: /home/cvs/src/share/man/man4/xen.4,v
> > retrieving revision 1.1
> > diff -u -p -r1.1 xen.4
> > --- xen.4   9 Dec 2015 00:26:39 -   1.1
> > +++ xen.4   21 Jul 2017 13:00:52 -
> > @@ -28,6 +28,51 @@ driver performs HVM domU guest initializ
> >  virtual Xen interrupts, access to the XenStore configuration storage as
> >  well as a device probing facility for paravirtualized devices such as
> >  disk and network interfaces.
> > +.Sh CAVEATS
> 
> I don't object to putting this into CAVEATS because XenServer does
> seem to be going out of its way in order to set up plenty of traps
> for the user.  On the other hand, putting it at the end of DESCRIPTION
> would seem even more logical to me, because it is just a description
> of how to use XenServer.  If DESCRIPTION seems too prominent to you
> for a blurb about one specific commercial product, EXAMPLES would
> also seem more logical to me than CAVEATS - anyway, your call.
> 
> If you decide to use CAVEATS, it has to go at the very end,
> after AUTHORS.  EXAMPLES would have to go between DESCRIPTION
> and SEE ALSO.
>

DESCRIPTION is fine.

> > +When running under XenServer, it's useful to let it know that the guest
> > +has finished initializing by setting a few XenStore properties with
> > +.Xr hostctl 8
> > +in the
> > +.Pa /etc/rc.local
> 
> Make that line
> 
>   .Xr rc.local 8
>

Done.

> > +Please note, that XenStore is capable of advertising a Hyper-V 
> > compatibility
> 
> No comma needed here.
> 
> > +layer called
> > +.Dq Viridian
> > +that may require an additional
> > +.Xr pvbus 4
> > +device node to be crated in
> > +.Pa /dev
> > +with
> > +.Xr MAKEDEV 8
> > +and all aforementioned invocations of
> > +.Xr hostctl 8
> > +to be amended with an
> > +.Fl f Ar /dev/pvbus1
> 
> Make that line
> 
>   .Fl f Pa /dev/pvbus1
> 
> > +command line option.
> > +Viridian can also be disabled in the virtual machine configuration.
> 
> Maybe
> 
>   Alternatively, Viridian can be disabled in the virtual machine
>   configuration.
> 
> because doing both ("also") does not seem to make much sense to me,
> but of course i'm not sure.
> 
> Yours,
>   Ingo

Thanks for suggestions, they're all good. In the meantime Theo
has suggested to create /dev/pvbus1 by default and I'm going to
send a diff for that in a moment. This means that I'm removing
this paragraph completely. I will however add some text to the
hostctl(8) man page to highlight the fact that multiple pvbus
devices may be present on a single VM.

OK?

Index: xen.4
===
RCS file: /home/cvs/src/share/man/man4/xen.4,v
retrieving revision 1.1
diff -u -p -r1.1 xen.4
--- xen.4   9 Dec 2015 00:26:39 -   1.1
+++ xen.4   21 Jul 2017 16:20:34 -
@@ -28,6 +28,31 @@ driver performs HVM domU guest initializ
 virtual Xen interrupts, access to the XenStore configuration storage as
 well as a device probing facility for paravirtualized devices such as
 disk and network interfaces.
+.Pp
+When running under XenServer, 

Re: Document hostctl commands for XenServer

[Erik van Westen]

Hi Ingo,

Op 21-7-2017 om 16:20 schreef Ingo Schwarze:
> Hi Mike,
>
> Mike Belopuhov wrote on Fri, Jul 21, 2017 at 03:14:08PM +0200:
>
>> Together with Maxim Khitrov we have figured out what needs to
>> be set for XenServer
> If XenServer were free software, i would say that the OpenBSD
> operating system should detect whether it is running under
> XenServer and then do all this automatically, by default, but ...
>
>> (that's a Citrix product based on Xen) to "recognize"
>> the OpenBSD VM and let it do things like reboot and so on.
>  ... that sounds like XenServer is a commercial product, so maybe
> we don't want to bloat OpenBSD with specific code targeting it,
> certainly not with large amounts of code.  If it can automatically
> be done in just a handful of lines of code, i don't feel strongly
> either way, even if XenServer is commercial.  I mean, other
> virtual machine hosts are completely commercial in the first
> place (unlike Xen) and we have large amounts of code in the kernel
> specifically targeting them (unless i misunderstand).

A quick check makes sure that XenServer is open source and not commercial.
http://xenserver.org.

[quote]


  ABOUT XENSERVER

XenServer is the leading open source virtualization platform, powered by
the Xen Project hypervisor
 and the XAPI
toolstack . It is used
in the world's largest clouds and enterprises.
 
Commercial support for XenServer
 is available from Citrix.
[/quote]

[snip]

Maybe that helps.

Regards,

Erik

Re: Document hostctl commands for XenServer

[Maxim Khitrov]

On Fri, Jul 21, 2017 at 9:14 AM, Mike Belopuhov  wrote:
> Hi,
>
> Together with Maxim Khitrov we have figured out what needs to
> be set for XenServer (that's a Citrix product based on Xen) to
> "recognize" the OpenBSD VM and let it do things like reboot and
> so on.
>
> I'd like to get this documented in the xen(4) man page instead
> of referring users to mailing list archives.
>
> There are two things that we can mention:
>
> 1) viridian capability, that XenServer comes with enabled by
>default, interferes with hostctl: you need to either disable
>it for your VM (if you have access) or MAKEDEV /dev/pvbus1
>and use that with hostctl(8).
>
> 2) to let XenServer management software know that OpenBSD is
>there in full glory we need to set a few XenStore properties
>with hostctl(8).  User needs to do this on every boot, so
>putting them somewhere around /etc/rc.local is necessary.
>
> I've come up with the diff below.  Please let me know if this
> makes sense and if we can improve it.
>
> Maxim, can you please double check the script itself.  Are all
> these values necessary?  I've changed a few things including
> the BuildVersion value.
>
> Thanks.

Hi Mike,

attr/PVAddons/* contains version information about XenServer Tools
installed in the VM, not the OS. Dinar copied the original values from
tools v6.2. I don't know whether XenServer actually cares about what
version is reported, but if it does, this would be tied to features
supported by xen, xbf, and xnf drivers. You typically update the tools
with each new XenServer release, which gives you most recent disk and
network drivers, at least on Windows.

Only attr/PVAddons/{MajorVersion,MinorVersion} and data/updated are
required to get graceful reboot and shutdown support in XenCenter, but
I would leave the rest in there to avoid confusing any tools that
might expect to find the other keys as well.
data/{os_name,os_uname,os_distro} provide OS information, which is
shown in VM properties (not required, but useful to have). Setting
data/updated to 1 triggers a refresh of this information in
XenStore/XenCenter.

-Max

Re: Document hostctl commands for XenServer

[Ingo Schwarze]

Hi Mike,

Mike Belopuhov wrote on Fri, Jul 21, 2017 at 03:14:08PM +0200:

> Together with Maxim Khitrov we have figured out what needs to
> be set for XenServer

If XenServer were free software, i would say that the OpenBSD
operating system should detect whether it is running under
XenServer and then do all this automatically, by default, but ...

> (that's a Citrix product based on Xen) to "recognize"
> the OpenBSD VM and let it do things like reboot and so on.

 ... that sounds like XenServer is a commercial product, so maybe
we don't want to bloat OpenBSD with specific code targeting it,
certainly not with large amounts of code.  If it can automatically
be done in just a handful of lines of code, i don't feel strongly
either way, even if XenServer is commercial.  I mean, other
virtual machine hosts are completely commercial in the first
place (unlike Xen) and we have large amounts of code in the kernel
specifically targeting them (unless i misunderstand).

> I'd like to get this documented in the xen(4) man page instead
> of referring users to mailing list archives.

Sure, if you decide to not do this automatically by default,
then documenting it in a manual page is a good idea, and xen(4)
seems like the logical place to me - FWTW, i know nothing about Xen,
pvbus(4), or hostctl(8).


> Index: xen.4
> ===
> RCS file: /home/cvs/src/share/man/man4/xen.4,v
> retrieving revision 1.1
> diff -u -p -r1.1 xen.4
> --- xen.4 9 Dec 2015 00:26:39 -   1.1
> +++ xen.4 21 Jul 2017 13:00:52 -
> @@ -28,6 +28,51 @@ driver performs HVM domU guest initializ
>  virtual Xen interrupts, access to the XenStore configuration storage as
>  well as a device probing facility for paravirtualized devices such as
>  disk and network interfaces.
> +.Sh CAVEATS

I don't object to putting this into CAVEATS because XenServer does
seem to be going out of its way in order to set up plenty of traps
for the user.  On the other hand, putting it at the end of DESCRIPTION
would seem even more logical to me, because it is just a description
of how to use XenServer.  If DESCRIPTION seems too prominent to you
for a blurb about one specific commercial product, EXAMPLES would
also seem more logical to me than CAVEATS - anyway, your call.

If you decide to use CAVEATS, it has to go at the very end,
after AUTHORS.  EXAMPLES would have to go between DESCRIPTION
and SEE ALSO.

> +When running under XenServer, it's useful to let it know that the guest
> +has finished initializing by setting a few XenStore properties with
> +.Xr hostctl 8
> +in the
> +.Pa /etc/rc.local

Make that line

  .Xr rc.local 8

> +script.
> +.Bd -literal -offset indent
> +ostype=$(sysctl -n kern.ostype)
> +osrelease=$(sysctl -n kern.osrelease)
> +osbuild=$(sysctl -n kern.osversion)
> +osvermaj=${osrelease%\.*}
> +osvermin=${osrelease#*\.}
> +
> +# PV driver version
> +hostctl attr/PVAddons/MajorVersion $osvermaj
> +hostctl attr/PVAddons/MinorVersion $osvermin
> +hostctl attr/PVAddons/MicroVersion 0
> +hostctl attr/PVAddons/BuildVersion $osbuild
> +hostctl attr/PVAddons/Installed 1
> +
> +# OS version
> +hostctl data/os_name "$ostype $osrelease"
> +hostctl data/os_uname $osrelease
> +hostctl data/os_distro $ostype
> +
> +# Update XenStore
> +hostctl data/updated 1
> +.Ed
> +.Pp
> +Please note, that XenStore is capable of advertising a Hyper-V compatibility

No comma needed here.

> +layer called
> +.Dq Viridian
> +that may require an additional
> +.Xr pvbus 4
> +device node to be crated in
> +.Pa /dev
> +with
> +.Xr MAKEDEV 8
> +and all aforementioned invocations of
> +.Xr hostctl 8
> +to be amended with an
> +.Fl f Ar /dev/pvbus1

Make that line

  .Fl f Pa /dev/pvbus1

> +command line option.
> +Viridian can also be disabled in the virtual machine configuration.

Maybe

  Alternatively, Viridian can be disabled in the virtual machine
  configuration.

because doing both ("also") does not seem to make much sense to me,
but of course i'm not sure.

Yours,
  Ingo