Re: Checking bhyve supported features (sysctls)

2018-08-17 Thread Rodney W. Grimes 
> From: owner-freebsd-virtualizat...@freebsd.org 
>  On Behalf Of Rodney W. Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude 
> Cc: Matt Churchyard ; 
> freebsd-virtualization@freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
> 
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
> >  wrote:
> > >> 
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >> 
> > >> > > Hello,
> > >> > > 
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> > 
> > >> > >Yes, it would be very good to remove that, as it usually tries 
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> > 
> > >> > 
> > >> > > 
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > > 
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > > 
> > >> > > Will these be available on both Intel & AMD processors as a way 
> > >> > > to determine if the module has loaded successfully and can run
> > >guests?
> > >> > > 
> > >> > > I also see the below sysctl related to iommu.
> > >> > > 
> > >> > > hw.vmm.iommu.initialized
> > >> > > 
> > >> > > Again, will this be set to 1 as soon as the module is loaded if 
> > >> > > iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to 
> > >> > > determine support on any processor.
> > >> > 
> > >> > >Probalby the safest way for a shell script to decide if bhyve is 
> > >> > >up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to process
> > >guests.
> > >> > 
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >> 
> > >> I'll investigate that, I was pretty sure that you should get this 
> > >> as soon as the vmm.ko module is finished initialzing, but you might 
> > >> be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >> 
> > >> > 
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> > 
> > >> > It would be nice if sysctls were better documented.
> > >> 
> > >> Agreed.
> > >> 
> > >> > If vmx.initialized is set once vmm has successfully loaded, I 
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not 
> > >Intel specific). This entry definitely exists and is set to 0 if you 
> > >load the module on a non-supported system, and set to 1 as soon as 
> > >vmm loads on my Intel test system.
> > >> 
> > >> Given its undocumented status you would be relying on an 
> > >> undocumented feature that could change in either name or behavior, 
> > >> and that is not desirable.
> > >> 
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do not 
> > >actually try to decide if vmm is supported or not, they simply 
> > >process the error from a vm_create() or vm_open() call and exit with 
> > >an error code if they can not handle it (some of the code can handle 
> > >a vm_create failure if infact we are trying to create a vm that 
> > >already exists).
> > >
> > >If you want to maintain full compatibility a similiar stratergy may 
> > >be in order.
> > >
> > >Why is it that vm-bhyve specifically needs to know if the kernel has 
> > >vmm support or not?
> > >Cant it just be written to handle the errors returned if the 
> > >supported functions do not exist?
> > 
> > I think the question vm-bhyve wants to answer is: does the CPU have 
> > the required features to run a multicore VM.
> 
> >Why does it need to know that?  If it tries to start a multicore/thread VM 
> >and the system can not support it an error is returned and the bhyve command 
> >fails.
> 
> >Actually determining that specific issue is non-trivial iirc as some vmm 
> >supported CPU's can only run a single VM with a single thread in that VM 
> >(early core cpu's).
> 
> > 
> > These or similar sysctls do seem to be the correct way to communicate 
> > that support.
> 
> >I do not believe any of those sysctl's communicate that your on a broken cpu 
> >or to what extent you can run vm's with multiple threads.
> 
> So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is not a 
> valid way to determine if the host has unrestricted guest support (required 
> for non-freebsd or multicore freebsd guests, and as you say missing from some 
> early VT-x capable processors)?

I overlooked that one, that could be used, and is probably fine to use,
though

RE: Checking bhyve supported features (sysctls)

2018-08-17 Thread Matt Churchyard 
-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Marcelo Araujo
Sent: 17 August 2018 10:14
To: Matt Churchyard 
Cc: freebsd-virtualization@freebsd.org
Subject: Re: Checking bhyve supported features (sysctls)

2018-08-17 16:54 GMT+08:00 Matt Churchyard :

>
>
>
>
> 2018-08-17 16:25 GMT+08:00 Matt Churchyard :
>
> -Original Message-
> From: owner-freebsd-virtualizat...@freebsd.org < 
> owner-freebsd-virtualizat...@freebsd.org> On Behalf Of Rodney W. 
> Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude 
> Cc: Matt Churchyard ; 
> freebsd-virtualization@ freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
>
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <
> freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> > >>
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >>
> > >> > > Hello,
> > >> > >
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> >
> > >> > >Yes, it would be very good to remove that, as it usually tries 
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> >
> > >> >
> > >> > >
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 
> > >> > > as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > >
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > >
> > >> > > Will these be available on both Intel & AMD processors as a 
> > >> > > way to determine if the module has loaded successfully and 
> > >> > > can run
> > >guests?
> > >> > >
> > >> > > I also see the below sysctl related to iommu.
> > >> > >
> > >> > > hw.vmm.iommu.initialized
> > >> > >
> > >> > > Again, will this be set to 1 as soon as the module is loaded 
> > >> > > if iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to 
> > >> > > determine support on any processor.
> > >> >
> > >> > >Probalby the safest way for a shell script to decide if bhyve 
> > >> > >is up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to 
> > >> > >process
> > >guests.
> > >> >
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >>
> > >> I'll investigate that, I was pretty sure that you should get this 
> > >> as soon as the vmm.ko module is finished initialzing, but you 
> > >> might be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >>
> > >> >
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> >
> > >> > It would be nice if sysctls were better documented.
> > >>
> > >> Agreed.
> > >>
> > >> > If vmx.initialized is set once vmm has successfully loaded, I 
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not 
> > >Intel specific). This entry definitely exists and is set to 0 if 
> > >you load the module on a non-supported system, and set to 1 as soon 
> > >as vmm loads on my Intel test system.
> > >>
> > >> Given its undocumented status you would be relying on an 
> > >> undocumented feature that could change in either name or 
> > >> behavior, and that is not desirable.
> > >>
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do 
> > >not actually try to decide if vmm is supported or not, they simply 
> > >process the error from a vm_create() or vm_open() call and exit 
> > >with an error code if they can not handle it (some of the code can 
> > >handle a vm_create failure if infact we are trying to create a vm 
> > >that already exists).
> > >
> > >If you want to maintain full compatibility a similiar stratergy may 
> > >be in order.
> > >
> > >Why is it that vm-bhyve specifically needs to know if the kernel 
> > >has vmm support or not?
> > >Cant it just be written to handle the errors returned if the 
> > >supported functions do not exist?
> >
> > I think the question vm-bhyve wants to answer is: does the CPU have 
> > the required features to run a multicore VM.
>
> >Why does it need to know that?  If it tries to start a 
> >multicore/thread
> VM and the system can not support it an error is returned and the 
> bhyve command fails.
>
> >Actually determining that specific issue is non-trivial iirc as some 
> >vmm
> supported CPU's can only run a single VM with a single thread in that 
> VM (early core cpu's).
>
> >
> > These or similar sysctls do seem to be the correct way to 
> > communicate that support.
>
> >I do not believe any of

Re: Checking bhyve supported features (sysctls)

2018-08-17 Thread Marcelo Araujo 
2018-08-17 16:54 GMT+08:00 Matt Churchyard :

>
>
>
>
> 2018-08-17 16:25 GMT+08:00 Matt Churchyard :
>
> -Original Message-
> From: owner-freebsd-virtualizat...@freebsd.org <
> owner-freebsd-virtualizat...@freebsd.org> On Behalf Of Rodney W. Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude 
> Cc: Matt Churchyard ; freebsd-virtualization@
> freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
>
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <
> freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> > >>
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >>
> > >> > > Hello,
> > >> > >
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> >
> > >> > >Yes, it would be very good to remove that, as it usually tries
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> >
> > >> >
> > >> > >
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > >
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > >
> > >> > > Will these be available on both Intel & AMD processors as a way
> > >> > > to determine if the module has loaded successfully and can run
> > >guests?
> > >> > >
> > >> > > I also see the below sysctl related to iommu.
> > >> > >
> > >> > > hw.vmm.iommu.initialized
> > >> > >
> > >> > > Again, will this be set to 1 as soon as the module is loaded if
> > >> > > iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to
> > >> > > determine support on any processor.
> > >> >
> > >> > >Probalby the safest way for a shell script to decide if bhyve is
> > >> > >up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to process
> > >guests.
> > >> >
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >>
> > >> I'll investigate that, I was pretty sure that you should get this
> > >> as soon as the vmm.ko module is finished initialzing, but you might
> > >> be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >>
> > >> >
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> >
> > >> > It would be nice if sysctls were better documented.
> > >>
> > >> Agreed.
> > >>
> > >> > If vmx.initialized is set once vmm has successfully loaded, I
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not
> > >Intel specific). This entry definitely exists and is set to 0 if you
> > >load the module on a non-supported system, and set to 1 as soon as
> > >vmm loads on my Intel test system.
> > >>
> > >> Given its undocumented status you would be relying on an
> > >> undocumented feature that could change in either name or behavior,
> > >> and that is not desirable.
> > >>
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do not
> > >actually try to decide if vmm is supported or not, they simply
> > >process the error from a vm_create() or vm_open() call and exit with
> > >an error code if they can not handle it (some of the code can handle
> > >a vm_create failure if infact we are trying to create a vm that
> > >already exists).
> > >
> > >If you want to maintain full compatibility a similiar stratergy may
> > >be in order.
> > >
> > >Why is it that vm-bhyve specifically needs to know if the kernel has
> > >vmm support or not?
> > >Cant it just be written to handle the errors returned if the
> > >supported functions do not exist?
> >
> > I think the question vm-bhyve wants to answer is: does the CPU have
> > the required features to run a multicore VM.
>
> >Why does it need to know that?  If it tries to start a multicore/thread
> VM and the system can not support it an error is returned and the bhyve
> command fails.
>
> >Actually determining that specific issue is non-trivial iirc as some vmm
> supported CPU's can only run a single VM with a single thread in that VM
> (early core cpu's).
>
> >
> > These or similar sysctls do seem to be the correct way to communicate
> > that support.
>
> >I do not believe any of those sysctl's communicate that your on a broken
> cpu or to what extent you can run vm's with multiple threads.
>
> So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is
> not a valid way to determine if the host has unrestricted guest support
> (required for non-freebsd or multicore freebsd guests, and as

RE: Checking bhyve supported features (sysctls)

2018-08-17 Thread Matt Churchyard 


2018-08-17 16:25 GMT+08:00 Matt Churchyard 
mailto:matt.churchy...@userve.net>>:
-Original Message-
From: 
owner-freebsd-virtualizat...@freebsd.org
 
mailto:owner-freebsd-virtualizat...@freebsd.org>>
 On Behalf Of Rodney W. Grimes
Sent: 16 August 2018 18:31
To: Allan Jude mailto:allanj...@freebsd.org>>
Cc: Matt Churchyard 
mailto:matt.churchy...@userve.net>>; 
freebsd-virtualization@freebsd.org
Subject: Re: Checking bhyve supported features (sysctls)

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
> mailto:freebsd-...@pdx.rh.cn85.dnsmgr.net>>
>  wrote:
> >>
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >>
> >> > > Hello,
> >> > >
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> >
> >> > >Yes, it would be very good to remove that, as it usually tries
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> >
> >> >
> >> > >
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon
> >> > > as the vmm module is loaded
> >> > >
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > >
> >> > > Will these be available on both Intel & AMD processors as a way
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > >
> >> > > I also see the below sysctl related to iommu.
> >> > >
> >> > > hw.vmm.iommu.initialized
> >> > >
> >> > > Again, will this be set to 1 as soon as the module is loaded if
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to
> >> > > determine support on any processor.
> >> >
> >> > >Probalby the safest way for a shell script to decide if bhyve is
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> >
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >>
> >> I'll investigate that, I was pretty sure that you should get this
> >> as soon as the vmm.ko module is finished initialzing, but you might
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm is created.
> >>
> >> >
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> >
> >> > It would be nice if sysctls were better documented.
> >>
> >> Agreed.
> >>
> >> > If vmx.initialized is set once vmm has successfully loaded, I
> >> > can't
> >see a better way of checking for bhyve support (assuming it's not
> >Intel specific). This entry definitely exists and is set to 0 if you
> >load the module on a non-supported system, and set to 1 as soon as
> >vmm loads on my Intel test system.
> >>
> >> Given its undocumented status you would be relying on an
> >> undocumented feature that could change in either name or behavior,
> >> and that is not desirable.
> >>
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and byhve.  They do not
> >actually try to decide if vmm is supported or not, they simply
> >process the error from a vm_create() or vm_open() call and exit with
> >an error code if they can not handle it (some of the code can handle
> >a vm_create failure if infact we are trying to create a vm that
> >already exists).
> >
> >If you want to maintain full compatibility a similiar stratergy may
> >be in order.
> >
> >Why is it that vm-bhyve specifically needs to know if the kernel has
> >vmm support or not?
> >Cant it just be written to handle the errors returned if the
> >supported functions do not exist?
>
> I think the question vm-bhyve wants to answer is: does the CPU have
> the required features to run a multicore VM.

>Why does it need to know that?  If it tries to start a multicore/thread VM and 
>the system can not support it an error is returned and the bhyve command fails.

>Actually determining that specific issue is non-trivial iirc as some vmm 
>supported CPU's can only run a single VM with a single thread in that VM 
>(early core cpu's).

>
> These or similar sysctls do seem to be the correct way to communicate
> that support.

>I do not believe any of those sysctl's communicate that your on a broken cpu 
>or to what extent you can run vm's with multiple threads.
So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is not a 
valid way to determine if the host has unrestricted guest support (required for 
non-freebsd or multicore freebsd guests, and as you say missing from some early 
VT-x capable processors)?

>I

Re: Checking bhyve supported features (sysctls)

2018-08-17 Thread Marcelo Araujo 
2018-08-17 16:25 GMT+08:00 Matt Churchyard :

> -Original Message-
> From: owner-freebsd-virtualizat...@freebsd.org <
> owner-freebsd-virtualizat...@freebsd.org> On Behalf Of Rodney W. Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude 
> Cc: Matt Churchyard ; freebsd-virtualization@
> freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
>
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <
> freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> > >>
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >>
> > >> > > Hello,
> > >> > >
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> >
> > >> > >Yes, it would be very good to remove that, as it usually tries
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> >
> > >> >
> > >> > >
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > >
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > >
> > >> > > Will these be available on both Intel & AMD processors as a way
> > >> > > to determine if the module has loaded successfully and can run
> > >guests?
> > >> > >
> > >> > > I also see the below sysctl related to iommu.
> > >> > >
> > >> > > hw.vmm.iommu.initialized
> > >> > >
> > >> > > Again, will this be set to 1 as soon as the module is loaded if
> > >> > > iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to
> > >> > > determine support on any processor.
> > >> >
> > >> > >Probalby the safest way for a shell script to decide if bhyve is
> > >> > >up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to process
> > >guests.
> > >> >
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >>
> > >> I'll investigate that, I was pretty sure that you should get this
> > >> as soon as the vmm.ko module is finished initialzing, but you might
> > >> be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >>
> > >> >
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> >
> > >> > It would be nice if sysctls were better documented.
> > >>
> > >> Agreed.
> > >>
> > >> > If vmx.initialized is set once vmm has successfully loaded, I
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not
> > >Intel specific). This entry definitely exists and is set to 0 if you
> > >load the module on a non-supported system, and set to 1 as soon as
> > >vmm loads on my Intel test system.
> > >>
> > >> Given its undocumented status you would be relying on an
> > >> undocumented feature that could change in either name or behavior,
> > >> and that is not desirable.
> > >>
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do not
> > >actually try to decide if vmm is supported or not, they simply
> > >process the error from a vm_create() or vm_open() call and exit with
> > >an error code if they can not handle it (some of the code can handle
> > >a vm_create failure if infact we are trying to create a vm that
> > >already exists).
> > >
> > >If you want to maintain full compatibility a similiar stratergy may
> > >be in order.
> > >
> > >Why is it that vm-bhyve specifically needs to know if the kernel has
> > >vmm support or not?
> > >Cant it just be written to handle the errors returned if the
> > >supported functions do not exist?
> >
> > I think the question vm-bhyve wants to answer is: does the CPU have
> > the required features to run a multicore VM.
>
> >Why does it need to know that?  If it tries to start a multicore/thread
> VM and the system can not support it an error is returned and the bhyve
> command fails.
>
> >Actually determining that specific issue is non-trivial iirc as some vmm
> supported CPU's can only run a single VM with a single thread in that VM
> (early core cpu's).
>
> >
> > These or similar sysctls do seem to be the correct way to communicate
> > that support.
>
> >I do not believe any of those sysctl's communicate that your on a broken
> cpu or to what extent you can run vm's with multiple threads.
>
> So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is
> not a valid way to determine if the host has unrestricted guest support
> (required for non-freebsd or multicore freebsd guests, and as you say
> missing from some early VT-x capable

RE: Checking bhyve supported features (sysctls)

2018-08-17 Thread Matt Churchyard 
-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Rodney W. Grimes
Sent: 16 August 2018 18:31
To: Allan Jude 
Cc: Matt Churchyard ; 
freebsd-virtualization@freebsd.org
Subject: Re: Checking bhyve supported features (sysctls)

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
>  wrote:
> >> 
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >> 
> >> > > Hello,
> >> > > 
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> > 
> >> > >Yes, it would be very good to remove that, as it usually tries 
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> > 
> >> > 
> >> > > 
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon
> >> > > as the vmm module is loaded
> >> > > 
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > > 
> >> > > Will these be available on both Intel & AMD processors as a way 
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > > 
> >> > > I also see the below sysctl related to iommu.
> >> > > 
> >> > > hw.vmm.iommu.initialized
> >> > > 
> >> > > Again, will this be set to 1 as soon as the module is loaded if 
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to 
> >> > > determine support on any processor.
> >> > 
> >> > >Probalby the safest way for a shell script to decide if bhyve is 
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> > 
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >> 
> >> I'll investigate that, I was pretty sure that you should get this 
> >> as soon as the vmm.ko module is finished initialzing, but you might 
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm is created.
> >> 
> >> > 
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> > 
> >> > It would be nice if sysctls were better documented.
> >> 
> >> Agreed.
> >> 
> >> > If vmx.initialized is set once vmm has successfully loaded, I 
> >> > can't
> >see a better way of checking for bhyve support (assuming it's not 
> >Intel specific). This entry definitely exists and is set to 0 if you 
> >load the module on a non-supported system, and set to 1 as soon as 
> >vmm loads on my Intel test system.
> >> 
> >> Given its undocumented status you would be relying on an 
> >> undocumented feature that could change in either name or behavior, 
> >> and that is not desirable.
> >> 
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and byhve.  They do not 
> >actually try to decide if vmm is supported or not, they simply 
> >process the error from a vm_create() or vm_open() call and exit with 
> >an error code if they can not handle it (some of the code can handle 
> >a vm_create failure if infact we are trying to create a vm that 
> >already exists).
> >
> >If you want to maintain full compatibility a similiar stratergy may 
> >be in order.
> >
> >Why is it that vm-bhyve specifically needs to know if the kernel has 
> >vmm support or not?
> >Cant it just be written to handle the errors returned if the 
> >supported functions do not exist?
> 
> I think the question vm-bhyve wants to answer is: does the CPU have 
> the required features to run a multicore VM.

>Why does it need to know that?  If it tries to start a multicore/thread VM and 
>the system can not support it an error is returned and the bhyve command fails.

>Actually determining that specific issue is non-trivial iirc as some vmm 
>supported CPU's can only run a single VM with a single thread in that VM 
>(early core cpu's).

> 
> These or similar sysctls do seem to be the correct way to communicate 
> that support.

>I do not believe any of those sysctl's communicate that your on a broken cpu 
>or to what extent you can run vm's with multiple threads.

So cap.unrestricted_guest from the vmm "capabilities" set of sysctls is not a 
valid way to determine if the host has unrestricted guest support (required for 
non-freebsd or multicore freebsd guests, and as you say missing from some early 
VT-x capable processors)?

>I went and looked at why vm-bhyve is groveling around in /var/run/dmesg.boot 
>and found that it is simply trying to determine if the host CPU is vmm capable,
>specifically:
>util::check_bhyve_support(){
>...

>These checks are already built into the kernel.
>This can all go in the bit