Re: FreeBSD 13 on bhyve freezes under load

[Michael Gmelin]

On Mon, 3 May 2021 13:14:53 +0200
Christian Stærk  wrote:

> Hi Michael,
> 
> I am not using ZFS and have plenty of free diskspace:
> Filesystem Size    Used   Avail Capacity Mounted on
> /dev/ada0p2    190G 92G 83G    52%    /
> 
> top looks like this on a not-hung system:
> 
> last pid: 38292;  load averages:  0.91, 0.87,  0.88    up 0+01:13:16  
> 13:12:12
> 32 processes:  2 running, 29 sleeping, 1 zombie
> CPU:  5.5% user,  0.0% nice,  7.8% system,  7.0% interrupt, 79.7% idle
> Mem: 59M Active, 3549M Inact, 1090M Wired, 610M Buf, 2250M Free
> Swap: 24G Total, 24G Free
> 
>    PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU
> COMMAND 38079 root  1  22    0    99M    43M RUN  0:03
> 14.02% pkg-static

Based on hanging pkg-static process I assume that top output is from
within the VM though, isn't it? Or is this on the host?

Best
Michael

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 13 on bhyve freezes under load

[Michael Gmelin]

On Mon, 3 May 2021 11:14:45 +0200
Christian Stærk  wrote:

> Hello,
> 
> I recently upgraded one of my bhyve guests to FreeBSD 13.0-RELEASE.
> The host is running 12.2-RELEASE-p4.
> 
> The guest in question is used for building packages with poudriere
> and is always the first one to get upgraded to new releases.
> 
> The guest freezes consistently every time poudriere tries to build
> the thunderbird package. Although it is not a complete freeze. It
> responds to ping and programs like top and systat continue updating.
> However, all I/O operations seem to hang.
> 
> systat looks like this when it happens: 
> https://borderworlds.dk/~xi/FreeBSD%2013%20frozen.png 
> <https://borderworlds.dk/~xi/FreeBSD%2013%20frozen.png>
> 
> What is the best way to debug this?

Sharing top output might help (memory/swap and ZFS arc usage). Also,
make sure you're not low on diskspace (these are the two most common
issues I encountered with hanging bhyve VMs - out of non-virtual memory
and out of diskspace of the host machine filesystem).

Some basic info on the host and your VM configuration might also help
others to reproduce the problem.

-m

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve with wlan0

[Michael Gmelin]

> On 28. Nov 2020, at 19:30, Rodney W. Grimes  
> wrote:
> 
> 
>> 
>> As far as I remember it works just fine other than the standard caveats 
>> about bridging over wireless interfaces which involve some odd 
>> inconsistencies that could crop up from time to time especially with like 
>> DHCP, but a lot of that depends on your network config as a whole. 
>> The approach I would consider taking for a wireless host is if the guest 
>> doesn't need direct access to the network just do a NAT network or a routed 
>> network and route through the wlan interface instead of trying to do 
>> bridging with it directly.
> 
> One very problematic area for wireless and bhyve is that each guests have a 
> different MAC, wireless does NOT like that, as the AP usually associates with 
> one and only one MAC address.
> 

Like Jason wrote, it‘s best to use NAT on wireless (especially if you’re moving 
between networks frequently). Using something like vm-bhyve, it’s very easy to 
set up.

-m


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve help

[Michael Gmelin]

You might want to read https://blog.grem.de/ayvn

It also covers using vm-bhyve to manage vms and will explain some things about 
configuring the network.

> On 6. May 2020, at 01:55, Brandon helsley  wrote:
> 
> Well how do i add a tap to the bridge the VM is on as well as my physical nic
> 
> Sent from Outlook Mobile
> 
> 
> From: Aryeh Friedman 
> Sent: Tuesday, May 5, 2020 5:02:50 PM
> To: Brandon helsley 
> Cc: freebsd-virtualization@freebsd.org ; 
> Robert Crowston 
> Subject: Re: Bhyve help
> 
> When setting up a VM you also need to add the tap it is on to the bridge as 
> well your physical NIC
> 
> On Tue, May 5, 2020 at 6:28 PM Brandon helsley 
> mailto:brandon.hels...@hotmail.com>> wrote:
> Bhyve gets me to Network installation: "some installation files were not 
> found on the boot volume. The next few screens will allow you to configure 
> networking so that they can be downloaded from the internet." This happens 
> after system components.
> 
> Sent from Outlook Mobile
> 
> From: Aryeh Friedman 
> mailto:aryeh.fried...@gmail.com>>
> Sent: Tuesday, May 5, 2020 4:08:10 AM
> To: Brandon helsley 
> mailto:brandon.hels...@hotmail.com>>
> Cc: 
> freebsd-virtualization@freebsd.org 
> mailto:freebsd-virtualization@freebsd.org>>;
>  Robert Crowston mailto:crows...@protonmail.com>>
> Subject: Re: Bhyve help
> 
> 
> 
> On Tue, May 5, 2020 at 6:00 AM Brandon helsley 
> mailto:brandon.hels...@hotmail.com>> wrote:
> The handbook only says vmm doesn't it. I got bhyve to start without virtio 
> kernel module as well as the if_tap but with vmm modules.
> 
> The handbook section on virtualization (FreeBSD host) is at best awful and 
> misleading your best guide is the bhyve man page (especially the examples 
> there) here is the modules I load for the same set up I recommended you try 
> as a advanced beginners lesson in virtualization and networking.   And an 
> example script for using them:
> 
> /boot/loader.conf:
> 
> kern.geom.label.disk_ident.enable="0"
> kern.geom.label.gptid.enable="0"
> opensolaris_load="YES"
> zfs_load="YES"
> ipfw_load="YES"
> ipdivert_load="YES"
> linux_load="YES"
> 
> In the real /boot/loader.conf the following are commented out because 
> PetiteCloud takes care of them when it loads but unless you want to get into 
> more advanced stuff (like how to install non-standard ports) you should 
> uncomment them.
> 
> vmm_load="YES"
> if_tap_load="YES"
> nmdm_load="YES"
> 
> The script to load the file server VM:
> 
> #!/bin/sh
> #
> # Generated by PetiteCloud 0.2.6
> #
> 
> ifconfig tap12 destroy
> ifconfig tap12 create
> ifconfig tap12 up
> sleep 5
> ifconfig bridge0 addm tap12 up
> bhyvectl --destroy --vm=b829e56e9b4289a1
> /usr/sbin/bhyve -c cores=4 -m 4096 -AI -H -P -w -g 0 -s 0:0,hostbridge -s 
> 1,virtio-net,tap12 -s 2,ahci-hd,/vms/import/win10  -s 
> 29,fbuf,tcp=0.0.0.0:5912,w=1920,h=1080 -s 30,xhci,tablet 
>  -s 31,lpc -l com1,/dev/nmdm12B -l 
> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd 
> b829e56e9b4289a1&>/dev/null
> echo $!>/var/run/petitecloud/b829e56e9b4289a1
> sleep 10
> 
> I'm at this screen:please choose the appropriate terminal type for your 
> system.common console types are: ansi, vt100, xterm, cons25w.
> 
> Console type [vt100]:
> 
> Sent from Outlook Mobile
> 
> From: Aryeh Friedman 
> mailto:aryeh.fried...@gmail.com>>
> Sent: Tuesday, May 5, 2020 3:52:26 AM
> To: Brandon helsley 
> mailto:brandon.hels...@hotmail.com>>
> Cc: 
> freebsd-virtualization@freebsd.org 
> mailto:freebsd-virtualization@freebsd.org>>;
>  Robert Crowston mailto:crows...@protonmail.com>>
> Subject: Re: Bhyve help
> 
> 
> 
> On Tue, May 5, 2020 at 5:21 AM Brandon helsley 
> mailto:brandon.hels...@hotmail.com>> wrote:
> I have re0 but the ip address is under wlan0 with my CenturyLink
> 
> In order to use vtnet and bridging (both required for bhyve networking) you 
> need to load the load the virtio kernel module as well as the if_tap and vmm 
> modules (the first one provides the host side of the virtual NIC and the 
> second loads the hypervisor components)
> 
> Sent from Outlook Mobile
> 
> 
> From: Robert Crowston 
> mailto:crows...@protonmail.com>>
> Sent: Tuesday, May 5, 2020 3:14:43 AM
> To: Brandon helsley 
> mailto:brandon.hels...@hotmail.com>>; 
> freebsd-virtualization@freebsd.org 
> mailto:freebsd-virtualization@freebsd.org>>
> Subject: Re: Bhyve help
> 
> What are your network interfaces called on your machine? You probably have to 
> change igb0 in the example to em0 (an Intel card) or re0 (a Realtek card). 
> Type “ifconfig” to list all the interfaces, and look for ones with an ip 
> address to figure it o

Re: syncing bhyve instances

[Michael Gmelin]

> On 22. Jan 2018, at 20:47, tech-lists  wrote:
> 
>> On 22/01/2018 16:38, Paul Vixie wrote:
>> for live sync you'll have to run software inside the guest that knows
>> how to properly freeze state. for example if there's a live database of
>> any kind you'll want it to be in its quiet state before you sync from
>> it. in those situations, i do use rsync.
> 
> Yeah, thought it might be this. Sorry I wasn't more clear initially
> about the use case.
> 
> Basically, the production server is in a datacentre and the reserve
> server is on a very fast vdsl service. The reason for the reserve server
> is, if the production server fails then I swap DNS to point at the
> reserved server and the guests on it without interruption of service.
> All guests are running databases (mysql) though they aren't especially
> busy. So I guess the best bet would be mysql replication for the
> databases and rsync for everything except mysql?
> 
> thanks everyone who took the time to answer

For the database, mysql replication is the way to go (otherwise you would need 
to stop/lock the server every time you do a snapshot and you'll get 
near-realtime replication which will always be more current than your 
snapshots). For  the system I'd suggest to create automation for the setup, so 
you can apply changes to both systems without a need to sync/clone/copy from 
one to the other. For anything else left (data files on disk, non-system, 
non-package, non-config) use regular rsync (or zfs send if your setup permits - 
I'd stick with rsync).

Yours,
Michael

p.s. Make sure to include monitoring (especially replication latency and data 
integrity)


> 
> -- 
> J.
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: iocage no longer supported?

[Michael Gmelin]

On Tue, 12 Apr 2016 11:44:14 -0400
Larry Baird  wrote:

> I was beginning to look into a replacement for ezjail, mainly because
> ezjail doesn't use jail.conf. After vistiting lots of sites and doing
> some reading, I had about decided upon iocage. I am now convered
> about this decision. When I visit https://github.com/iocage/iocage is
> now says "**No longer supported. iocage is being rewritten in a
> differnt language." Does anybody have any details?
> 

https://github.com/iocage/iocage/commit/3f394561a3dde55cd3ac7911be313c5df5865183


-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: can a bhyve instance be resized? adding another virtual disk?

[Michael Gmelin]

On Wed, 15 Oct 2014 12:56:38 +0100
freebsd-li...@potato.growveg.org wrote:

> Hello list,
> 
> Can a bhyve instance be resized? I'm talking about the disk. 
> Say your end user needs more diskspace. They have 32GB. They need
> 64GB. How do you do it? I presume one has to stop the guest, then use
> truncate. What about if the guest OS isn't freebsd, and they use say
> ext2 or 3? Will ext3 start yelling at me because I've resized it?
> 
> What if they just want another disk? How does one refer to a 
> newly created virtual disk from a guest? How is it mounted to the
> guest?
> 
> thanks

If you're using a zvol with UFS on top the procedure is quite simple,
this is how I did it last time (at your own risk), I resized the disk
to 15G (was like 5G beforehand).

zvol is at tank/bhyve/virt03.

Shutdown the VM.

zfs set volsize=15G tank/bhyve/virt03
zfs rename tank/bhyve/virt03  tank/bhyve/virtxx
gpart recover zvol/tank/bhyve/virtxx
zfs rename tank/bhyve/virtxx  tank/bhyve/virt03

# this is not the exact size, sorry
gpart resize -i1 -s15359M zvol/tank/bhyve/virt03
growfs /dev/zvol/tank/bhyve/virt03p1

Start VM

The renaming step was necessary so that GEOM picks up the new zvol size
(this was on 10.0-RELEASE, might work without this on CURRENT). You
might want to make a backup before doing this.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: High load average inside idle FreeBSD guest running on bhyve

[Michael Gmelin]

On Thu, 02 Oct 2014 11:49:19 +0100
Karl Pielorz  wrote:

> 
> 
> --On 02 October 2014 10:06 +0200 Christian Laursen
>  wrote:
> 
> > Hello
> >
> > I have a few bhyve instances running that all seem to have a higher
> > load average than what would seem normal for a system that is idle.
> >
> > [snip]
> >
> > last pid:  6109;  load averages:  0.41,  0.38,  0.39up
> > 1+14:27:48 09:56:25
> 
> Hi,
> 
> I think this has been covered on other lists - and is apparently
> "normal" (i.e. cosmetic).
> 
> I've installed a number of FBSD10 boxes recently, and they all seem
> to idle at around that (even when doing absolutely nothing) - I did
> post about it and was told it was just cosmetic... It certainly seems
> to be cosmetic.
> 
> If it's the same issue - I'd guess it's not bhyve related...
> 
> "
> last pid:  1294;  load averages:  0.42,  0.32,  0.27 
> up 1+20:44:55  11:47:16
> 18 processes:  1 running, 17 sleeping
> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.1% interrupt, 99.9% idle
> "
> 
> That's from a bare metal amd64 10.0-RELEASE-p9 box with no services /
> users on it.
> 
> Thinking about it - we had the same 'symptom' on a Xen based FBSD 10
> box as well...

For your bare metal box, could you send the output of vmstat -i ? I had
the same issue on older boxes and it went away after setting

hint.hpet.0.legacy_route=1
hint.attimer.0.clock=0
hint.atrtc.0.clock=0

in /boot/loader.conf (see hpet(4)).

> 
> -Karl
> 
> ___
> freebsd-virtualization@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscr...@freebsd.org"



-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Call for Submissions: 1st Creative use of Cloud Computing Contest

[Michael Gmelin]

On Tue, 4 Mar 2014 10:35:26 -0500
Aryeh Friedman  wrote:

> http://www.petitecloud.org/contest.jsp
> 

How did this end up in my inbox?

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Current problem reports assigned to freebsd-virtualization@FreeBSD.org

[Michael Gmelin]

On Mon, 3 Feb 2014 19:32:21 +0100
Mark Martinec  wrote:

> Aryeh Friedman wrote:
> > By now we have made enough progress in runninng FB as a VM on FB ...
> 
> Please avoid shortening FreeBSD as FB.  It is confusing for a casual
> reader as FB is commonly associated with Facebook, and it is
> unfriendly to search engines.  If FreeBSD is too much to type, use
> FBSD.
> 

I restrained myself from asking the same, but I agree 100% with Mark,
FB == Facebook, FreeBSD == FreeBSD.

Also, please don't top post (I know it's hard these days, especially
with gmail, but it's FreeBSD etiquette for a reason, see
http://www.freebsd.org/doc/en/articles/mailing-list-faq/etiquette.html
).

Thank you.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Public Service Announcement: The name is "bhyve"

[Michael Gmelin]

On Fri, 24 Jan 2014 02:46:06 -0800
Michael Dexter  wrote:

> On 1/24/14 2:41 AM, Michael Gmelin wrote:
> >> Haters gonna hate?
> >>
> > Lack of attention to detail != hate.
> 
> I guess you didn't see the phoronix.com article on bhyve a few months
> ago. :) They pulled the worst of it.
> 
> The whole "BSD is dying" or "is poo" in their case FUD is still very
> active in various circles and you never know.
> 

Oh I've seen it and I usually ignore Phoronix and its comments for
exactly this reason, but in this case serverwatch reported on the
topic in a factual/neutral way and just got the details wrong. It
doesn't seem like a high margin website to me, so they probably
didn't spend much time on it ;)

Recent bhyve progress is quite impressive by the way.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Public Service Announcement: The name is "bhyve"

[Michael Gmelin]

On Fri, 24 Jan 2014 02:18:03 -0800
Michael Dexter  wrote:

> 
> Hello all,
> 
> It was "BHyVe" and Neel and Peter mercifully simplified it to "bhyve".
> 
> It was never "BHyve" and how they came up with "byhyve" and "byhve" in
> the same article I do not know:
> 
> http://www.serverwatch.com/server-news/open-source-freebsd-10-takes-on-virtualization.html
> 
> Haters gonna hate?
> 

Lack of attention to detail != hate.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: reboot stops vm

[Michael Gmelin]

On Tue, 21 Jan 2014 03:31:40 +0400
Sergey Matveychuk  wrote:

> Hi.
> 
> Is it expected behaviour?
> 
> sem@vm0[17]% sudo reboot
> Jan 21 07:27:51 vm0 reboot: rebooted by sem
> Jan 21 07:27:51 vm0 syslogd: exiting on signal 15
> Waiting (max 60 seconds) for system process `vnlru' to stop...done
> Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
> Waiting (max 60 seconds) for system process `syncer' to stop...
> Syncing disks, vnodes remaining...2 0 done
> All buffers synced.
> Uptime: 1m17s
> Rebooting...
> cpu_reset: Stopping other CPUs
> root@bhyve[36]#

Yes

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Real Device with BHyve

[Michael Gmelin]

On Fri, 03 Jan 2014 09:43:24 -0800
Peter Grehan  wrote:

> Hi Andrea,
> > Well, the machine starts ok but when the "child" FreeBSD starts
> > installation something strange happens. When I get to the
> > partitioning screen I can see the device avaiable as /dev/vtdb0
> > with the correct size and such. I choose autopartitioning, the
> > installer writes the partition table but when it start to
> > write /dev/vtdb0p2 a very cryptic error appears about being unable
> > to write - sorry, did not write it down.
> >
> > The installer then stops.
> >
> > If I do a fdisk /dev/vtdb0 in the VM I can see the GPT partition
> > being there. If I do a fdisk /dev/da2 on the host machine, I can
> > see the GPT partition as well, but the VM just doesn't want to
> > write on it.
> >
> > I even tried changing kern.geom.debugflags=16 as I thought the host
> > machine could be locking somehow the drive, but that didn't seem to
> > make any difference. I know it was a lame check but I was out of
> > ideas.
> >
> > So I just wanted to understand if such a scenario is supposed to be
> > supported
> 
>   It is. Been a while since I've done this but will try a repro.
> Other folk have supported success using zvols so I'd assumed it was
> working.

If it helps, I had the same problem using zvol as well (booting the
installer and install on the "raw" device) on 10-CURRENT from mid 2013.

I didn't try with a more recent release. I ended up installing FreeBSD
from the outside (create partitions and untar base.txz, kernel.txz etc.
- afaik the same thing the Dexter's vm0 script does now) and could then
boot into it just fine.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Upcoming FreeBSD 10.x + bhyve ...

[Michael Gmelin]

On Mon, 2 Dec 2013 16:11:53 -0800
Marc Fournier  wrote:

> 
> On Dec 2, 2013, at 16:08 , Peter Grehan  wrote:
> 
> > Hi Marc,
> > 
> >> I just read through http://bhyve.org/bhyve-manual.txt, and one
> >> thing that doesn’t seem to be supported (or, I’ve missed it) is
> >> HeadLess support … I get the impression that using this on a
> >> remote server isn’t currently possible, or am I missing something
> >> in the docs?
> > 
> > It is. The easiest is using the recent change (r258668) which
> > allows bhyveload and bhyve to direct output to a tty instead of
> > just stdio.
> 
> ‘k, that would work … so *if* I’m reading the docs right, the intiial
> build of the file system / install of guest OS, happens on an
> interactive remote ssh session, then I’d shutdown and restart it
> redirecting output … ?
> 
> I’m plannin gon building a 10.x system over Christmas to start
> playing with this, just trying to do as much research as possible
> before … 
> 
> thx
> 
> 

Even though it's not the official solution, my little bhyve-rc script
works reliably (requires sysutils/tmux). You can basically "tmux attach"
into your session anytime. It also provides bhyve_guest for a clean
shutdown of FreeBSD guests (a workaround until bhyve can inject the
necessary acpi instructions to trigger a shutdown). I've been using it
for half a year now and it gets the job done. You can find it here
http://druidbsd.cvs.sourceforge.net/viewvc/druidbsd/bhyve_rc/

Please share your experience with the changes in r258668,
quite interested if this works ok for you.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 9.1-Rel: Jail-rc startup of vnet jails with new configuration syntax

[Michael Gmelin]

On Thu, 7 Nov 2013 13:44:32 +0100
Michael Gmelin  wrote:

> 
> 
> On Thu, 07 Nov 2013 13:26:10 +0100
> Kristen Nielsen  wrote:
> 
> > Hi freebsd-virtualization list
> > 
> > I am trying to figure out how to start configured jails within
> > FreeBSD 9.1Release at boot time.
> > I am using new C-style jail configuration syntax in /etc/jail.conf.
> > all jail starts successfully when i do a jail -c from the
> > commandline.
> > 
> > I am trying to configure jails to start at boot time using the
> > features in the /etc/rc.conf file.
> > 
> > in /etc/rc.conf at base host i have
> > jail_enable="YES"
> > jail_list=" jail1 jail2 jail3 "
> > 
> > and no more jail related lines, evereything else is placed into the
> > jail configuration (C-style) file.
> > 
> > running "service jail start":
> > Configuring jails:.
> > Starting jails:/etc/rc.d/jail: ERROR: jail: No hostname has been
> > defined for jail1
> > 
> > Looking into the /etc/rc.d/jail script it seems it expects that the 
> > /etc/rc.conf contains old style jail configuration statement like
> > 
> > jail__hostname="jail1"
> > etc.
> > 
> > I have looked at the patch at: 
> > http://wiki.polymorf.fr/index.php/Howto:FreeBSD_jail_vnet
> > but it also uses the old style (rc.config) jail configuration
> > format, but with extended number of parameters. The patch is dated
> > before the 9-1 release where the C-style jail configuration syntax
> > was taken into use.
> > 
> > is there a patch for /etc/rc.d/jail script that make it handles the
> > new style jail configuration file format (the C-style format). or am
> > I doing this the wrong way.
> > 
> > If you can point me into the right direction it will be very
> > helpfull.
> > 
> > Whishlist
> > It would be great with something like a /etc/jail.d/
> > setup where each jail configuration file is contained in a separate
> > file, the format is the same as the /etc/jail.conf, the latter fil
> > just holds configurations for all jails. Both files with the C-style
> > syntax.
> 
> Have you considered using sysutils/ezjail (the de-facto standard for
> managing jails under FreeBSD right now) or a similar port
> (sysutils/jailrc, sysutils/qjail etc.)?
> 

I missed the vnet requirement in my first reply. There has been a vnet
patch for ezjail floating around a while ago, but I think it was never
adopted to ezjail. Sorry for the noise.

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD 9.1-Rel: Jail-rc startup of vnet jails with new configuration syntax

[Michael Gmelin]

On Thu, 07 Nov 2013 13:26:10 +0100
Kristen Nielsen  wrote:

> Hi freebsd-virtualization list
> 
> I am trying to figure out how to start configured jails within
> FreeBSD 9.1Release at boot time.
> I am using new C-style jail configuration syntax in /etc/jail.conf.
> all jail starts successfully when i do a jail -c from the commandline.
> 
> I am trying to configure jails to start at boot time using the
> features in the /etc/rc.conf file.
> 
> in /etc/rc.conf at base host i have
> jail_enable="YES"
> jail_list=" jail1 jail2 jail3 "
> 
> and no more jail related lines, evereything else is placed into the
> jail configuration (C-style) file.
> 
> running "service jail start":
> Configuring jails:.
> Starting jails:/etc/rc.d/jail: ERROR: jail: No hostname has been
> defined for jail1
> 
> Looking into the /etc/rc.d/jail script it seems it expects that the 
> /etc/rc.conf contains old style jail configuration statement like
> 
> jail__hostname="jail1"
> etc.
> 
> I have looked at the patch at: 
> http://wiki.polymorf.fr/index.php/Howto:FreeBSD_jail_vnet
> but it also uses the old style (rc.config) jail configuration format, 
> but with extended number of parameters. The patch is dated before the 
> 9-1 release where the C-style jail configuration syntax was taken
> into use.
> 
> is there a patch for /etc/rc.d/jail script that make it handles the
> new style jail configuration file format (the C-style format). or am
> I doing this the wrong way.
> 
> If you can point me into the right direction it will be very helpfull.
> 
> Whishlist
> It would be great with something like a /etc/jail.d/ setup 
> where each jail configuration file is contained in a separate file,
> the format is the same as the /etc/jail.conf, the latter fil just
> holds configurations for all jails. Both files with the C-style
> syntax.

Have you considered using sysutils/ezjail (the de-facto standard for
managing jails under FreeBSD right now) or a similar port
(sysutils/jailrc, sysutils/qjail etc.)?

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: rc-style script for bhyve VM provisioning and management

[Michael Gmelin]

On Sun, 03 Nov 2013 00:18:09 -0700
Michael Dexter  wrote:

> 
> Hello all from the FreeBSD 20th Anniversary Party in San Francisco,
> 
> I have published a bhyve virtual machine provisioning and management
> framework to simplify the building and deploying of bhyve VMs during
> the run-up to the FreeBSD 10.0 release. As you may know, bhyve is
> rapidly finalizing for release in FreeBSD 10.0 and most features are
> finally becoming stable enough to document in a canonical manner.
> These scripts demonstrate the many opportunities for customization of
> bhyve deployments and will embrace features like OpenBSD and Linux
> support as soon as key components like the userland grub* arrive
> (currently in final testing). It will also demonstrate the use of pf
> to allow bhyve networking on wireless interfaces.
> 
> It can be downloaded from:
> 
> http://bhyve.org/bhyve-script.tar
> 
> It works like a rc script but does not comply with the FreeBSD way. I
> am working with several developers to determine to what degree it
> should be compatible with existing frameworks like ezjail.
> 
> ./vm0 will give the usage:
> 
> usage (start|stop|load|boot|destroy|restart|list|debug
> mount|umount|jail|format|fetch|install|provision|wipe)
> 
> Some routines like "provision" include steps like fetch (distribution
> sets), format (disk images or volumes) and install them.
> 
> The "jail" routine demonstrates how a virtual machine can be launched
> using the jail(8) command, because we can, given that bhyve FreeBSD
> VMs are simply instances of FreeBSD with the ABI limitations of
> different releases. bhyve will support any vision of FreeBSD with
> VirtIO, specifically FreeBSD 8.4 and 8-STABLE, 9.2 and 9-STABLE, and
> any recent version of 10 or 11.
> 
> Detailed instructions are included in the vm0 script and it is
> designed to simply be copied to vm1, vm2 etc. with easily-configured
> global variables.
> 
> While tmux(1) is optional and supports both captive and detached
> execution, a script is included to list and attach to running
> VMs/sessions. A script is also provided for the easy duplication of
> VMs and I have tested this with up to 15 X 16 vCPU instances though
> far more should work without issue.
> 
> Please throw everything you can at bhyve to expose any remaining flaws
> prior to the FreeBSD 10.0 release.
> 
> All the best,
> 
> Michael Dexter

Sounds similar to how I ran my VMs for a while (well, I ln -s from the
original VM script instead of copying, so it's easier to update all at
once). I finally changed the script to use profiles a few days ago,
since it seemed much cleaner in general and better for automatic control
and also complies to the FreeBSD way. I submitted a PR for
sysutils/bhyve-rc literally hours before you released vm0
(http://www.freebsd.org/cgi/query-pr.cgi?pr=183604), great timing ;)
It's only good for one use case, but does that very well.

I like the idea of the script using the general structure of ezjail. I
would also suggest to split this it two scripts, one rc script, that
only provides non-destructive operations like
(start|stop|restart|rcvar) and one called vm-admin which provides the
provisions functions on top, e.g.

service vm start vm0 # ok
vm-admin start vm0 # ok
service vm format vm0 ... # not ok
vm-admin format vm0 ... # ok

-- 
Michael Gmelin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

rc.d script for running bhyve in tmux

[Michael Gmelin]

Hi,

I hacked together a small rc.d script for starting bhyve(4) in tmux,
maybe it's useful to others too. Place it in /usr/local/etc/rc.d and
install sysutils/tmux. Might need some love to fit your needs.

Minimal configuration example:
pkg install tmux

echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf

cat >> /boot/loader.conf << EOF
if_tap_load="YES"
if_bridge_load="YES"
vmm_load="YES"
EOF

kldload tap; kldload bridge; kldload vmm

cat >> /etc/rc.conf << EOF
cloned_interfaces="tap0 bridge0"
bhyve_enable="YES"
bhyve_diskdev="/dev/zvol/tank/bhyve/virt"
EOF

ifconfig tap0 create
ifconfig bridge0 create

service bhyve start
tmux list-sessions
tmux attach -t bhyve
service bhyve status
service bhyve stop

Multi profile configuration example:
cat >> /etc/rc.conf << EOF
cloned_interfaces="tap0 tap1 bridge0"
bhyve_enable="YES"
bhyve_profiles="virt1 virt2"
bhyve_virt1_diskdev="/dev/zvol/tank/bhyve/virt1"

bhyve_virt2_tapdev="tap1"
bhyve_virt2_diskdev="/dev/zvol/tank/bhyve/virt2"
bhyve_virt2_memsize="8192"
bhyve_virt2_ncpu="4"
EOF

ifconfig tap0 create
ifconfig tap1 create
ifconfig bridge0 create

service bhyve start # start all
service bhyve start virt2 # start individual
tmux attach -t bhyve_virt1
tmux attach -t bhyve_virt1
service bhyve stop virt2 # stop individual
service bhyve stop # stop all

(by default ctrl-b d detaches from tmux).

Cheers,
Michael

-- 
Michael Gmelin
#!/bin/sh
#
# $FreeBSD: mgmelin
#

# PROVIDE: bhyve
# REQUIRE: LOGIN
# KEYWORD: nojail
#

#
# Add the following lines to /etc/rc.conf to enable bhyve:
# bhyve_enable (bool):  Set to "NO" by default.
#   Set it to "YES" to enable bhyve
# bhyve_profiles (str): Set to "" by default.
#   Define your profiles here.
# bhyve_tapdev (str):   Set to "tap0" by default.
#   Set to the tap(4) device to use.
# bhyve_diskdev (str):  Must be set, no default.
#   Set to the disk device to use.
# bhyve_ncpu (int): Set to 1 by default.
#   Set to the number of CPUs for the VM.
# bhyve_memsize (int):  Set to 512 by default.
#   Set to the number of MB of memory for the VM.

. /etc/rc.subr

name="bhyve"
rcvar=bhyve_enable

start_precmd="bhyve_prestart"
status_cmd="bhyve_status"
poll_cmd="bhyve_poll"
stop_cmd="bhyve_stop"
_session=$name
command="/usr/local/bin/tmux"
procname="-sh"

[ -z "$bhyve_enable" ]  && bhyve_enable="NO"
[ -z "$bhyve_tapdev" ]  && bhyve_tapdev="tap0"
[ -z "$bhyve_diskdev" ] && bhyve_diskdev="none"
[ -z "$bhyve_ncpu" ]&& bhyve_ncpu="1"
[ -z "$bhyve_memsize" ] && bhyve_memsize="512"

load_rc_config $name

if [ -n "$2" ]; then
profile="$2"
_session="${_session}_${profile}"
if [ "x${bhyve_profiles}" != "x" ]; then
eval bhyve_enable="\${${_session}_enable:-${bhyve_enable}}"
eval bhyve_tapdev="\${${_session}_tapdev:-${bhyve_tapdev}}"
eval bhyve_diskdev="\${${_session}_diskdev:-${bhyve_diskdev}}"
eval bhyve_ncpu="\${${_session}_ncpu:-${bhyve_ncpu}}"
eval bhyve_memsize="\${${_session}_memsize:-${bhyve_memsize}}"
else
echo "$0: extra argument ignored"
fi
else
if [ "x${bhyve_profiles}" != "x" -a "x$1" != "x" ]; then
for profile in ${bhyve_profiles}; do
eval _enable="\${bhyve_${profile}_enable}"
case "x${_enable:-${bhyve_enable}}" in
x|x[Nn][Oo]|x[Nn][Oo][Nn][Ee])
continue
;;
x[Yy][Ee][Ss])
;;
*)
if test -z "$_enable"; then
_var=bhyve_enable
else
_var=bhyve_"${profile}"_enable
fi
echo "Bad value" \
"'${_enable:-${bhyve_enable}}'" \
"for ${_var}. " \
"Profile ${profile} skipped."
continue
;;